2023-11-15T06:06:12.932 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-15T06:06:12.934 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458591 --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} --name yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi -- /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458591/orig.config.yaml 2023-11-15T06:06:12.984 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-15T06:06:13.079 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458591 branch: wip-yuri4-testing-2023-11-13-0820-pacific description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: yweinste@redhat.com first_in_suite: false job_id: '7458591' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri4-testing-2023-11-13-0820-pacific ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 620f25c1794f62207424fc9319427943fb8d1fae ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 620f25c1794f62207424fc9319427943fb8d1fae selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2023-11-13-0820-pacific sha1: 620f25c1794f62207424fc9319427943fb8d1fae owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 4886 sha1: 620f25c1794f62207424fc9319427943fb8d1fae sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri4-testing-2023-11-13-0820-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 620f25c1794f62207424fc9319427943fb8d1fae targets: smithi062.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEO4f9qXeLsyaEg8zm5TX1dLSdZfIsPBopSJGvx+tD1N5Rn3euSdRmOB7TSf8U9/zLF6g0j1B3WLxLEBSgYDXz8= smithi081.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBM5LBq+RfAdX5BWtHj8tKPv/mmy/VE0f2udeNzpMPJ0BW8X2ba+xgk0g2pU/d6UbD/pXqkBqqI6OCS0Vli8ETM= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - pexec: clients: - sudo modprobe -r ceph - sudo modprobe ceph disable_send_metrics=on - install: exclude_packages: - ceph-volume tag: v16.2.4 - print: '**** done install task...' - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6899cd26fceddb2fec83dc1a1349394b28c8998e timestamp: 2023-11-14_20:45:49 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3590714 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2023-11-15T06:06:13.080 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa; will attempt to use it 2023-11-15T06:06:13.081 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks 2023-11-15T06:06:13.081 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-11-15T06:06:13.083 INFO:teuthology.task.internal:Checking packages... 2023-11-15T06:06:13.106 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '620f25c1794f62207424fc9319427943fb8d1fae' 2023-11-15T06:06:13.107 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-15T06:06:13.107 INFO:teuthology.packaging:ref: None 2023-11-15T06:06:13.108 INFO:teuthology.packaging:tag: None 2023-11-15T06:06:13.108 INFO:teuthology.packaging:branch: wip-yuri4-testing-2023-11-13-0820-pacific 2023-11-15T06:06:13.108 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T06:06:13.109 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri4-testing-2023-11-13-0820-pacific 2023-11-15T06:06:13.517 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-170.g620f25c1 2023-11-15T06:06:13.518 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-11-15T06:06:13.525 INFO:teuthology.task.internal:no buildpackages task found 2023-11-15T06:06:13.526 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-11-15T06:06:13.559 INFO:teuthology.task.internal:Saving configuration 2023-11-15T06:06:13.579 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-11-15T06:06:13.591 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-15T06:06:13.615 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi062.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458591', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-15 06:01:05.773227', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEO4f9qXeLsyaEg8zm5TX1dLSdZfIsPBopSJGvx+tD1N5Rn3euSdRmOB7TSf8U9/zLF6g0j1B3WLxLEBSgYDXz8='} 2023-11-15T06:06:13.638 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi081.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458591', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-15 06:01:05.771625', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBM5LBq+RfAdX5BWtHj8tKPv/mmy/VE0f2udeNzpMPJ0BW8X2ba+xgk0g2pU/d6UbD/pXqkBqqI6OCS0Vli8ETM='} 2023-11-15T06:06:13.639 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-11-15T06:06:13.645 INFO:teuthology.task.internal:roles: ubuntu@smithi062.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2023-11-15T06:06:13.645 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2023-11-15T06:06:13.646 INFO:teuthology.run_tasks:Running task console_log... 2023-11-15T06:06:13.755 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f16f72b4b80>, signals=[15]) 2023-11-15T06:06:13.756 INFO:teuthology.run_tasks:Running task internal.connect... 2023-11-15T06:06:13.763 INFO:teuthology.task.internal:Opening connections... 2023-11-15T06:06:13.763 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi062.front.sepia.ceph.com 2023-11-15T06:06:13.765 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T06:06:13.839 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi081.front.sepia.ceph.com 2023-11-15T06:06:13.839 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T06:06:13.908 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-11-15T06:06:13.915 DEBUG:teuthology.orchestra.run.smithi062:> uname -m 2023-11-15T06:06:13.940 INFO:teuthology.orchestra.run.smithi062.stdout:x86_64 2023-11-15T06:06:13.940 DEBUG:teuthology.orchestra.run.smithi062:> cat /etc/os-release 2023-11-15T06:06:14.001 INFO:teuthology.orchestra.run.smithi062.stdout:NAME="CentOS Stream" 2023-11-15T06:06:14.002 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION="8" 2023-11-15T06:06:14.002 INFO:teuthology.orchestra.run.smithi062.stdout:ID="centos" 2023-11-15T06:06:14.002 INFO:teuthology.orchestra.run.smithi062.stdout:ID_LIKE="rhel fedora" 2023-11-15T06:06:14.002 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION_ID="8" 2023-11-15T06:06:14.003 INFO:teuthology.orchestra.run.smithi062.stdout:PLATFORM_ID="platform:el8" 2023-11-15T06:06:14.003 INFO:teuthology.orchestra.run.smithi062.stdout:PRETTY_NAME="CentOS Stream 8" 2023-11-15T06:06:14.003 INFO:teuthology.orchestra.run.smithi062.stdout:ANSI_COLOR="0;31" 2023-11-15T06:06:14.003 INFO:teuthology.orchestra.run.smithi062.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-11-15T06:06:14.003 INFO:teuthology.orchestra.run.smithi062.stdout:HOME_URL="https://centos.org/" 2023-11-15T06:06:14.004 INFO:teuthology.orchestra.run.smithi062.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-15T06:06:14.004 INFO:teuthology.orchestra.run.smithi062.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-15T06:06:14.004 INFO:teuthology.orchestra.run.smithi062.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-15T06:06:14.005 INFO:teuthology.lock.ops:Updating smithi062.front.sepia.ceph.com on lock server 2023-11-15T06:06:14.024 DEBUG:teuthology.orchestra.run.smithi081:> uname -m 2023-11-15T06:06:14.047 INFO:teuthology.orchestra.run.smithi081.stdout:x86_64 2023-11-15T06:06:14.047 DEBUG:teuthology.orchestra.run.smithi081:> cat /etc/os-release 2023-11-15T06:06:14.107 INFO:teuthology.orchestra.run.smithi081.stdout:NAME="CentOS Stream" 2023-11-15T06:06:14.107 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION="8" 2023-11-15T06:06:14.108 INFO:teuthology.orchestra.run.smithi081.stdout:ID="centos" 2023-11-15T06:06:14.108 INFO:teuthology.orchestra.run.smithi081.stdout:ID_LIKE="rhel fedora" 2023-11-15T06:06:14.108 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION_ID="8" 2023-11-15T06:06:14.108 INFO:teuthology.orchestra.run.smithi081.stdout:PLATFORM_ID="platform:el8" 2023-11-15T06:06:14.108 INFO:teuthology.orchestra.run.smithi081.stdout:PRETTY_NAME="CentOS Stream 8" 2023-11-15T06:06:14.109 INFO:teuthology.orchestra.run.smithi081.stdout:ANSI_COLOR="0;31" 2023-11-15T06:06:14.109 INFO:teuthology.orchestra.run.smithi081.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-11-15T06:06:14.109 INFO:teuthology.orchestra.run.smithi081.stdout:HOME_URL="https://centos.org/" 2023-11-15T06:06:14.109 INFO:teuthology.orchestra.run.smithi081.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-15T06:06:14.109 INFO:teuthology.orchestra.run.smithi081.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-15T06:06:14.110 INFO:teuthology.orchestra.run.smithi081.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-15T06:06:14.110 INFO:teuthology.lock.ops:Updating smithi081.front.sepia.ceph.com on lock server 2023-11-15T06:06:14.132 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-11-15T06:06:14.142 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-11-15T06:06:14.175 INFO:teuthology.task.internal:Checking for old test directory... 2023-11-15T06:06:14.175 DEBUG:teuthology.orchestra.run.smithi062:> test '!' -e /home/ubuntu/cephtest 2023-11-15T06:06:14.178 DEBUG:teuthology.orchestra.run.smithi081:> test '!' -e /home/ubuntu/cephtest 2023-11-15T06:06:14.198 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-11-15T06:06:14.208 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-11-15T06:06:14.208 DEBUG:teuthology.orchestra.run.smithi062:> test -z $(ls -A /var/lib/ceph) 2023-11-15T06:06:14.241 DEBUG:teuthology.orchestra.run.smithi081:> test -z $(ls -A /var/lib/ceph) 2023-11-15T06:06:14.276 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-11-15T06:06:14.327 INFO:teuthology.run_tasks:Running task kernel... 2023-11-15T06:06:14.344 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-11-15T06:06:14.345 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-11-15T06:06:14.345 DEBUG:teuthology.orchestra.run.smithi062:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-15T06:06:14.347 DEBUG:teuthology.orchestra.run.smithi081:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-15T06:06:14.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T06:06:14.367 DEBUG:teuthology.orchestra.run.smithi081:> uname -r 2023-11-15T06:06:14.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T06:06:14.368 DEBUG:teuthology.orchestra.run.smithi062:> uname -r 2023-11-15T06:06:14.426 INFO:teuthology.orchestra.run.smithi081.stdout:4.18.0-489.el8.x86_64 2023-11-15T06:06:14.426 INFO:teuthology.task.kernel:Running kernel on smithi081: 4.18.0-489.el8.x86_64 2023-11-15T06:06:14.426 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum install -y kernel 2023-11-15T06:06:14.430 INFO:teuthology.orchestra.run.smithi062.stdout:4.18.0-489.el8.x86_64 2023-11-15T06:06:14.431 INFO:teuthology.task.kernel:Running kernel on smithi062: 4.18.0-489.el8.x86_64 2023-11-15T06:06:14.431 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum install -y kernel 2023-11-15T06:06:24.777 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 8 - AppStream 4.3 MB/s | 34 MB 00:07 2023-11-15T06:06:24.785 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 8 - AppStream 4.4 MB/s | 34 MB 00:07 2023-11-15T06:06:35.173 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 8 - BaseOS 13 MB/s | 53 MB 00:04 2023-11-15T06:06:40.785 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 8 - BaseOS 5.7 MB/s | 53 MB 00:09 2023-11-15T06:06:44.168 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 8 - Extras 14 kB/s | 18 kB 00:01 2023-11-15T06:06:44.612 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 8 - Extras common packages 27 kB/s | 6.9 kB 00:00 2023-11-15T06:06:44.928 INFO:teuthology.orchestra.run.smithi081.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2023-11-15T06:06:51.551 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 8 - Extras 6.6 kB/s | 18 kB 00:02 2023-11-15T06:06:51.976 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 8 - Extras common packages 31 kB/s | 6.9 kB 00:00 2023-11-15T06:06:52.278 INFO:teuthology.orchestra.run.smithi062.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2023-11-15T06:06:53.606 INFO:teuthology.orchestra.run.smithi062.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2023-11-15T06:06:57.612 INFO:teuthology.orchestra.run.smithi062.stdout:lab-extras 296 kB/s | 24 kB 00:00 2023-11-15T06:07:01.620 INFO:teuthology.orchestra.run.smithi062.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-11-15T06:07:01.620 INFO:teuthology.orchestra.run.smithi062.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-11-15T06:07:01.909 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2023-11-15T06:07:01.910 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:07:01.910 INFO:teuthology.orchestra.run.smithi062.stdout: Package Architecture Version Repository Size 2023-11-15T06:07:01.910 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:07:01.910 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2023-11-15T06:07:01.911 INFO:teuthology.orchestra.run.smithi062.stdout: kernel x86_64 4.18.0-521.el8 baseos 10 M 2023-11-15T06:07:01.911 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2023-11-15T06:07:01.911 INFO:teuthology.orchestra.run.smithi062.stdout: kernel-core x86_64 4.18.0-521.el8 baseos 43 M 2023-11-15T06:07:01.911 INFO:teuthology.orchestra.run.smithi062.stdout: kernel-modules x86_64 4.18.0-521.el8 baseos 36 M 2023-11-15T06:07:01.911 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:07:01.912 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2023-11-15T06:07:01.912 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:07:01.912 INFO:teuthology.orchestra.run.smithi062.stdout:Install 3 Packages 2023-11-15T06:07:01.912 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:07:01.913 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 89 M 2023-11-15T06:07:01.913 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 96 M 2023-11-15T06:07:01.913 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2023-11-15T06:07:03.222 INFO:teuthology.orchestra.run.smithi062.stdout:(1/3): kernel-4.18.0-521.el8.x86_64.rpm 7.9 MB/s | 10 MB 00:01 2023-11-15T06:07:04.530 INFO:teuthology.orchestra.run.smithi062.stdout:(2/3): kernel-modules-4.18.0-521.el8.x86_64.rpm 14 MB/s | 36 MB 00:02 2023-11-15T06:07:05.132 INFO:teuthology.orchestra.run.smithi062.stdout:(3/3): kernel-core-4.18.0-521.el8.x86_64.rpm 14 MB/s | 43 MB 00:03 2023-11-15T06:07:05.132 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:07:05.133 INFO:teuthology.orchestra.run.smithi062.stdout:Total 28 MB/s | 89 MB 00:03 2023-11-15T06:07:05.839 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2023-11-15T06:07:06.285 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2023-11-15T06:07:06.285 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2023-11-15T06:07:07.582 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2023-11-15T06:07:07.614 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2023-11-15T06:07:10.595 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2023-11-15T06:07:11.123 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-15T06:07:14.864 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-15T06:07:15.248 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-15T06:07:18.919 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-15T06:07:19.200 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : kernel-4.18.0-521.el8.x86_64 3/3 2023-11-15T06:07:42.555 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 3/3 2023-11-15T06:07:42.579 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-15T06:07:43.242 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: kernel-4.18.0-521.el8.x86_64 3/3 2023-11-15T06:07:43.242 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : kernel-4.18.0-521.el8.x86_64 1/3 2023-11-15T06:07:43.242 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : kernel-core-4.18.0-521.el8.x86_64 2/3 2023-11-15T06:07:44.124 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-15T06:07:44.203 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:07:44.203 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2023-11-15T06:07:44.204 INFO:teuthology.orchestra.run.smithi062.stdout: kernel-4.18.0-521.el8.x86_64 kernel-core-4.18.0-521.el8.x86_64 2023-11-15T06:07:44.204 INFO:teuthology.orchestra.run.smithi062.stdout: kernel-modules-4.18.0-521.el8.x86_64 2023-11-15T06:07:44.204 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:07:44.204 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2023-11-15T06:07:44.298 DEBUG:teuthology.orchestra.run.smithi062:> rpm -q kernel | sort -rV | head -n 1 2023-11-15T06:07:44.526 INFO:teuthology.orchestra.run.smithi062.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-15T06:07:44.527 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-15T06:07:44.527 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-521.el8.x86_64 2023-11-15T06:07:44.527 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-15T06:07:44.528 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-15T06:07:44.528 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2023-11-15T06:07:44.528 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi062.front.sepia.ceph.com, path=None, version=distro) 2023-11-15T06:07:44.529 DEBUG:teuthology.orchestra.run.smithi062:> rpm -q kernel | sort -rV | head -n 1 2023-11-15T06:07:44.748 INFO:teuthology.orchestra.run.smithi062.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-15T06:07:44.748 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-15T06:07:44.748 DEBUG:teuthology.orchestra.run.smithi062:> sudo rpm -qi grub2-tools 2023-11-15T06:07:44.783 INFO:teuthology.orchestra.run.smithi062.stdout:Name : grub2-tools 2023-11-15T06:07:44.783 INFO:teuthology.orchestra.run.smithi062.stdout:Epoch : 1 2023-11-15T06:07:44.783 INFO:teuthology.orchestra.run.smithi062.stdout:Version : 2.02 2023-11-15T06:07:44.783 INFO:teuthology.orchestra.run.smithi062.stdout:Release : 129.el8 2023-11-15T06:07:44.784 INFO:teuthology.orchestra.run.smithi062.stdout:Architecture: x86_64 2023-11-15T06:07:44.784 INFO:teuthology.orchestra.run.smithi062.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-11-15T06:07:44.784 INFO:teuthology.orchestra.run.smithi062.stdout:Group : System Environment/Base 2023-11-15T06:07:44.784 INFO:teuthology.orchestra.run.smithi062.stdout:Size : 9265586 2023-11-15T06:07:44.784 INFO:teuthology.orchestra.run.smithi062.stdout:License : GPLv3+ 2023-11-15T06:07:44.785 INFO:teuthology.orchestra.run.smithi062.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-11-15T06:07:44.785 INFO:teuthology.orchestra.run.smithi062.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-11-15T06:07:44.785 INFO:teuthology.orchestra.run.smithi062.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-11-15T06:07:44.785 INFO:teuthology.orchestra.run.smithi062.stdout:Build Host : kbuilder.bsys.centos.org 2023-11-15T06:07:44.785 INFO:teuthology.orchestra.run.smithi062.stdout:Relocations : (not relocatable) 2023-11-15T06:07:44.786 INFO:teuthology.orchestra.run.smithi062.stdout:Packager : CentOS BuildSystem 2023-11-15T06:07:44.786 INFO:teuthology.orchestra.run.smithi062.stdout:Vendor : CentOS 2023-11-15T06:07:44.786 INFO:teuthology.orchestra.run.smithi062.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-15T06:07:44.786 INFO:teuthology.orchestra.run.smithi062.stdout:Summary : Support tools for GRUB. 2023-11-15T06:07:44.786 INFO:teuthology.orchestra.run.smithi062.stdout:Description : 2023-11-15T06:07:44.787 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:07:44.787 INFO:teuthology.orchestra.run.smithi062.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-15T06:07:44.787 INFO:teuthology.orchestra.run.smithi062.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-15T06:07:44.787 INFO:teuthology.orchestra.run.smithi062.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-15T06:07:44.787 INFO:teuthology.orchestra.run.smithi062.stdout:hardware devices. 2023-11-15T06:07:44.787 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:07:44.788 INFO:teuthology.orchestra.run.smithi062.stdout:This subpackage provides tools for support of all platforms. 2023-11-15T06:07:44.789 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-15T06:07:44.789 INFO:teuthology.task.kernel:Updating grub on smithi062 to boot 4.18.0-521.el8.x86_64 2023-11-15T06:07:44.789 DEBUG:teuthology.orchestra.run.smithi062:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-15T06:07:46.123 INFO:teuthology.orchestra.run.smithi062.stderr:Generating grub configuration file ... 2023-11-15T06:07:48.021 INFO:teuthology.orchestra.run.smithi062.stderr:done 2023-11-15T06:07:48.022 DEBUG:teuthology.orchestra.run.smithi062:> mktemp 2023-11-15T06:07:48.047 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/tmp.kUPikpjwoo 2023-11-15T06:07:48.047 DEBUG:teuthology.orchestra.run.smithi062:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.kUPikpjwoo 2023-11-15T06:07:48.119 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 0666 /tmp/tmp.kUPikpjwoo 2023-11-15T06:07:48.269 DEBUG:teuthology.orchestra.remote:smithi062:/tmp/tmp.kUPikpjwoo is 6KB 2023-11-15T06:07:48.317 DEBUG:teuthology.orchestra.run.smithi062:> rm -fr /tmp/tmp.kUPikpjwoo 2023-11-15T06:07:48.335 DEBUG:teuthology.orchestra.run.smithi062:> sudo /bin/ls /boot/loader/entries || true 2023-11-15T06:07:48.416 INFO:teuthology.orchestra.run.smithi062.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-11-15T06:07:48.417 INFO:teuthology.orchestra.run.smithi062.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-11-15T06:07:48.417 INFO:teuthology.orchestra.run.smithi062.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-11-15T06:07:48.417 INFO:teuthology.orchestra.run.smithi062.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64.conf 2023-11-15T06:07:48.418 DEBUG:teuthology.orchestra.run.smithi062:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64 2023-11-15T06:07:48.606 DEBUG:teuthology.orchestra.run.smithi062:> sudo shutdown -r now 2023-11-15T06:08:18.610 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2023-11-15T06:08:18.610 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2023-11-15T06:08:18.611 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T06:09:18.613 DEBUG:teuthology.orchestra.remote:timed out 2023-11-15T06:09:27.615 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2023-11-15T06:09:27.617 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T06:09:27.901 DEBUG:teuthology.orchestra.run.smithi062:> true 2023-11-15T06:09:28.209 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi062.front.sepia.ceph.com' 2023-11-15T06:09:28.210 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-521.el8.x86_64"... 2023-11-15T06:09:28.210 DEBUG:teuthology.orchestra.run.smithi062:> uname -r 2023-11-15T06:09:28.230 INFO:teuthology.orchestra.run.smithi062.stdout:4.18.0-521.el8.x86_64 2023-11-15T06:09:28.231 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-521.el8.x86_64 vs 4.18.0-521.el8.x86_64 2023-11-15T06:09:28.231 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-15T06:09:28.231 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-15T06:09:29.233 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2023-11-15T06:09:29.233 DEBUG:teuthology.orchestra.run.smithi062:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-15T06:09:29.315 INFO:teuthology.orchestra.run.smithi062.stdout:ttyS1 2023-11-15T06:09:29.327 DEBUG:teuthology.parallel:result is None 2023-11-15T06:10:23.507 INFO:teuthology.orchestra.run.smithi081.stdout:Extra Packages for Enterprise Linux 75 kB/s | 16 MB 03:38 2023-11-15T06:10:27.398 INFO:teuthology.orchestra.run.smithi081.stdout:lab-extras 169 kB/s | 24 kB 00:00 2023-11-15T06:10:30.989 INFO:teuthology.orchestra.run.smithi081.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-11-15T06:10:30.989 INFO:teuthology.orchestra.run.smithi081.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-11-15T06:10:31.271 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2023-11-15T06:10:31.273 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:10:31.273 INFO:teuthology.orchestra.run.smithi081.stdout: Package Architecture Version Repository Size 2023-11-15T06:10:31.274 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:10:31.274 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2023-11-15T06:10:31.275 INFO:teuthology.orchestra.run.smithi081.stdout: kernel x86_64 4.18.0-521.el8 baseos 10 M 2023-11-15T06:10:31.276 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2023-11-15T06:10:31.276 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-core x86_64 4.18.0-521.el8 baseos 43 M 2023-11-15T06:10:31.277 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-modules x86_64 4.18.0-521.el8 baseos 36 M 2023-11-15T06:10:31.277 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:10:31.277 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2023-11-15T06:10:31.278 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:10:31.278 INFO:teuthology.orchestra.run.smithi081.stdout:Install 3 Packages 2023-11-15T06:10:31.278 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:10:31.279 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 89 M 2023-11-15T06:10:31.279 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 96 M 2023-11-15T06:10:31.279 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2023-11-15T06:10:34.116 INFO:teuthology.orchestra.run.smithi081.stdout:(1/3): kernel-4.18.0-521.el8.x86_64.rpm 4.1 MB/s | 10 MB 00:02 2023-11-15T06:10:35.516 INFO:teuthology.orchestra.run.smithi081.stdout:(2/3): kernel-modules-4.18.0-521.el8.x86_64.rpm 9.3 MB/s | 36 MB 00:03 2023-11-15T06:10:36.775 INFO:teuthology.orchestra.run.smithi081.stdout:(3/3): kernel-core-4.18.0-521.el8.x86_64.rpm 8.4 MB/s | 43 MB 00:05 2023-11-15T06:10:36.775 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:10:36.775 INFO:teuthology.orchestra.run.smithi081.stdout:Total 16 MB/s | 89 MB 00:05 2023-11-15T06:10:37.452 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2023-11-15T06:10:37.871 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2023-11-15T06:10:37.872 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2023-11-15T06:10:39.097 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2023-11-15T06:10:39.121 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2023-11-15T06:10:41.867 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2023-11-15T06:10:42.458 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-15T06:10:46.070 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-15T06:10:46.458 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-15T06:10:49.878 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-15T06:10:50.138 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : kernel-4.18.0-521.el8.x86_64 3/3 2023-11-15T06:11:12.237 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 3/3 2023-11-15T06:11:12.262 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-15T06:11:12.652 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-4.18.0-521.el8.x86_64 3/3 2023-11-15T06:11:12.652 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : kernel-4.18.0-521.el8.x86_64 1/3 2023-11-15T06:11:12.652 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : kernel-core-4.18.0-521.el8.x86_64 2/3 2023-11-15T06:11:13.443 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-15T06:11:13.444 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:11:13.444 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2023-11-15T06:11:13.444 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-4.18.0-521.el8.x86_64 kernel-core-4.18.0-521.el8.x86_64 2023-11-15T06:11:13.444 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-modules-4.18.0-521.el8.x86_64 2023-11-15T06:11:13.445 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:11:13.445 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2023-11-15T06:11:13.584 DEBUG:teuthology.orchestra.run.smithi081:> rpm -q kernel | sort -rV | head -n 1 2023-11-15T06:11:13.802 INFO:teuthology.orchestra.run.smithi081.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-15T06:11:13.802 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-15T06:11:13.802 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-521.el8.x86_64 2023-11-15T06:11:13.803 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-15T06:11:13.803 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-15T06:11:13.803 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2023-11-15T06:11:13.803 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi081.front.sepia.ceph.com, path=None, version=distro) 2023-11-15T06:11:13.804 DEBUG:teuthology.orchestra.run.smithi081:> rpm -q kernel | sort -rV | head -n 1 2023-11-15T06:11:14.013 INFO:teuthology.orchestra.run.smithi081.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-15T06:11:14.013 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-15T06:11:14.014 DEBUG:teuthology.orchestra.run.smithi081:> sudo rpm -qi grub2-tools 2023-11-15T06:11:14.047 INFO:teuthology.orchestra.run.smithi081.stdout:Name : grub2-tools 2023-11-15T06:11:14.047 INFO:teuthology.orchestra.run.smithi081.stdout:Epoch : 1 2023-11-15T06:11:14.048 INFO:teuthology.orchestra.run.smithi081.stdout:Version : 2.02 2023-11-15T06:11:14.048 INFO:teuthology.orchestra.run.smithi081.stdout:Release : 129.el8 2023-11-15T06:11:14.048 INFO:teuthology.orchestra.run.smithi081.stdout:Architecture: x86_64 2023-11-15T06:11:14.048 INFO:teuthology.orchestra.run.smithi081.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-11-15T06:11:14.048 INFO:teuthology.orchestra.run.smithi081.stdout:Group : System Environment/Base 2023-11-15T06:11:14.049 INFO:teuthology.orchestra.run.smithi081.stdout:Size : 9265586 2023-11-15T06:11:14.049 INFO:teuthology.orchestra.run.smithi081.stdout:License : GPLv3+ 2023-11-15T06:11:14.049 INFO:teuthology.orchestra.run.smithi081.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-11-15T06:11:14.049 INFO:teuthology.orchestra.run.smithi081.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-11-15T06:11:14.049 INFO:teuthology.orchestra.run.smithi081.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-11-15T06:11:14.050 INFO:teuthology.orchestra.run.smithi081.stdout:Build Host : kbuilder.bsys.centos.org 2023-11-15T06:11:14.050 INFO:teuthology.orchestra.run.smithi081.stdout:Relocations : (not relocatable) 2023-11-15T06:11:14.050 INFO:teuthology.orchestra.run.smithi081.stdout:Packager : CentOS BuildSystem 2023-11-15T06:11:14.050 INFO:teuthology.orchestra.run.smithi081.stdout:Vendor : CentOS 2023-11-15T06:11:14.050 INFO:teuthology.orchestra.run.smithi081.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-15T06:11:14.051 INFO:teuthology.orchestra.run.smithi081.stdout:Summary : Support tools for GRUB. 2023-11-15T06:11:14.051 INFO:teuthology.orchestra.run.smithi081.stdout:Description : 2023-11-15T06:11:14.051 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:11:14.051 INFO:teuthology.orchestra.run.smithi081.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-15T06:11:14.051 INFO:teuthology.orchestra.run.smithi081.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-15T06:11:14.051 INFO:teuthology.orchestra.run.smithi081.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-15T06:11:14.052 INFO:teuthology.orchestra.run.smithi081.stdout:hardware devices. 2023-11-15T06:11:14.052 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:11:14.052 INFO:teuthology.orchestra.run.smithi081.stdout:This subpackage provides tools for support of all platforms. 2023-11-15T06:11:14.053 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-15T06:11:14.053 INFO:teuthology.task.kernel:Updating grub on smithi081 to boot 4.18.0-521.el8.x86_64 2023-11-15T06:11:14.054 DEBUG:teuthology.orchestra.run.smithi081:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-15T06:11:15.471 INFO:teuthology.orchestra.run.smithi081.stderr:Generating grub configuration file ... 2023-11-15T06:11:17.302 INFO:teuthology.orchestra.run.smithi081.stderr:done 2023-11-15T06:11:17.303 DEBUG:teuthology.orchestra.run.smithi081:> mktemp 2023-11-15T06:11:17.333 INFO:teuthology.orchestra.run.smithi081.stdout:/tmp/tmp.VeYtphv641 2023-11-15T06:11:17.334 DEBUG:teuthology.orchestra.run.smithi081:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.VeYtphv641 2023-11-15T06:11:17.430 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0666 /tmp/tmp.VeYtphv641 2023-11-15T06:11:17.549 DEBUG:teuthology.orchestra.remote:smithi081:/tmp/tmp.VeYtphv641 is 6KB 2023-11-15T06:11:17.598 DEBUG:teuthology.orchestra.run.smithi081:> rm -fr /tmp/tmp.VeYtphv641 2023-11-15T06:11:17.616 DEBUG:teuthology.orchestra.run.smithi081:> sudo /bin/ls /boot/loader/entries || true 2023-11-15T06:11:17.696 INFO:teuthology.orchestra.run.smithi081.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-11-15T06:11:17.696 INFO:teuthology.orchestra.run.smithi081.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-11-15T06:11:17.696 INFO:teuthology.orchestra.run.smithi081.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-11-15T06:11:17.697 INFO:teuthology.orchestra.run.smithi081.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64.conf 2023-11-15T06:11:17.698 DEBUG:teuthology.orchestra.run.smithi081:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64 2023-11-15T06:11:17.848 DEBUG:teuthology.orchestra.run.smithi081:> sudo shutdown -r now 2023-11-15T06:11:47.851 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2023-11-15T06:11:47.852 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2023-11-15T06:11:47.853 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T06:12:06.398 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81 2023-11-15T06:12:15.400 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2023-11-15T06:12:15.401 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T06:12:18.458 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81 2023-11-15T06:12:30.459 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2023-11-15T06:12:30.460 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T06:12:48.890 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81 2023-11-15T06:13:03.892 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2023-11-15T06:13:03.893 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T06:13:04.274 DEBUG:teuthology.orchestra.run.smithi081:> true 2023-11-15T06:13:04.952 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi081.front.sepia.ceph.com' 2023-11-15T06:13:04.953 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-521.el8.x86_64"... 2023-11-15T06:13:04.953 DEBUG:teuthology.orchestra.run.smithi081:> uname -r 2023-11-15T06:13:05.118 INFO:teuthology.orchestra.run.smithi081.stdout:4.18.0-521.el8.x86_64 2023-11-15T06:13:05.118 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-521.el8.x86_64 vs 4.18.0-521.el8.x86_64 2023-11-15T06:13:05.118 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-15T06:13:05.119 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-15T06:13:06.120 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2023-11-15T06:13:06.121 DEBUG:teuthology.orchestra.run.smithi081:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-15T06:13:06.465 INFO:teuthology.orchestra.run.smithi081.stdout:ttyS1 2023-11-15T06:13:06.480 DEBUG:teuthology.parallel:result is None 2023-11-15T06:13:06.480 INFO:teuthology.run_tasks:Running task internal.base... 2023-11-15T06:13:06.489 INFO:teuthology.task.internal:Creating test directory... 2023-11-15T06:13:06.490 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-15T06:13:06.498 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-15T06:13:06.515 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-11-15T06:13:06.527 INFO:teuthology.run_tasks:Running task internal.archive... 2023-11-15T06:13:06.568 INFO:teuthology.task.internal:Creating archive directory... 2023-11-15T06:13:06.568 DEBUG:teuthology.orchestra.run.smithi062:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-15T06:13:06.571 DEBUG:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-15T06:13:06.744 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-11-15T06:13:06.751 INFO:teuthology.task.internal:Enabling coredump saving... 2023-11-15T06:13:06.752 DEBUG:teuthology.orchestra.run.smithi062:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-15T06:13:06.754 DEBUG:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-15T06:13:06.794 INFO:teuthology.orchestra.run.smithi062.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-15T06:13:06.806 INFO:teuthology.orchestra.run.smithi062.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-15T06:13:06.844 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-15T06:13:06.853 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-15T06:13:06.854 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-11-15T06:13:06.862 INFO:teuthology.task.internal:Configuring sudo... 2023-11-15T06:13:06.863 DEBUG:teuthology.orchestra.run.smithi062:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-15T06:13:06.865 DEBUG:teuthology.orchestra.run.smithi081:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-15T06:13:06.894 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-11-15T06:13:06.905 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-11-15T06:13:06.905 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-15T06:13:06.938 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-15T06:13:06.955 DEBUG:teuthology.orchestra.run.smithi062:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-15T06:13:07.041 DEBUG:teuthology.orchestra.run.smithi062:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-15T06:13:07.095 DEBUG:teuthology.orchestra.run.smithi062:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-15T06:13:07.181 DEBUG:teuthology.orchestra.run.smithi062:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-15T06:13:07.223 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-11-15T06:13:07.224 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-15T06:13:07.290 DEBUG:teuthology.orchestra.run.smithi081:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-15T06:13:07.336 DEBUG:teuthology.orchestra.run.smithi081:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-15T06:13:07.384 DEBUG:teuthology.orchestra.run.smithi081:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-15T06:13:07.469 DEBUG:teuthology.orchestra.run.smithi081:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-15T06:13:07.509 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2023-11-15T06:13:07.509 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-15T06:13:07.576 DEBUG:teuthology.orchestra.run.smithi062:> sudo service rsyslog restart 2023-11-15T06:13:07.578 DEBUG:teuthology.orchestra.run.smithi081:> sudo service rsyslog restart 2023-11-15T06:13:07.657 INFO:teuthology.orchestra.run.smithi062.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-15T06:13:07.677 INFO:teuthology.orchestra.run.smithi081.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-15T06:13:08.085 INFO:teuthology.run_tasks:Running task internal.timer... 2023-11-15T06:13:08.093 INFO:teuthology.task.internal:Starting timer... 2023-11-15T06:13:08.094 INFO:teuthology.run_tasks:Running task pcp... 2023-11-15T06:13:08.142 INFO:teuthology.run_tasks:Running task selinux... 2023-11-15T06:13:08.187 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-11-15T06:13:08.188 DEBUG:teuthology.orchestra.run.smithi062:> sudo service auditd rotate 2023-11-15T06:13:08.257 INFO:teuthology.orchestra.run.smithi062.stdout:Rotating logs: 2023-11-15T06:13:08.258 DEBUG:teuthology.orchestra.run.smithi081:> sudo service auditd rotate 2023-11-15T06:13:08.345 INFO:teuthology.orchestra.run.smithi081.stdout:Rotating logs: 2023-11-15T06:13:08.346 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-11-15T06:13:08.347 DEBUG:teuthology.orchestra.run.smithi062:> /usr/sbin/getenforce 2023-11-15T06:13:08.370 INFO:teuthology.orchestra.run.smithi062.stdout:Permissive 2023-11-15T06:13:08.371 DEBUG:teuthology.orchestra.run.smithi081:> /usr/sbin/getenforce 2023-11-15T06:13:08.395 INFO:teuthology.orchestra.run.smithi081.stdout:Permissive 2023-11-15T06:13:08.396 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi062.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi081.front.sepia.ceph.com': 'permissive'} 2023-11-15T06:13:08.396 DEBUG:teuthology.orchestra.run.smithi062:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-15T06:13:08.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T06:13:08.439 DEBUG:teuthology.orchestra.run.smithi081:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-15T06:13:08.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T06:13:08.466 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-11-15T06:13:08.467 DEBUG:teuthology.orchestra.run.smithi062:> sudo /usr/sbin/setenforce permissive 2023-11-15T06:13:08.507 DEBUG:teuthology.orchestra.run.smithi081:> sudo /usr/sbin/setenforce permissive 2023-11-15T06:13:08.535 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-11-15T06:13:08.545 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-11-15T06:13:08.545 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-11-15T06:13:08.563 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-11-15T06:13:08.564 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi062.front.sepia.ceph.com,smithi081.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-11-15T06:20:43.709 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi062.front.sepia.ceph.com'), Remote(name='ubuntu@smithi081.front.sepia.ceph.com')] 2023-11-15T06:20:43.710 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2023-11-15T06:20:43.711 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T06:20:43.783 DEBUG:teuthology.orchestra.run.smithi062:> true 2023-11-15T06:20:43.854 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi062.front.sepia.ceph.com' 2023-11-15T06:20:43.855 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2023-11-15T06:20:43.855 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T06:20:43.926 DEBUG:teuthology.orchestra.run.smithi081:> true 2023-11-15T06:20:43.996 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi081.front.sepia.ceph.com' 2023-11-15T06:20:43.997 INFO:teuthology.run_tasks:Running task clock... 2023-11-15T06:20:44.007 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-11-15T06:20:44.009 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-15T06:20:44.009 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-15T06:20:44.012 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-15T06:20:44.014 DEBUG:teuthology.orchestra.run.smithi081:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-15T06:20:44.046 INFO:teuthology.orchestra.run.smithi062.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-15T06:20:44.061 INFO:teuthology.orchestra.run.smithi062.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-15T06:20:44.067 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-15T06:20:44.081 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-15T06:20:44.128 INFO:teuthology.orchestra.run.smithi062.stderr:sudo: ntpd: command not found 2023-11-15T06:20:44.139 INFO:teuthology.orchestra.run.smithi062.stdout:506 Cannot talk to daemon 2023-11-15T06:20:44.144 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: ntpd: command not found 2023-11-15T06:20:44.153 INFO:teuthology.orchestra.run.smithi062.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-15T06:20:44.155 INFO:teuthology.orchestra.run.smithi081.stdout:506 Cannot talk to daemon 2023-11-15T06:20:44.167 INFO:teuthology.orchestra.run.smithi062.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-15T06:20:44.169 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-15T06:20:44.182 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-15T06:20:44.235 INFO:teuthology.orchestra.run.smithi062.stderr:bash: ntpq: command not found 2023-11-15T06:20:44.239 INFO:teuthology.orchestra.run.smithi062.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-15T06:20:44.239 INFO:teuthology.orchestra.run.smithi062.stdout:=============================================================================== 2023-11-15T06:20:44.239 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T06:20:44.239 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T06:20:44.240 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T06:20:44.240 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T06:20:44.255 INFO:teuthology.orchestra.run.smithi081.stderr:bash: ntpq: command not found 2023-11-15T06:20:44.259 INFO:teuthology.orchestra.run.smithi081.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-15T06:20:44.259 INFO:teuthology.orchestra.run.smithi081.stdout:=============================================================================== 2023-11-15T06:20:44.259 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T06:20:44.259 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T06:20:44.259 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T06:20:44.260 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T06:20:44.261 INFO:teuthology.run_tasks:Running task pexec... 2023-11-15T06:20:44.271 INFO:teuthology.task.pexec:Executing custom commands... 2023-11-15T06:20:44.271 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi062.front.sepia.ceph.com 2023-11-15T06:20:44.272 DEBUG:teuthology.orchestra.run.smithi062:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-15T06:20:44.272 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi081.front.sepia.ceph.com 2023-11-15T06:20:44.273 DEBUG:teuthology.orchestra.run.smithi081:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-15T06:20:45.176 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:45 ago on Wed 15 Nov 2023 06:19:00 AM UTC. 2023-11-15T06:20:45.194 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:37 ago on Wed 15 Nov 2023 06:19:08 AM UTC. 2023-11-15T06:20:47.252 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2023-11-15T06:20:47.252 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:20:47.253 INFO:teuthology.orchestra.run.smithi081.stdout: Package Architecture Version Repository Size 2023-11-15T06:20:47.253 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:20:47.253 INFO:teuthology.orchestra.run.smithi081.stdout:Resetting modules: 2023-11-15T06:20:47.253 INFO:teuthology.orchestra.run.smithi081.stdout: container-tools 2023-11-15T06:20:47.253 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:20:47.254 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2023-11-15T06:20:47.254 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:20:47.254 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:20:47.474 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2023-11-15T06:20:47.475 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:20:47.475 INFO:teuthology.orchestra.run.smithi062.stdout: Package Architecture Version Repository Size 2023-11-15T06:20:47.476 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:20:47.476 INFO:teuthology.orchestra.run.smithi062.stdout:Resetting modules: 2023-11-15T06:20:47.476 INFO:teuthology.orchestra.run.smithi062.stdout: container-tools 2023-11-15T06:20:47.476 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:20:47.476 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2023-11-15T06:20:47.477 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:20:47.477 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:20:47.509 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2023-11-15T06:20:47.730 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2023-11-15T06:20:48.458 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:48 ago on Wed 15 Nov 2023 06:19:00 AM UTC. 2023-11-15T06:20:48.696 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:40 ago on Wed 15 Nov 2023 06:19:08 AM UTC. 2023-11-15T06:20:50.581 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2023-11-15T06:20:50.760 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:20:50.760 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2023-11-15T06:20:50.761 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:20:50.761 INFO:teuthology.orchestra.run.smithi081.stdout:Installing group/module packages: 2023-11-15T06:20:50.761 INFO:teuthology.orchestra.run.smithi081.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2023-11-15T06:20:50.761 INFO:teuthology.orchestra.run.smithi081.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2023-11-15T06:20:50.762 INFO:teuthology.orchestra.run.smithi081.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2023-11-15T06:20:50.762 INFO:teuthology.orchestra.run.smithi081.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2023-11-15T06:20:50.762 INFO:teuthology.orchestra.run.smithi081.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2023-11-15T06:20:50.762 INFO:teuthology.orchestra.run.smithi081.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2023-11-15T06:20:50.762 INFO:teuthology.orchestra.run.smithi081.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2023-11-15T06:20:50.763 INFO:teuthology.orchestra.run.smithi081.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2023-11-15T06:20:50.763 INFO:teuthology.orchestra.run.smithi081.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2023-11-15T06:20:50.763 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2023-11-15T06:20:50.763 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-11-15T06:20:50.763 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-11-15T06:20:50.764 INFO:teuthology.orchestra.run.smithi081.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-11-15T06:20:50.764 INFO:teuthology.orchestra.run.smithi081.stdout:Installing weak dependencies: 2023-11-15T06:20:50.764 INFO:teuthology.orchestra.run.smithi081.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-11-15T06:20:50.764 INFO:teuthology.orchestra.run.smithi081.stdout:Installing module profiles: 2023-11-15T06:20:50.764 INFO:teuthology.orchestra.run.smithi081.stdout: container-tools/common 2023-11-15T06:20:50.765 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:20:50.765 INFO:teuthology.orchestra.run.smithi081.stdout:Enabling module streams: 2023-11-15T06:20:50.765 INFO:teuthology.orchestra.run.smithi081.stdout: container-tools rhel8 2023-11-15T06:20:50.765 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:20:50.765 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2023-11-15T06:20:50.766 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:20:50.766 INFO:teuthology.orchestra.run.smithi081.stdout:Install 13 Packages 2023-11-15T06:20:50.766 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:20:50.767 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 27 M 2023-11-15T06:20:50.767 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 89 M 2023-11-15T06:20:50.768 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2023-11-15T06:20:50.949 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2023-11-15T06:20:51.049 INFO:teuthology.orchestra.run.smithi081.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 3.0 MB/s | 750 kB 00:00 2023-11-15T06:20:51.107 INFO:teuthology.orchestra.run.smithi081.stdout:(2/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 3.3 MB/s | 1.0 MB 00:00 2023-11-15T06:20:51.125 INFO:teuthology.orchestra.run.smithi081.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 503 kB/s | 38 kB 00:00 2023-11-15T06:20:51.142 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:20:51.142 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2023-11-15T06:20:51.142 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:20:51.143 INFO:teuthology.orchestra.run.smithi062.stdout:Installing group/module packages: 2023-11-15T06:20:51.143 INFO:teuthology.orchestra.run.smithi062.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2023-11-15T06:20:51.143 INFO:teuthology.orchestra.run.smithi062.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2023-11-15T06:20:51.143 INFO:teuthology.orchestra.run.smithi062.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2023-11-15T06:20:51.143 INFO:teuthology.orchestra.run.smithi062.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2023-11-15T06:20:51.144 INFO:teuthology.orchestra.run.smithi062.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2023-11-15T06:20:51.144 INFO:teuthology.orchestra.run.smithi062.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2023-11-15T06:20:51.144 INFO:teuthology.orchestra.run.smithi062.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2023-11-15T06:20:51.144 INFO:teuthology.orchestra.run.smithi062.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2023-11-15T06:20:51.144 INFO:teuthology.orchestra.run.smithi062.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2023-11-15T06:20:51.145 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2023-11-15T06:20:51.145 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-11-15T06:20:51.145 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-11-15T06:20:51.145 INFO:teuthology.orchestra.run.smithi062.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-11-15T06:20:51.145 INFO:teuthology.orchestra.run.smithi062.stdout:Installing weak dependencies: 2023-11-15T06:20:51.146 INFO:teuthology.orchestra.run.smithi062.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-11-15T06:20:51.146 INFO:teuthology.orchestra.run.smithi062.stdout:Installing module profiles: 2023-11-15T06:20:51.146 INFO:teuthology.orchestra.run.smithi062.stdout: container-tools/common 2023-11-15T06:20:51.146 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:20:51.146 INFO:teuthology.orchestra.run.smithi062.stdout:Enabling module streams: 2023-11-15T06:20:51.147 INFO:teuthology.orchestra.run.smithi062.stdout: container-tools rhel8 2023-11-15T06:20:51.147 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:20:51.147 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2023-11-15T06:20:51.147 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:20:51.147 INFO:teuthology.orchestra.run.smithi062.stdout:Install 13 Packages 2023-11-15T06:20:51.148 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:20:51.148 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 27 M 2023-11-15T06:20:51.149 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 89 M 2023-11-15T06:20:51.149 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2023-11-15T06:20:51.200 INFO:teuthology.orchestra.run.smithi081.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 2.6 MB/s | 241 kB 00:00 2023-11-15T06:20:51.267 INFO:teuthology.orchestra.run.smithi081.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 2.6 MB/s | 178 kB 00:00 2023-11-15T06:20:51.334 INFO:teuthology.orchestra.run.smithi081.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 371 kB/s | 25 kB 00:00 2023-11-15T06:20:51.392 INFO:teuthology.orchestra.run.smithi081.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.6 MB/s | 94 kB 00:00 2023-11-15T06:20:51.551 INFO:teuthology.orchestra.run.smithi081.stdout:(8/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 9.2 MB/s | 3.9 MB 00:00 2023-11-15T06:20:51.650 INFO:teuthology.orchestra.run.smithi062.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 2.5 MB/s | 750 kB 00:00 2023-11-15T06:20:51.734 INFO:teuthology.orchestra.run.smithi062.stdout:(2/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 2.7 MB/s | 1.0 MB 00:00 2023-11-15T06:20:51.751 INFO:teuthology.orchestra.run.smithi062.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 378 kB/s | 38 kB 00:00 2023-11-15T06:20:51.793 INFO:teuthology.orchestra.run.smithi062.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 4.0 MB/s | 241 kB 00:00 2023-11-15T06:20:51.819 INFO:teuthology.orchestra.run.smithi081.stdout:(9/13): toolbox-0.0.99.4-6.module_el8+712+4cd1b 9.5 MB/s | 2.5 MB 00:00 2023-11-15T06:20:51.852 INFO:teuthology.orchestra.run.smithi062.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 3.0 MB/s | 178 kB 00:00 2023-11-15T06:20:51.869 INFO:teuthology.orchestra.run.smithi081.stdout:(10/13): udica-0.2.6-20.module_el8+712+4cd1bd69 1.0 MB/s | 53 kB 00:00 2023-11-15T06:20:51.911 INFO:teuthology.orchestra.run.smithi081.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 981 kB/s | 41 kB 00:00 2023-11-15T06:20:51.912 INFO:teuthology.orchestra.run.smithi062.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 424 kB/s | 25 kB 00:00 2023-11-15T06:20:51.977 INFO:teuthology.orchestra.run.smithi062.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.4 MB/s | 94 kB 00:00 2023-11-15T06:20:52.102 INFO:teuthology.orchestra.run.smithi062.stdout:(8/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 11 MB/s | 3.9 MB 00:00 2023-11-15T06:20:52.136 INFO:teuthology.orchestra.run.smithi081.stdout:(12/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 11 MB/s | 8.1 MB 00:00 2023-11-15T06:20:52.294 INFO:teuthology.orchestra.run.smithi062.stdout:(9/13): toolbox-0.0.99.4-6.module_el8+712+4cd1b 13 MB/s | 2.5 MB 00:00 2023-11-15T06:20:52.345 INFO:teuthology.orchestra.run.smithi081.stdout:(13/13): buildah-1.32.0-1.module_el8+712+4cd1bd 6.2 MB/s | 9.6 MB 00:01 2023-11-15T06:20:52.345 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:20:52.346 INFO:teuthology.orchestra.run.smithi081.stdout:Total 17 MB/s | 27 MB 00:01 2023-11-15T06:20:52.403 INFO:teuthology.orchestra.run.smithi062.stdout:(10/13): buildah-1.32.0-1.module_el8+712+4cd1bd 9.2 MB/s | 9.6 MB 00:01 2023-11-15T06:20:52.429 INFO:teuthology.orchestra.run.smithi062.stdout:(11/13): udica-0.2.6-20.module_el8+712+4cd1bd69 393 kB/s | 53 kB 00:00 2023-11-15T06:20:52.462 INFO:teuthology.orchestra.run.smithi062.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 701 kB/s | 41 kB 00:00 2023-11-15T06:20:52.614 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2023-11-15T06:20:52.634 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2023-11-15T06:20:52.635 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2023-11-15T06:20:52.688 INFO:teuthology.orchestra.run.smithi062.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 11 MB/s | 8.1 MB 00:00 2023-11-15T06:20:52.688 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:20:52.688 INFO:teuthology.orchestra.run.smithi062.stdout:Total 17 MB/s | 27 MB 00:01 2023-11-15T06:20:52.899 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2023-11-15T06:20:52.902 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2023-11-15T06:20:52.965 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2023-11-15T06:20:52.988 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2023-11-15T06:20:52.988 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2023-11-15T06:20:53.249 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2023-11-15T06:20:53.253 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2023-11-15T06:20:53.650 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2023-11-15T06:20:53.928 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-11-15T06:20:53.963 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2023-11-15T06:20:54.089 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-11-15T06:20:54.183 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-11-15T06:20:54.305 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-11-15T06:20:54.345 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-11-15T06:20:54.517 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-11-15T06:20:54.574 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-15T06:20:54.726 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-15T06:20:55.129 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2023-11-15T06:20:55.295 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2023-11-15T06:20:56.136 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-15T06:20:56.342 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2023-11-15T06:20:56.378 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-15T06:20:56.570 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2023-11-15T06:20:56.606 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2023-11-15T06:20:56.797 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2023-11-15T06:20:56.852 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2023-11-15T06:20:57.033 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2023-11-15T06:20:57.268 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2023-11-15T06:20:57.427 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2023-11-15T06:20:58.192 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-15T06:20:58.356 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-15T06:20:58.451 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2023-11-15T06:20:58.567 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2023-11-15T06:20:58.663 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-15T06:20:58.760 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-15T06:20:59.171 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-15T06:20:59.172 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2023-11-15T06:20:59.172 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2023-11-15T06:20:59.172 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2023-11-15T06:20:59.172 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-15T06:20:59.172 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2023-11-15T06:20:59.173 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-15T06:20:59.173 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2023-11-15T06:20:59.173 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-11-15T06:20:59.173 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-11-15T06:20:59.173 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2023-11-15T06:20:59.174 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-15T06:20:59.174 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2023-11-15T06:20:59.215 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-15T06:20:59.215 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2023-11-15T06:20:59.215 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2023-11-15T06:20:59.215 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2023-11-15T06:20:59.216 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-15T06:20:59.216 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2023-11-15T06:20:59.216 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-15T06:20:59.216 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2023-11-15T06:20:59.216 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-11-15T06:20:59.217 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-11-15T06:20:59.217 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2023-11-15T06:20:59.217 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-15T06:20:59.217 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2023-11-15T06:21:00.029 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-11-15T06:21:00.029 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:21:00.029 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2023-11-15T06:21:00.029 INFO:teuthology.orchestra.run.smithi062.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T06:21:00.030 INFO:teuthology.orchestra.run.smithi062.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T06:21:00.030 INFO:teuthology.orchestra.run.smithi062.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2023-11-15T06:21:00.030 INFO:teuthology.orchestra.run.smithi062.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-11-15T06:21:00.030 INFO:teuthology.orchestra.run.smithi062.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T06:21:00.030 INFO:teuthology.orchestra.run.smithi062.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2023-11-15T06:21:00.031 INFO:teuthology.orchestra.run.smithi062.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2023-11-15T06:21:00.031 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-11-15T06:21:00.031 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-11-15T06:21:00.031 INFO:teuthology.orchestra.run.smithi062.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T06:21:00.031 INFO:teuthology.orchestra.run.smithi062.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2023-11-15T06:21:00.032 INFO:teuthology.orchestra.run.smithi062.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2023-11-15T06:21:00.032 INFO:teuthology.orchestra.run.smithi062.stdout: yajl-2.1.0-12.el8.x86_64 2023-11-15T06:21:00.032 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:21:00.032 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2023-11-15T06:21:00.228 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-11-15T06:21:00.228 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:21:00.228 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2023-11-15T06:21:00.228 INFO:teuthology.orchestra.run.smithi081.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T06:21:00.229 INFO:teuthology.orchestra.run.smithi081.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T06:21:00.229 INFO:teuthology.orchestra.run.smithi081.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2023-11-15T06:21:00.229 INFO:teuthology.orchestra.run.smithi081.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-11-15T06:21:00.229 INFO:teuthology.orchestra.run.smithi081.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T06:21:00.229 INFO:teuthology.orchestra.run.smithi081.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2023-11-15T06:21:00.230 INFO:teuthology.orchestra.run.smithi081.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2023-11-15T06:21:00.230 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-11-15T06:21:00.230 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-11-15T06:21:00.230 INFO:teuthology.orchestra.run.smithi081.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T06:21:00.230 INFO:teuthology.orchestra.run.smithi081.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2023-11-15T06:21:00.231 INFO:teuthology.orchestra.run.smithi081.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2023-11-15T06:21:00.231 INFO:teuthology.orchestra.run.smithi081.stdout: yajl-2.1.0-12.el8.x86_64 2023-11-15T06:21:00.231 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:21:00.231 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2023-11-15T06:21:00.265 DEBUG:teuthology.parallel:result is None 2023-11-15T06:21:00.546 DEBUG:teuthology.parallel:result is None 2023-11-15T06:21:00.547 INFO:teuthology.run_tasks:Running task pexec... 2023-11-15T06:21:00.558 INFO:teuthology.task.pexec:Executing custom commands... 2023-11-15T06:21:00.558 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi062.front.sepia.ceph.com 2023-11-15T06:21:00.559 DEBUG:teuthology.orchestra.run.smithi062:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-15T06:21:00.559 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi081.front.sepia.ceph.com 2023-11-15T06:21:00.560 DEBUG:teuthology.orchestra.run.smithi081:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-15T06:21:00.763 DEBUG:teuthology.parallel:result is None 2023-11-15T06:21:00.979 DEBUG:teuthology.parallel:result is None 2023-11-15T06:21:00.979 INFO:teuthology.run_tasks:Running task install... 2023-11-15T06:21:00.988 DEBUG:teuthology.task.install:project ceph 2023-11-15T06:21:00.989 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '620f25c1794f62207424fc9319427943fb8d1fae'}} 2023-11-15T06:21:00.989 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'flavor': 'default', 'sha1': '620f25c1794f62207424fc9319427943fb8d1fae'} 2023-11-15T06:21:00.989 INFO:teuthology.task.install:Using flavor: default 2023-11-15T06:21:00.995 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-11-15T06:21:00.996 INFO:teuthology.task.install:extra packages: [] 2023-11-15T06:21:00.996 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '620f25c1794f62207424fc9319427943fb8d1fae', 'tag': 'v16.2.4', 'wait_for_package': False} 2023-11-15T06:21:00.997 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T06:21:00.997 INFO:teuthology.packaging:ref: None 2023-11-15T06:21:00.997 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T06:21:00.998 INFO:teuthology.packaging:branch: None 2023-11-15T06:21:00.998 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T06:21:01.201 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T06:21:01.202 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T06:21:01.204 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '620f25c1794f62207424fc9319427943fb8d1fae', 'tag': 'v16.2.4', 'wait_for_package': False} 2023-11-15T06:21:01.205 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T06:21:01.205 INFO:teuthology.packaging:ref: None 2023-11-15T06:21:01.206 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T06:21:01.206 INFO:teuthology.packaging:branch: None 2023-11-15T06:21:01.206 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T06:21:01.411 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T06:21:01.412 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T06:21:01.571 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2023-11-15T06:21:01.571 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2023-11-15T06:21:01.624 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-15T06:21:01.624 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-11-15T06:21:01.625 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-15T06:21:01.644 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2023-11-15T06:21:01.645 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2023-11-15T06:21:01.656 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-11-15T06:21:01.657 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T06:21:01.657 INFO:teuthology.packaging:ref: None 2023-11-15T06:21:01.657 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T06:21:01.658 INFO:teuthology.packaging:branch: None 2023-11-15T06:21:01.658 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T06:21:01.659 DEBUG:teuthology.orchestra.run.smithi062:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-15T06:21:01.674 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-15T06:21:01.675 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2023-11-15T06:21:01.675 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-15T06:21:01.711 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-11-15T06:21:01.712 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T06:21:01.712 INFO:teuthology.packaging:ref: None 2023-11-15T06:21:01.712 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T06:21:01.712 INFO:teuthology.packaging:branch: None 2023-11-15T06:21:01.713 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T06:21:01.713 DEBUG:teuthology.orchestra.run.smithi081:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-15T06:21:01.727 DEBUG:teuthology.orchestra.run.smithi062:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-15T06:21:01.783 DEBUG:teuthology.orchestra.run.smithi081:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-15T06:21:01.805 DEBUG:teuthology.orchestra.run.smithi062:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-15T06:21:01.863 DEBUG:teuthology.orchestra.run.smithi081:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-15T06:21:01.873 INFO:teuthology.orchestra.run.smithi062.stdout:check_obsoletes = 1 2023-11-15T06:21:01.874 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum clean all 2023-11-15T06:21:01.931 INFO:teuthology.orchestra.run.smithi081.stdout:check_obsoletes = 1 2023-11-15T06:21:01.932 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum clean all 2023-11-15T06:21:02.163 INFO:teuthology.orchestra.run.smithi062.stdout:76 files removed 2023-11-15T06:21:02.195 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-radosgw 2023-11-15T06:21:02.196 INFO:teuthology.orchestra.run.smithi081.stdout:76 files removed 2023-11-15T06:21:02.222 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-radosgw 2023-11-15T06:21:02.698 INFO:teuthology.orchestra.run.smithi062.stdout:ceph packages for x86_64 974 kB/s | 74 kB 00:00 2023-11-15T06:21:02.704 INFO:teuthology.orchestra.run.smithi081.stdout:ceph packages for x86_64 961 kB/s | 74 kB 00:00 2023-11-15T06:21:02.815 INFO:teuthology.orchestra.run.smithi062.stdout:ceph noarch packages 196 kB/s | 15 kB 00:00 2023-11-15T06:21:02.822 INFO:teuthology.orchestra.run.smithi081.stdout:ceph noarch packages 191 kB/s | 15 kB 00:00 2023-11-15T06:21:02.924 INFO:teuthology.orchestra.run.smithi062.stdout:ceph source packages 22 kB/s | 1.7 kB 00:00 2023-11-15T06:21:02.939 INFO:teuthology.orchestra.run.smithi081.stdout:ceph source packages 20 kB/s | 1.7 kB 00:00 2023-11-15T06:21:04.974 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 8 - AppStream 17 MB/s | 34 MB 00:02 2023-11-15T06:21:05.345 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 8 - AppStream 14 MB/s | 34 MB 00:02 2023-11-15T06:21:13.991 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 8 - BaseOS 22 MB/s | 53 MB 00:02 2023-11-15T06:21:16.467 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 8 - BaseOS 11 MB/s | 53 MB 00:04 2023-11-15T06:21:21.707 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 8 - Extras 153 kB/s | 18 kB 00:00 2023-11-15T06:21:22.241 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 8 - Extras common packages 19 kB/s | 6.9 kB 00:00 2023-11-15T06:21:23.476 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 8 - PowerTools 6.1 MB/s | 6.6 MB 00:01 2023-11-15T06:21:24.633 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 8 - Extras 147 kB/s | 18 kB 00:00 2023-11-15T06:21:25.042 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 8 - Extras common packages 32 kB/s | 6.9 kB 00:00 2023-11-15T06:21:25.103 INFO:teuthology.orchestra.run.smithi081.stdout:Copr repo for python3-asyncssh owned by ceph 10 kB/s | 3.5 kB 00:00 2023-11-15T06:21:26.127 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 8 - PowerTools 7.2 MB/s | 6.6 MB 00:00 2023-11-15T06:21:26.472 INFO:teuthology.orchestra.run.smithi081.stdout:Extra Packages for Enterprise Linux 13 MB/s | 16 MB 00:01 2023-11-15T06:21:27.620 INFO:teuthology.orchestra.run.smithi062.stdout:Copr repo for python3-asyncssh owned by ceph 32 kB/s | 3.5 kB 00:00 2023-11-15T06:21:30.338 INFO:teuthology.orchestra.run.smithi062.stdout:Extra Packages for Enterprise Linux 6.2 MB/s | 16 MB 00:02 2023-11-15T06:21:30.804 INFO:teuthology.orchestra.run.smithi081.stdout:lab-extras 47 kB/s | 24 kB 00:00 2023-11-15T06:21:34.438 INFO:teuthology.orchestra.run.smithi062.stdout:lab-extras 302 kB/s | 24 kB 00:00 2023-11-15T06:21:35.323 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2023-11-15T06:21:35.325 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:21:35.326 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2023-11-15T06:21:35.326 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:21:35.326 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2023-11-15T06:21:35.327 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2023-11-15T06:21:35.327 INFO:teuthology.orchestra.run.smithi081.stdout:Upgrading: 2023-11-15T06:21:35.327 INFO:teuthology.orchestra.run.smithi081.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-11-15T06:21:35.327 INFO:teuthology.orchestra.run.smithi081.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2023-11-15T06:21:35.327 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2023-11-15T06:21:35.328 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2023-11-15T06:21:35.328 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2023-11-15T06:21:35.328 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2023-11-15T06:21:35.328 INFO:teuthology.orchestra.run.smithi081.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2023-11-15T06:21:35.328 INFO:teuthology.orchestra.run.smithi081.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2023-11-15T06:21:35.329 INFO:teuthology.orchestra.run.smithi081.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-15T06:21:35.329 INFO:teuthology.orchestra.run.smithi081.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-11-15T06:21:35.329 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2023-11-15T06:21:35.329 INFO:teuthology.orchestra.run.smithi081.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-15T06:21:35.329 INFO:teuthology.orchestra.run.smithi081.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2023-11-15T06:21:35.330 INFO:teuthology.orchestra.run.smithi081.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-11-15T06:21:35.331 INFO:teuthology.orchestra.run.smithi081.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2023-11-15T06:21:35.331 INFO:teuthology.orchestra.run.smithi081.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-11-15T06:21:35.331 INFO:teuthology.orchestra.run.smithi081.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2023-11-15T06:21:35.331 INFO:teuthology.orchestra.run.smithi081.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-15T06:21:35.331 INFO:teuthology.orchestra.run.smithi081.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-11-15T06:21:35.332 INFO:teuthology.orchestra.run.smithi081.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2023-11-15T06:21:35.332 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2023-11-15T06:21:35.332 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2023-11-15T06:21:35.332 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2023-11-15T06:21:35.332 INFO:teuthology.orchestra.run.smithi081.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-11-15T06:21:35.333 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2023-11-15T06:21:35.333 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2023-11-15T06:21:35.333 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2023-11-15T06:21:35.333 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:21:35.334 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2023-11-15T06:21:35.334 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:21:35.334 INFO:teuthology.orchestra.run.smithi081.stdout:Install 25 Packages 2023-11-15T06:21:35.334 INFO:teuthology.orchestra.run.smithi081.stdout:Upgrade 2 Packages 2023-11-15T06:21:35.334 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:21:35.335 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 59 M 2023-11-15T06:21:35.335 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2023-11-15T06:21:35.723 INFO:teuthology.orchestra.run.smithi081.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 37 MB/s | 6.0 MB 00:00 2023-11-15T06:21:35.748 INFO:teuthology.orchestra.run.smithi081.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 997 kB/s | 25 kB 00:00 2023-11-15T06:21:35.790 INFO:teuthology.orchestra.run.smithi081.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 19 MB/s | 801 kB 00:00 2023-11-15T06:21:35.824 INFO:teuthology.orchestra.run.smithi081.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2023-11-15T06:21:35.915 INFO:teuthology.orchestra.run.smithi081.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 44 MB/s | 4.1 MB 00:00 2023-11-15T06:21:35.982 INFO:teuthology.orchestra.run.smithi081.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 668 kB/s | 44 kB 00:00 2023-11-15T06:21:36.016 INFO:teuthology.orchestra.run.smithi081.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 2.3 MB/s | 80 kB 00:00 2023-11-15T06:21:36.041 INFO:teuthology.orchestra.run.smithi081.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.4 MB/s | 213 kB 00:00 2023-11-15T06:21:36.075 INFO:teuthology.orchestra.run.smithi081.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2023-11-15T06:21:36.233 INFO:teuthology.orchestra.run.smithi081.stdout:(10/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 19 MB/s | 13 MB 00:00 2023-11-15T06:21:36.267 INFO:teuthology.orchestra.run.smithi081.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 3.3 MB/s | 114 kB 00:00 2023-11-15T06:21:36.301 INFO:teuthology.orchestra.run.smithi081.stdout:(12/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 1.6 MB/s | 368 kB 00:00 2023-11-15T06:21:36.593 INFO:teuthology.orchestra.run.smithi081.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 23 MB/s | 24 MB 00:01 2023-11-15T06:21:36.618 INFO:teuthology.orchestra.run.smithi081.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 414 kB/s | 145 kB 00:00 2023-11-15T06:21:36.668 INFO:teuthology.orchestra.run.smithi081.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 878 kB/s | 44 kB 00:00 2023-11-15T06:21:36.693 INFO:teuthology.orchestra.run.smithi081.stdout:(16/27): librdkafka-0.11.4-3.el8.x86_64.rpm 901 kB/s | 354 kB 00:00 2023-11-15T06:21:36.810 INFO:teuthology.orchestra.run.smithi081.stdout:(17/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.2 MB/s | 259 kB 00:00 2023-11-15T06:21:37.086 INFO:teuthology.orchestra.run.smithi081.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 101 kB/s | 42 kB 00:00 2023-11-15T06:21:37.111 INFO:teuthology.orchestra.run.smithi081.stdout:(19/27): librabbitmq-0.9.0-4.el8.x86_64.rpm 112 kB/s | 47 kB 00:00 2023-11-15T06:21:37.136 INFO:teuthology.orchestra.run.smithi081.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 246 kB/s | 80 kB 00:00 2023-11-15T06:21:37.728 INFO:teuthology.orchestra.run.smithi081.stdout:(21/27): gperftools-libs-2.7-9.el8.x86_64.rpm 496 kB/s | 306 kB 00:00 2023-11-15T06:21:38.003 INFO:teuthology.orchestra.run.smithi081.stdout:(22/27): liboath-2.6.2-3.el8.x86_64.rpm 214 kB/s | 59 kB 00:00 2023-11-15T06:21:38.212 INFO:teuthology.orchestra.run.smithi081.stdout:(23/27): fmt-6.2.1-1.el8.x86_64.rpm 103 kB/s | 116 kB 00:01 2023-11-15T06:21:38.238 INFO:teuthology.orchestra.run.smithi081.stdout:(24/27): libunwind-1.3.1-3.el8.x86_64.rpm 321 kB/s | 75 kB 00:00 2023-11-15T06:21:38.321 INFO:teuthology.orchestra.run.smithi081.stdout:(25/27): librbd1-16.2.4-0.el8.x86_64.rpm 49 MB/s | 4.0 MB 00:00 2023-11-15T06:21:38.346 INFO:teuthology.orchestra.run.smithi081.stdout:(26/27): leveldb-1.22-1.el8.x86_64.rpm 150 kB/s | 181 kB 00:01 2023-11-15T06:21:38.448 INFO:teuthology.orchestra.run.smithi081.stdout:(27/27): librados2-16.2.4-0.el8.x86_64.rpm 16 MB/s | 3.7 MB 00:00 2023-11-15T06:21:38.453 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:21:38.453 INFO:teuthology.orchestra.run.smithi081.stdout:Total 19 MB/s | 59 MB 00:03 2023-11-15T06:21:38.519 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2023-11-15T06:21:38.570 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2023-11-15T06:21:38.570 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2023-11-15T06:21:38.911 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2023-11-15T06:21:38.914 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2023-11-15T06:21:39.244 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2023-11-15T06:21:39.247 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:21:39.247 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2023-11-15T06:21:39.247 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:21:39.248 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2023-11-15T06:21:39.248 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2023-11-15T06:21:39.248 INFO:teuthology.orchestra.run.smithi062.stdout:Upgrading: 2023-11-15T06:21:39.248 INFO:teuthology.orchestra.run.smithi062.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-11-15T06:21:39.248 INFO:teuthology.orchestra.run.smithi062.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2023-11-15T06:21:39.249 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2023-11-15T06:21:39.249 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2023-11-15T06:21:39.249 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2023-11-15T06:21:39.249 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2023-11-15T06:21:39.249 INFO:teuthology.orchestra.run.smithi062.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2023-11-15T06:21:39.250 INFO:teuthology.orchestra.run.smithi062.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2023-11-15T06:21:39.250 INFO:teuthology.orchestra.run.smithi062.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-15T06:21:39.250 INFO:teuthology.orchestra.run.smithi062.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-11-15T06:21:39.250 INFO:teuthology.orchestra.run.smithi062.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2023-11-15T06:21:39.250 INFO:teuthology.orchestra.run.smithi062.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-15T06:21:39.251 INFO:teuthology.orchestra.run.smithi062.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2023-11-15T06:21:39.251 INFO:teuthology.orchestra.run.smithi062.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-11-15T06:21:39.251 INFO:teuthology.orchestra.run.smithi062.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2023-11-15T06:21:39.251 INFO:teuthology.orchestra.run.smithi062.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-11-15T06:21:39.251 INFO:teuthology.orchestra.run.smithi062.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2023-11-15T06:21:39.252 INFO:teuthology.orchestra.run.smithi062.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-15T06:21:39.252 INFO:teuthology.orchestra.run.smithi062.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-11-15T06:21:39.252 INFO:teuthology.orchestra.run.smithi062.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2023-11-15T06:21:39.253 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2023-11-15T06:21:39.253 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2023-11-15T06:21:39.253 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2023-11-15T06:21:39.253 INFO:teuthology.orchestra.run.smithi062.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-11-15T06:21:39.253 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2023-11-15T06:21:39.254 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2023-11-15T06:21:39.254 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2023-11-15T06:21:39.254 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:21:39.254 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2023-11-15T06:21:39.254 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:21:39.255 INFO:teuthology.orchestra.run.smithi062.stdout:Install 25 Packages 2023-11-15T06:21:39.255 INFO:teuthology.orchestra.run.smithi062.stdout:Upgrade 2 Packages 2023-11-15T06:21:39.255 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:21:39.256 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 59 M 2023-11-15T06:21:39.256 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2023-11-15T06:21:39.648 INFO:teuthology.orchestra.run.smithi062.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 44 MB/s | 6.0 MB 00:00 2023-11-15T06:21:39.664 INFO:teuthology.orchestra.run.smithi062.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 1.5 MB/s | 25 kB 00:00 2023-11-15T06:21:39.707 INFO:teuthology.orchestra.run.smithi062.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 19 MB/s | 801 kB 00:00 2023-11-15T06:21:39.740 INFO:teuthology.orchestra.run.smithi062.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2023-11-15T06:21:39.849 INFO:teuthology.orchestra.run.smithi062.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 38 MB/s | 4.1 MB 00:00 2023-11-15T06:21:39.866 INFO:teuthology.orchestra.run.smithi062.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 2.6 MB/s | 44 kB 00:00 2023-11-15T06:21:39.891 INFO:teuthology.orchestra.run.smithi062.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2023-11-15T06:21:39.917 INFO:teuthology.orchestra.run.smithi062.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.3 MB/s | 213 kB 00:00 2023-11-15T06:21:39.950 INFO:teuthology.orchestra.run.smithi062.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2023-11-15T06:21:39.975 INFO:teuthology.orchestra.run.smithi062.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 15 MB/s | 368 kB 00:00 2023-11-15T06:21:40.002 INFO:teuthology.orchestra.run.smithi062.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2023-11-15T06:21:40.105 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2023-11-15T06:21:40.160 INFO:teuthology.orchestra.run.smithi062.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 20 MB/s | 13 MB 00:00 2023-11-15T06:21:40.185 INFO:teuthology.orchestra.run.smithi062.stdout:(13/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 791 kB/s | 145 kB 00:00 2023-11-15T06:21:40.301 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2023-11-15T06:21:40.468 INFO:teuthology.orchestra.run.smithi062.stdout:(14/27): ceph-common-16.2.4-0.el8.x86_64.rpm 25 MB/s | 24 MB 00:00 2023-11-15T06:21:40.494 INFO:teuthology.orchestra.run.smithi062.stdout:(15/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 841 kB/s | 259 kB 00:00 2023-11-15T06:21:40.546 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2023-11-15T06:21:40.611 INFO:teuthology.orchestra.run.smithi062.stdout:(16/27): librdkafka-0.11.4-3.el8.x86_64.rpm 785 kB/s | 354 kB 00:00 2023-11-15T06:21:40.653 INFO:teuthology.orchestra.run.smithi062.stdout:(17/27): python3-prettytable-0.7.2-14.el8.noarc 239 kB/s | 44 kB 00:00 2023-11-15T06:21:40.828 INFO:teuthology.orchestra.run.smithi062.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 126 kB/s | 42 kB 00:00 2023-11-15T06:21:40.945 INFO:teuthology.orchestra.run.smithi062.stdout:(19/27): librabbitmq-0.9.0-4.el8.x86_64.rpm 140 kB/s | 47 kB 00:00 2023-11-15T06:21:40.979 INFO:teuthology.orchestra.run.smithi062.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 246 kB/s | 80 kB 00:00 2023-11-15T06:21:41.025 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-11-15T06:21:41.438 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-11-15T06:21:41.667 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T06:21:41.795 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T06:21:42.024 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2023-11-15T06:21:42.183 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T06:21:42.251 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T06:21:42.482 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/29 2023-11-15T06:21:42.683 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/29 2023-11-15T06:21:42.729 INFO:teuthology.orchestra.run.smithi062.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 61 kB/s | 116 kB 00:01 2023-11-15T06:21:42.754 INFO:teuthology.orchestra.run.smithi062.stdout:(22/27): leveldb-1.22-1.el8.x86_64.rpm 102 kB/s | 181 kB 00:01 2023-11-15T06:21:42.854 INFO:teuthology.orchestra.run.smithi062.stdout:(23/27): gperftools-libs-2.7-9.el8.x86_64.rpm 160 kB/s | 306 kB 00:01 2023-11-15T06:21:42.888 INFO:teuthology.orchestra.run.smithi062.stdout:(24/27): liboath-2.6.2-3.el8.x86_64.rpm 370 kB/s | 59 kB 00:00 2023-11-15T06:21:42.905 INFO:teuthology.orchestra.run.smithi062.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 499 kB/s | 75 kB 00:00 2023-11-15T06:21:43.005 INFO:teuthology.orchestra.run.smithi062.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 25 MB/s | 3.7 MB 00:00 2023-11-15T06:21:43.063 INFO:teuthology.orchestra.run.smithi062.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 23 MB/s | 4.0 MB 00:00 2023-11-15T06:21:43.065 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:21:43.066 INFO:teuthology.orchestra.run.smithi062.stdout:Total 15 MB/s | 59 MB 00:03 2023-11-15T06:21:43.109 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/29 2023-11-15T06:21:43.135 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2023-11-15T06:21:43.188 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2023-11-15T06:21:43.188 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2023-11-15T06:21:43.308 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T06:21:43.368 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T06:21:43.572 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2023-11-15T06:21:43.643 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2023-11-15T06:21:43.646 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2023-11-15T06:21:43.791 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2023-11-15T06:21:43.858 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2023-11-15T06:21:44.056 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2023-11-15T06:21:44.259 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2023-11-15T06:21:44.441 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2023-11-15T06:21:44.535 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2023-11-15T06:21:44.683 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-11-15T06:21:44.784 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2023-11-15T06:21:44.997 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2023-11-15T06:21:45.108 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-11-15T06:21:45.236 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2023-11-15T06:21:45.333 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-11-15T06:21:45.396 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-11-15T06:21:45.429 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-11-15T06:21:45.645 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2023-11-15T06:21:45.810 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-11-15T06:21:45.880 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2023-11-15T06:21:45.971 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T06:21:46.066 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T06:21:46.074 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-11-15T06:21:46.186 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-11-15T06:21:46.310 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2023-11-15T06:21:46.384 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2023-11-15T06:21:46.454 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T06:21:46.519 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T06:21:46.583 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2023-11-15T06:21:46.679 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/29 2023-11-15T06:21:46.783 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2023-11-15T06:21:46.846 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/29 2023-11-15T06:21:47.008 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2023-11-15T06:21:47.291 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/29 2023-11-15T06:21:47.429 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T06:21:47.509 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T06:21:47.668 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2023-11-15T06:21:47.804 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2023-11-15T06:21:47.858 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2023-11-15T06:21:48.002 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2023-11-15T06:21:48.155 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2023-11-15T06:21:48.295 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2023-11-15T06:21:48.360 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2023-11-15T06:21:48.537 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-11-15T06:21:48.999 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-11-15T06:21:49.154 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-11-15T06:21:49.259 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-11-15T06:21:49.425 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2023-11-15T06:21:49.608 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2023-11-15T06:21:49.663 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T06:21:49.778 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-11-15T06:21:49.836 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-11-15T06:21:49.891 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T06:21:49.998 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2023-11-15T06:21:50.154 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2023-11-15T06:21:50.314 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2023-11-15T06:21:50.487 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2023-11-15T06:21:50.602 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T06:21:50.896 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-11-15T06:21:51.011 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-11-15T06:21:51.174 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T06:21:53.186 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T06:21:53.437 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T06:21:54.179 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T06:21:54.661 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-11-15T06:21:54.774 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-11-15T06:21:54.929 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T06:22:09.533 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T06:22:09.533 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /sys 2023-11-15T06:22:09.533 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /proc 2023-11-15T06:22:09.534 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /dev 2023-11-15T06:22:09.534 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /run 2023-11-15T06:22:09.534 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /mnt 2023-11-15T06:22:09.534 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /var/tmp 2023-11-15T06:22:09.534 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /home 2023-11-15T06:22:09.535 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /tmp 2023-11-15T06:22:09.535 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /dev 2023-11-15T06:22:09.535 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:22:10.927 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-11-15T06:22:11.001 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-11-15T06:22:11.002 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-11-15T06:22:11.199 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-11-15T06:22:11.199 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T06:22:11.886 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T06:22:11.886 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2023-11-15T06:22:11.886 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2023-11-15T06:22:11.886 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2023-11-15T06:22:11.887 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T06:22:11.887 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2023-11-15T06:22:11.887 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T06:22:11.887 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2023-11-15T06:22:11.887 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2023-11-15T06:22:11.888 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T06:22:11.888 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2023-11-15T06:22:11.888 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2023-11-15T06:22:11.888 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2023-11-15T06:22:11.888 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2023-11-15T06:22:11.889 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2023-11-15T06:22:11.889 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 15/29 2023-11-15T06:22:11.889 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2023-11-15T06:22:11.889 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2023-11-15T06:22:11.889 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2023-11-15T06:22:11.889 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 19/29 2023-11-15T06:22:11.890 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2023-11-15T06:22:11.890 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2023-11-15T06:22:11.890 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2023-11-15T06:22:11.890 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2023-11-15T06:22:11.890 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2023-11-15T06:22:11.891 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2023-11-15T06:22:11.891 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T06:22:11.891 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2023-11-15T06:22:11.891 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2023-11-15T06:22:13.133 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T06:22:13.134 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:22:13.134 INFO:teuthology.orchestra.run.smithi081.stdout:Upgraded: 2023-11-15T06:22:13.134 INFO:teuthology.orchestra.run.smithi081.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2023-11-15T06:22:13.134 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2023-11-15T06:22:13.135 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2023-11-15T06:22:13.135 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2023-11-15T06:22:13.135 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2023-11-15T06:22:13.135 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2023-11-15T06:22:13.135 INFO:teuthology.orchestra.run.smithi081.stdout: daxctl-libs-71.1-7.el8.x86_64 2023-11-15T06:22:13.136 INFO:teuthology.orchestra.run.smithi081.stdout: fmt-6.2.1-1.el8.x86_64 2023-11-15T06:22:13.136 INFO:teuthology.orchestra.run.smithi081.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-15T06:22:13.136 INFO:teuthology.orchestra.run.smithi081.stdout: leveldb-1.22-1.el8.x86_64 2023-11-15T06:22:13.136 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2023-11-15T06:22:13.136 INFO:teuthology.orchestra.run.smithi081.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-15T06:22:13.137 INFO:teuthology.orchestra.run.smithi081.stdout: libpmemobj-1.6.1-1.el8.x86_64 2023-11-15T06:22:13.137 INFO:teuthology.orchestra.run.smithi081.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-11-15T06:22:13.137 INFO:teuthology.orchestra.run.smithi081.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2023-11-15T06:22:13.137 INFO:teuthology.orchestra.run.smithi081.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-15T06:22:13.137 INFO:teuthology.orchestra.run.smithi081.stdout: librgw2-2:16.2.4-0.el8.x86_64 2023-11-15T06:22:13.138 INFO:teuthology.orchestra.run.smithi081.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-15T06:22:13.138 INFO:teuthology.orchestra.run.smithi081.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-15T06:22:13.138 INFO:teuthology.orchestra.run.smithi081.stdout: ndctl-libs-71.1-7.el8.x86_64 2023-11-15T06:22:13.138 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2023-11-15T06:22:13.138 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2023-11-15T06:22:13.139 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2023-11-15T06:22:13.139 INFO:teuthology.orchestra.run.smithi081.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-15T06:22:13.139 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2023-11-15T06:22:13.139 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2023-11-15T06:22:13.139 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2023-11-15T06:22:13.140 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:22:13.140 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2023-11-15T06:22:13.467 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-test 2023-11-15T06:22:14.029 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T06:22:14.030 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /sys 2023-11-15T06:22:14.030 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /proc 2023-11-15T06:22:14.030 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /dev 2023-11-15T06:22:14.031 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /run 2023-11-15T06:22:14.031 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /mnt 2023-11-15T06:22:14.031 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /var/tmp 2023-11-15T06:22:14.031 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /home 2023-11-15T06:22:14.031 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /tmp 2023-11-15T06:22:14.032 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /dev 2023-11-15T06:22:14.034 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:22:14.313 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:00:44 ago on Wed 15 Nov 2023 06:21:30 AM UTC. 2023-11-15T06:22:15.458 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-11-15T06:22:15.542 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-11-15T06:22:15.543 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-11-15T06:22:15.761 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-11-15T06:22:15.762 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T06:22:15.971 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2023-11-15T06:22:15.972 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:22:15.972 INFO:teuthology.orchestra.run.smithi081.stdout: Package Architecture Version Repository Size 2023-11-15T06:22:15.972 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:22:15.973 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2023-11-15T06:22:15.973 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2023-11-15T06:22:15.973 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2023-11-15T06:22:15.973 INFO:teuthology.orchestra.run.smithi081.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-11-15T06:22:15.973 INFO:teuthology.orchestra.run.smithi081.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2023-11-15T06:22:15.974 INFO:teuthology.orchestra.run.smithi081.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-11-15T06:22:15.974 INFO:teuthology.orchestra.run.smithi081.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-11-15T06:22:15.974 INFO:teuthology.orchestra.run.smithi081.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-11-15T06:22:15.974 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:22:15.975 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2023-11-15T06:22:15.975 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:22:15.975 INFO:teuthology.orchestra.run.smithi081.stdout:Install 6 Packages 2023-11-15T06:22:15.975 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:22:15.976 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 53 M 2023-11-15T06:22:15.977 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 229 M 2023-11-15T06:22:15.977 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2023-11-15T06:22:16.076 INFO:teuthology.orchestra.run.smithi081.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 2.6 MB/s | 169 kB 00:00 2023-11-15T06:22:16.269 INFO:teuthology.orchestra.run.smithi081.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 793 kB/s | 203 kB 00:00 2023-11-15T06:22:16.302 INFO:teuthology.orchestra.run.smithi081.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 832 kB/s | 187 kB 00:00 2023-11-15T06:22:16.361 INFO:teuthology.orchestra.run.smithi081.stdout:(4/6): socat-1.7.4.1-1.el8.x86_64.rpm 3.5 MB/s | 323 kB 00:00 2023-11-15T06:22:16.394 INFO:teuthology.orchestra.run.smithi081.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 767 kB/s | 71 kB 00:00 2023-11-15T06:22:16.441 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T06:22:16.441 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2023-11-15T06:22:16.442 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2023-11-15T06:22:16.442 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2023-11-15T06:22:16.442 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T06:22:16.442 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2023-11-15T06:22:16.443 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T06:22:16.443 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2023-11-15T06:22:16.443 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2023-11-15T06:22:16.444 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T06:22:16.444 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2023-11-15T06:22:16.444 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2023-11-15T06:22:16.444 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2023-11-15T06:22:16.444 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2023-11-15T06:22:16.444 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2023-11-15T06:22:16.445 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 15/29 2023-11-15T06:22:16.445 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2023-11-15T06:22:16.445 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2023-11-15T06:22:16.445 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2023-11-15T06:22:16.446 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 19/29 2023-11-15T06:22:16.446 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2023-11-15T06:22:16.446 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2023-11-15T06:22:16.446 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2023-11-15T06:22:16.446 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2023-11-15T06:22:16.447 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2023-11-15T06:22:16.447 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2023-11-15T06:22:16.447 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T06:22:16.447 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2023-11-15T06:22:16.447 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2023-11-15T06:22:16.878 INFO:teuthology.orchestra.run.smithi081.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 60 MB/s | 52 MB 00:00 2023-11-15T06:22:16.880 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:22:16.880 INFO:teuthology.orchestra.run.smithi081.stdout:Total 58 MB/s | 53 MB 00:00 2023-11-15T06:22:16.922 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2023-11-15T06:22:16.943 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2023-11-15T06:22:16.944 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2023-11-15T06:22:17.250 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2023-11-15T06:22:17.253 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2023-11-15T06:22:17.737 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T06:22:17.738 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:22:17.738 INFO:teuthology.orchestra.run.smithi062.stdout:Upgraded: 2023-11-15T06:22:17.738 INFO:teuthology.orchestra.run.smithi062.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2023-11-15T06:22:17.738 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2023-11-15T06:22:17.739 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2023-11-15T06:22:17.739 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2023-11-15T06:22:17.739 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2023-11-15T06:22:17.739 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2023-11-15T06:22:17.739 INFO:teuthology.orchestra.run.smithi062.stdout: daxctl-libs-71.1-7.el8.x86_64 2023-11-15T06:22:17.739 INFO:teuthology.orchestra.run.smithi062.stdout: fmt-6.2.1-1.el8.x86_64 2023-11-15T06:22:17.740 INFO:teuthology.orchestra.run.smithi062.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-15T06:22:17.740 INFO:teuthology.orchestra.run.smithi062.stdout: leveldb-1.22-1.el8.x86_64 2023-11-15T06:22:17.740 INFO:teuthology.orchestra.run.smithi062.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2023-11-15T06:22:17.740 INFO:teuthology.orchestra.run.smithi062.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-15T06:22:17.740 INFO:teuthology.orchestra.run.smithi062.stdout: libpmemobj-1.6.1-1.el8.x86_64 2023-11-15T06:22:17.741 INFO:teuthology.orchestra.run.smithi062.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-11-15T06:22:17.741 INFO:teuthology.orchestra.run.smithi062.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2023-11-15T06:22:17.741 INFO:teuthology.orchestra.run.smithi062.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-15T06:22:17.741 INFO:teuthology.orchestra.run.smithi062.stdout: librgw2-2:16.2.4-0.el8.x86_64 2023-11-15T06:22:17.741 INFO:teuthology.orchestra.run.smithi062.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-15T06:22:17.742 INFO:teuthology.orchestra.run.smithi062.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-15T06:22:17.742 INFO:teuthology.orchestra.run.smithi062.stdout: ndctl-libs-71.1-7.el8.x86_64 2023-11-15T06:22:17.742 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2023-11-15T06:22:17.742 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2023-11-15T06:22:17.742 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2023-11-15T06:22:17.743 INFO:teuthology.orchestra.run.smithi062.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-15T06:22:17.743 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2023-11-15T06:22:17.743 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2023-11-15T06:22:17.743 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2023-11-15T06:22:17.743 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:22:17.744 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2023-11-15T06:22:17.756 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2023-11-15T06:22:17.948 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-11-15T06:22:17.957 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-test 2023-11-15T06:22:18.131 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-11-15T06:22:18.320 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-15T06:22:18.391 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-15T06:22:18.586 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2023-11-15T06:22:18.737 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-11-15T06:22:18.853 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:00:44 ago on Wed 15 Nov 2023 06:21:34 AM UTC. 2023-11-15T06:22:20.646 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2023-11-15T06:22:20.647 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:22:20.647 INFO:teuthology.orchestra.run.smithi062.stdout: Package Architecture Version Repository Size 2023-11-15T06:22:20.648 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:22:20.648 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2023-11-15T06:22:20.648 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2023-11-15T06:22:20.648 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2023-11-15T06:22:20.648 INFO:teuthology.orchestra.run.smithi062.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-11-15T06:22:20.649 INFO:teuthology.orchestra.run.smithi062.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2023-11-15T06:22:20.649 INFO:teuthology.orchestra.run.smithi062.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-11-15T06:22:20.649 INFO:teuthology.orchestra.run.smithi062.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-11-15T06:22:20.649 INFO:teuthology.orchestra.run.smithi062.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-11-15T06:22:20.649 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:22:20.650 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2023-11-15T06:22:20.650 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:22:20.650 INFO:teuthology.orchestra.run.smithi062.stdout:Install 6 Packages 2023-11-15T06:22:20.650 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:22:20.651 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 53 M 2023-11-15T06:22:20.651 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 229 M 2023-11-15T06:22:20.651 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2023-11-15T06:22:20.933 INFO:teuthology.orchestra.run.smithi062.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 1.8 MB/s | 169 kB 00:00 2023-11-15T06:22:21.041 INFO:teuthology.orchestra.run.smithi062.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 1.0 MB/s | 203 kB 00:00 2023-11-15T06:22:21.100 INFO:teuthology.orchestra.run.smithi062.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2023-11-15T06:22:21.125 INFO:teuthology.orchestra.run.smithi062.stdout:(4/6): socat-1.7.4.1-1.el8.x86_64.rpm 3.8 MB/s | 323 kB 00:00 2023-11-15T06:22:21.159 INFO:teuthology.orchestra.run.smithi062.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2023-11-15T06:22:21.654 INFO:teuthology.orchestra.run.smithi062.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 64 MB/s | 52 MB 00:00 2023-11-15T06:22:21.654 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:22:21.654 INFO:teuthology.orchestra.run.smithi062.stdout:Total 53 MB/s | 53 MB 00:01 2023-11-15T06:22:21.698 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2023-11-15T06:22:21.721 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2023-11-15T06:22:21.722 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2023-11-15T06:22:22.049 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2023-11-15T06:22:22.052 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2023-11-15T06:22:22.552 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2023-11-15T06:22:22.771 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-11-15T06:22:22.928 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-11-15T06:22:23.091 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-15T06:22:23.173 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-15T06:22:23.341 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2023-11-15T06:22:23.491 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-11-15T06:22:23.614 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-11-15T06:22:23.885 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-11-15T06:22:24.204 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-11-15T06:22:24.204 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2023-11-15T06:22:24.204 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2023-11-15T06:22:24.204 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2023-11-15T06:22:24.205 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-11-15T06:22:24.205 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-11-15T06:22:24.952 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-11-15T06:22:24.952 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:22:24.952 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2023-11-15T06:22:24.953 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2023-11-15T06:22:24.953 INFO:teuthology.orchestra.run.smithi081.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2023-11-15T06:22:24.953 INFO:teuthology.orchestra.run.smithi081.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2023-11-15T06:22:24.953 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:22:24.953 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2023-11-15T06:22:25.241 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph 2023-11-15T06:22:26.094 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:00:56 ago on Wed 15 Nov 2023 06:21:30 AM UTC. 2023-11-15T06:22:27.764 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2023-11-15T06:22:27.767 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:22:27.768 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2023-11-15T06:22:27.768 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:22:27.768 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2023-11-15T06:22:27.768 INFO:teuthology.orchestra.run.smithi081.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2023-11-15T06:22:27.768 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2023-11-15T06:22:27.769 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2023-11-15T06:22:27.769 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2023-11-15T06:22:27.769 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2023-11-15T06:22:27.769 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2023-11-15T06:22:27.769 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2023-11-15T06:22:27.770 INFO:teuthology.orchestra.run.smithi081.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-15T06:22:27.770 INFO:teuthology.orchestra.run.smithi081.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-15T06:22:27.770 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-11-15T06:22:27.770 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-15T06:22:27.771 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-15T06:22:27.771 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-11-15T06:22:27.771 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-15T06:22:27.771 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-15T06:22:27.771 INFO:teuthology.orchestra.run.smithi081.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-15T06:22:27.772 INFO:teuthology.orchestra.run.smithi081.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-11-15T06:22:27.772 INFO:teuthology.orchestra.run.smithi081.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-11-15T06:22:27.772 INFO:teuthology.orchestra.run.smithi081.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-15T06:22:27.772 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-15T06:22:27.772 INFO:teuthology.orchestra.run.smithi081.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-15T06:22:27.773 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-11-15T06:22:27.773 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-11-15T06:22:27.773 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-11-15T06:22:27.773 INFO:teuthology.orchestra.run.smithi081.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-15T06:22:27.773 INFO:teuthology.orchestra.run.smithi081.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-15T06:22:27.774 INFO:teuthology.orchestra.run.smithi081.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-15T06:22:27.774 INFO:teuthology.orchestra.run.smithi081.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-15T06:22:27.774 INFO:teuthology.orchestra.run.smithi081.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-15T06:22:27.774 INFO:teuthology.orchestra.run.smithi081.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-15T06:22:27.774 INFO:teuthology.orchestra.run.smithi081.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-15T06:22:27.775 INFO:teuthology.orchestra.run.smithi081.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-11-15T06:22:27.775 INFO:teuthology.orchestra.run.smithi081.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-15T06:22:27.775 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:22:27.775 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2023-11-15T06:22:27.775 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:22:27.776 INFO:teuthology.orchestra.run.smithi081.stdout:Install 32 Packages 2023-11-15T06:22:27.776 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:22:27.777 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 31 M 2023-11-15T06:22:27.777 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 111 M 2023-11-15T06:22:27.777 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2023-11-15T06:22:28.138 INFO:teuthology.orchestra.run.smithi081.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 87 kB/s | 6.3 kB 00:00 2023-11-15T06:22:28.197 INFO:teuthology.orchestra.run.smithi081.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2023-11-15T06:22:28.247 INFO:teuthology.orchestra.run.smithi081.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 13 MB/s | 2.4 MB 00:00 2023-11-15T06:22:28.314 INFO:teuthology.orchestra.run.smithi081.stdout:(4/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 28 MB/s | 5.0 MB 00:00 2023-11-15T06:22:28.340 INFO:teuthology.orchestra.run.smithi081.stdout:(5/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 2.3 MB/s | 220 kB 00:00 2023-11-15T06:22:28.573 INFO:teuthology.orchestra.run.smithi081.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 48 MB/s | 18 MB 00:00 2023-11-15T06:22:28.590 INFO:teuthology.orchestra.run.smithi081.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 155 kB/s | 39 kB 00:00 2023-11-15T06:22:28.616 INFO:teuthology.orchestra.run.smithi081.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 522 kB/s | 157 kB 00:00 2023-11-15T06:22:28.663 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-11-15T06:22:28.674 INFO:teuthology.orchestra.run.smithi081.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 683 kB/s | 57 kB 00:00 2023-11-15T06:22:28.700 INFO:teuthology.orchestra.run.smithi081.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 5.3 MB/s | 457 kB 00:00 2023-11-15T06:22:28.758 INFO:teuthology.orchestra.run.smithi081.stdout:(11/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 561 kB/s | 103 kB 00:00 2023-11-15T06:22:29.006 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-11-15T06:22:29.150 INFO:teuthology.orchestra.run.smithi081.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 278 kB/s | 109 kB 00:00 2023-11-15T06:22:29.205 INFO:teuthology.orchestra.run.smithi081.stdout:(13/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 469 kB/s | 249 kB 00:00 2023-11-15T06:22:29.267 INFO:teuthology.orchestra.run.smithi081.stdout:(14/32): python3-cryptography-3.2.1-6.el8.x86_6 1.1 MB/s | 641 kB 00:00 2023-11-15T06:22:29.293 INFO:teuthology.orchestra.run.smithi081.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 306 kB/s | 44 kB 00:00 2023-11-15T06:22:29.308 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-11-15T06:22:29.308 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2023-11-15T06:22:29.308 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2023-11-15T06:22:29.308 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2023-11-15T06:22:29.309 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-11-15T06:22:29.309 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-11-15T06:22:29.318 INFO:teuthology.orchestra.run.smithi081.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.6 MB/s | 185 kB 00:00 2023-11-15T06:22:29.352 INFO:teuthology.orchestra.run.smithi081.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 2.0 MB/s | 173 kB 00:00 2023-11-15T06:22:29.377 INFO:teuthology.orchestra.run.smithi081.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 189 kB/s | 11 kB 00:00 2023-11-15T06:22:29.402 INFO:teuthology.orchestra.run.smithi081.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 351 kB/s | 18 kB 00:00 2023-11-15T06:22:29.428 INFO:teuthology.orchestra.run.smithi081.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 971 kB/s | 49 kB 00:00 2023-11-15T06:22:29.479 INFO:teuthology.orchestra.run.smithi081.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 786 kB/s | 59 kB 00:00 2023-11-15T06:22:29.511 INFO:teuthology.orchestra.run.smithi081.stdout:(22/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.7 MB/s | 384 kB 00:00 2023-11-15T06:22:29.537 INFO:teuthology.orchestra.run.smithi081.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 265 kB/s | 16 kB 00:00 2023-11-15T06:22:29.562 INFO:teuthology.orchestra.run.smithi081.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 373 kB/s | 19 kB 00:00 2023-11-15T06:22:29.588 INFO:teuthology.orchestra.run.smithi081.stdout:(25/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.7 MB/s | 283 kB 00:00 2023-11-15T06:22:29.613 INFO:teuthology.orchestra.run.smithi081.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 316 kB/s | 24 kB 00:00 2023-11-15T06:22:29.638 INFO:teuthology.orchestra.run.smithi081.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 388 kB/s | 29 kB 00:00 2023-11-15T06:22:29.663 INFO:teuthology.orchestra.run.smithi081.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 365 kB/s | 27 kB 00:00 2023-11-15T06:22:29.688 INFO:teuthology.orchestra.run.smithi081.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 3.3 MB/s | 255 kB 00:00 2023-11-15T06:22:29.714 INFO:teuthology.orchestra.run.smithi081.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 3.2 MB/s | 251 kB 00:00 2023-11-15T06:22:29.772 INFO:teuthology.orchestra.run.smithi081.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 274 kB/s | 23 kB 00:00 2023-11-15T06:22:29.797 INFO:teuthology.orchestra.run.smithi081.stdout:(32/32): python3-webtest-2.0.33-1.el8.noarch.rp 638 kB/s | 85 kB 00:00 2023-11-15T06:22:29.799 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:22:29.799 INFO:teuthology.orchestra.run.smithi081.stdout:Total 15 MB/s | 31 MB 00:02 2023-11-15T06:22:29.881 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2023-11-15T06:22:29.920 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2023-11-15T06:22:29.921 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2023-11-15T06:22:30.044 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-11-15T06:22:30.044 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:22:30.044 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2023-11-15T06:22:30.044 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2023-11-15T06:22:30.045 INFO:teuthology.orchestra.run.smithi062.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2023-11-15T06:22:30.045 INFO:teuthology.orchestra.run.smithi062.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2023-11-15T06:22:30.045 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:22:30.045 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2023-11-15T06:22:30.234 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2023-11-15T06:22:30.237 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2023-11-15T06:22:30.262 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph 2023-11-15T06:22:31.144 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:00:57 ago on Wed 15 Nov 2023 06:21:34 AM UTC. 2023-11-15T06:22:31.446 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2023-11-15T06:22:31.649 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-11-15T06:22:31.859 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-11-15T06:22:32.118 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-11-15T06:22:32.341 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-11-15T06:22:32.522 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-11-15T06:22:32.694 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-11-15T06:22:32.896 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-11-15T06:22:32.942 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2023-11-15T06:22:32.945 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:22:32.945 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2023-11-15T06:22:32.945 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:22:32.946 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2023-11-15T06:22:32.946 INFO:teuthology.orchestra.run.smithi062.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2023-11-15T06:22:32.946 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2023-11-15T06:22:32.946 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2023-11-15T06:22:32.946 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2023-11-15T06:22:32.947 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2023-11-15T06:22:32.947 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2023-11-15T06:22:32.947 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2023-11-15T06:22:32.947 INFO:teuthology.orchestra.run.smithi062.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-15T06:22:32.947 INFO:teuthology.orchestra.run.smithi062.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-15T06:22:32.948 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-11-15T06:22:32.948 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-15T06:22:32.948 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-15T06:22:32.948 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-11-15T06:22:32.948 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-15T06:22:32.948 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-15T06:22:32.949 INFO:teuthology.orchestra.run.smithi062.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-15T06:22:32.949 INFO:teuthology.orchestra.run.smithi062.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-11-15T06:22:32.949 INFO:teuthology.orchestra.run.smithi062.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-11-15T06:22:32.949 INFO:teuthology.orchestra.run.smithi062.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-15T06:22:32.949 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-15T06:22:32.950 INFO:teuthology.orchestra.run.smithi062.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-15T06:22:32.950 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-11-15T06:22:32.950 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-11-15T06:22:32.950 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-11-15T06:22:32.950 INFO:teuthology.orchestra.run.smithi062.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-15T06:22:32.951 INFO:teuthology.orchestra.run.smithi062.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-15T06:22:32.951 INFO:teuthology.orchestra.run.smithi062.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-15T06:22:32.951 INFO:teuthology.orchestra.run.smithi062.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-15T06:22:32.952 INFO:teuthology.orchestra.run.smithi062.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-15T06:22:32.952 INFO:teuthology.orchestra.run.smithi062.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-15T06:22:32.952 INFO:teuthology.orchestra.run.smithi062.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-15T06:22:32.953 INFO:teuthology.orchestra.run.smithi062.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-11-15T06:22:32.953 INFO:teuthology.orchestra.run.smithi062.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-15T06:22:32.953 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:22:32.953 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2023-11-15T06:22:32.954 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:22:32.954 INFO:teuthology.orchestra.run.smithi062.stdout:Install 32 Packages 2023-11-15T06:22:32.954 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:22:32.955 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 31 M 2023-11-15T06:22:32.955 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 111 M 2023-11-15T06:22:32.955 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2023-11-15T06:22:33.107 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-11-15T06:22:33.170 INFO:teuthology.orchestra.run.smithi062.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 111 kB/s | 6.3 kB 00:00 2023-11-15T06:22:33.211 INFO:teuthology.orchestra.run.smithi062.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2023-11-15T06:22:33.262 INFO:teuthology.orchestra.run.smithi062.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 16 MB/s | 2.4 MB 00:00 2023-11-15T06:22:33.289 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-11-15T06:22:33.296 INFO:teuthology.orchestra.run.smithi062.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 6.5 MB/s | 220 kB 00:00 2023-11-15T06:22:33.380 INFO:teuthology.orchestra.run.smithi062.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 24 MB/s | 5.0 MB 00:00 2023-11-15T06:22:33.485 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-11-15T06:22:33.613 INFO:teuthology.orchestra.run.smithi062.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 45 MB/s | 18 MB 00:00 2023-11-15T06:22:33.638 INFO:teuthology.orchestra.run.smithi062.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 150 kB/s | 39 kB 00:00 2023-11-15T06:22:33.664 INFO:teuthology.orchestra.run.smithi062.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 427 kB/s | 157 kB 00:00 2023-11-15T06:22:33.722 INFO:teuthology.orchestra.run.smithi062.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 680 kB/s | 57 kB 00:00 2023-11-15T06:22:33.736 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-11-15T06:22:33.781 INFO:teuthology.orchestra.run.smithi062.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 3.8 MB/s | 457 kB 00:00 2023-11-15T06:22:33.815 INFO:teuthology.orchestra.run.smithi062.stdout:(11/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 515 kB/s | 103 kB 00:00 2023-11-15T06:22:34.007 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-11-15T06:22:34.223 INFO:teuthology.orchestra.run.smithi062.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 497 kB/s | 249 kB 00:00 2023-11-15T06:22:34.249 INFO:teuthology.orchestra.run.smithi062.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 251 kB/s | 109 kB 00:00 2023-11-15T06:22:34.357 INFO:teuthology.orchestra.run.smithi062.stdout:(14/32): python3-cryptography-3.2.1-6.el8.x86_6 1.1 MB/s | 641 kB 00:00 2023-11-15T06:22:34.382 INFO:teuthology.orchestra.run.smithi062.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 275 kB/s | 44 kB 00:00 2023-11-15T06:22:34.389 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-11-15T06:22:34.458 INFO:teuthology.orchestra.run.smithi062.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 888 kB/s | 185 kB 00:00 2023-11-15T06:22:34.499 INFO:teuthology.orchestra.run.smithi062.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 272 kB/s | 11 kB 00:00 2023-11-15T06:22:34.541 INFO:teuthology.orchestra.run.smithi062.stdout:(18/32): python3-jaraco-functools-2.0-4.el8.noa 423 kB/s | 18 kB 00:00 2023-11-15T06:22:34.567 INFO:teuthology.orchestra.run.smithi062.stdout:(19/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 825 kB/s | 173 kB 00:00 2023-11-15T06:22:34.592 INFO:teuthology.orchestra.run.smithi062.stdout:(20/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.8 MB/s | 384 kB 00:00 2023-11-15T06:22:34.617 INFO:teuthology.orchestra.run.smithi062.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 646 kB/s | 49 kB 00:00 2023-11-15T06:22:34.643 INFO:teuthology.orchestra.run.smithi062.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 778 kB/s | 59 kB 00:00 2023-11-15T06:22:34.654 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-11-15T06:22:34.668 INFO:teuthology.orchestra.run.smithi062.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 310 kB/s | 16 kB 00:00 2023-11-15T06:22:34.693 INFO:teuthology.orchestra.run.smithi062.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 374 kB/s | 19 kB 00:00 2023-11-15T06:22:34.718 INFO:teuthology.orchestra.run.smithi062.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 477 kB/s | 24 kB 00:00 2023-11-15T06:22:34.744 INFO:teuthology.orchestra.run.smithi062.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 581 kB/s | 29 kB 00:00 2023-11-15T06:22:34.769 INFO:teuthology.orchestra.run.smithi062.stdout:(27/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.6 MB/s | 283 kB 00:00 2023-11-15T06:22:34.786 INFO:teuthology.orchestra.run.smithi062.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 410 kB/s | 27 kB 00:00 2023-11-15T06:22:34.811 INFO:teuthology.orchestra.run.smithi062.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 3.7 MB/s | 255 kB 00:00 2023-11-15T06:22:34.836 INFO:teuthology.orchestra.run.smithi062.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.7 MB/s | 85 kB 00:00 2023-11-15T06:22:34.853 INFO:teuthology.orchestra.run.smithi062.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 552 kB/s | 23 kB 00:00 2023-11-15T06:22:34.867 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-11-15T06:22:34.887 INFO:teuthology.orchestra.run.smithi062.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.1 MB/s | 251 kB 00:00 2023-11-15T06:22:34.891 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:22:34.891 INFO:teuthology.orchestra.run.smithi062.stdout:Total 16 MB/s | 31 MB 00:01 2023-11-15T06:22:34.978 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2023-11-15T06:22:35.020 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2023-11-15T06:22:35.021 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2023-11-15T06:22:35.127 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-11-15T06:22:35.344 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-15T06:22:35.359 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2023-11-15T06:22:35.362 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2023-11-15T06:22:35.674 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-11-15T06:22:35.907 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-11-15T06:22:36.121 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-11-15T06:22:36.313 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-11-15T06:22:36.451 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2023-11-15T06:22:36.585 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-11-15T06:22:36.673 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-11-15T06:22:36.839 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-11-15T06:22:36.840 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-11-15T06:22:37.078 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-11-15T06:22:37.084 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-11-15T06:22:37.279 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-11-15T06:22:37.441 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-11-15T06:22:37.452 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-11-15T06:22:37.634 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-11-15T06:22:37.716 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-11-15T06:22:37.818 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-11-15T06:22:37.987 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-11-15T06:22:38.094 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2023-11-15T06:22:38.173 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-11-15T06:22:38.332 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-11-15T06:22:38.376 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-11-15T06:22:38.587 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-11-15T06:22:38.823 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-11-15T06:22:39.205 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-11-15T06:22:39.459 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-11-15T06:22:39.613 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-11-15T06:22:39.849 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-11-15T06:22:39.860 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-11-15T06:22:40.024 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-15T06:22:40.065 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-11-15T06:22:40.328 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-11-15T06:22:40.498 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-11-15T06:22:40.512 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-11-15T06:22:40.683 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-11-15T06:22:40.765 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-11-15T06:22:40.875 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-11-15T06:22:40.987 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-11-15T06:22:41.109 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-11-15T06:22:41.168 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-11-15T06:22:41.185 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-11-15T06:22:41.327 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-11-15T06:22:41.359 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2023-11-15T06:22:41.573 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-11-15T06:22:41.763 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2023-11-15T06:22:41.763 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2023-11-15T06:22:41.763 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2023-11-15T06:22:41.763 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2023-11-15T06:22:41.764 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2023-11-15T06:22:41.764 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2023-11-15T06:22:41.764 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2023-11-15T06:22:41.764 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-11-15T06:22:41.764 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-11-15T06:22:41.765 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-11-15T06:22:41.765 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-11-15T06:22:41.765 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-11-15T06:22:41.765 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-11-15T06:22:41.765 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-11-15T06:22:41.766 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-11-15T06:22:41.766 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-11-15T06:22:41.766 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-11-15T06:22:41.766 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-15T06:22:41.766 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-11-15T06:22:41.767 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-11-15T06:22:41.767 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-11-15T06:22:41.767 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-11-15T06:22:41.767 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-11-15T06:22:41.768 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-11-15T06:22:41.768 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-11-15T06:22:41.768 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-11-15T06:22:41.768 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-11-15T06:22:41.768 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-11-15T06:22:41.769 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-11-15T06:22:41.769 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-11-15T06:22:41.770 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-11-15T06:22:41.770 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-11-15T06:22:41.892 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-11-15T06:22:42.184 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-11-15T06:22:42.534 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2023-11-15T06:22:42.730 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-11-15T06:22:42.887 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-11-15T06:22:42.887 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:22:42.887 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2023-11-15T06:22:42.888 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-2:16.2.4-0.el8.x86_64 2023-11-15T06:22:42.888 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2023-11-15T06:22:42.888 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2023-11-15T06:22:42.888 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2023-11-15T06:22:42.888 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2023-11-15T06:22:42.889 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2023-11-15T06:22:42.889 INFO:teuthology.orchestra.run.smithi081.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-15T06:22:42.889 INFO:teuthology.orchestra.run.smithi081.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-15T06:22:42.889 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-11-15T06:22:42.889 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-15T06:22:42.890 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-15T06:22:42.890 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-11-15T06:22:42.890 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-15T06:22:42.890 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-15T06:22:42.890 INFO:teuthology.orchestra.run.smithi081.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-15T06:22:42.891 INFO:teuthology.orchestra.run.smithi081.stdout: python3-mako-1.0.6-14.el8.noarch 2023-11-15T06:22:42.891 INFO:teuthology.orchestra.run.smithi081.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-15T06:22:42.891 INFO:teuthology.orchestra.run.smithi081.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-15T06:22:42.891 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-15T06:22:42.891 INFO:teuthology.orchestra.run.smithi081.stdout: python3-portend-2.6-1.el8.noarch 2023-11-15T06:22:42.892 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-15T06:22:42.892 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-15T06:22:42.892 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pytz-2017.2-11.el8.noarch 2023-11-15T06:22:42.892 INFO:teuthology.orchestra.run.smithi081.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-15T06:22:42.892 INFO:teuthology.orchestra.run.smithi081.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-15T06:22:42.893 INFO:teuthology.orchestra.run.smithi081.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-15T06:22:42.893 INFO:teuthology.orchestra.run.smithi081.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-15T06:22:42.893 INFO:teuthology.orchestra.run.smithi081.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-15T06:22:42.893 INFO:teuthology.orchestra.run.smithi081.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-15T06:22:42.893 INFO:teuthology.orchestra.run.smithi081.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-15T06:22:42.894 INFO:teuthology.orchestra.run.smithi081.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-15T06:22:42.894 INFO:teuthology.orchestra.run.smithi081.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-15T06:22:42.894 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:22:42.895 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2023-11-15T06:22:43.142 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-base 2023-11-15T06:22:44.004 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:14 ago on Wed 15 Nov 2023 06:21:30 AM UTC. 2023-11-15T06:22:44.355 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-11-15T06:22:44.513 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-11-15T06:22:44.973 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-11-15T06:22:45.179 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-11-15T06:22:45.415 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-11-15T06:22:45.452 INFO:teuthology.orchestra.run.smithi081.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T06:22:45.589 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-11-15T06:22:45.607 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-11-15T06:22:45.667 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2023-11-15T06:22:45.668 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2023-11-15T06:22:45.668 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2023-11-15T06:22:45.730 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2023-11-15T06:22:45.784 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install cephadm 2023-11-15T06:22:46.152 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2023-11-15T06:22:46.153 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2023-11-15T06:22:46.153 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2023-11-15T06:22:46.153 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2023-11-15T06:22:46.153 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2023-11-15T06:22:46.154 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2023-11-15T06:22:46.155 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2023-11-15T06:22:46.155 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-11-15T06:22:46.155 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-11-15T06:22:46.156 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-11-15T06:22:46.156 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-11-15T06:22:46.156 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-11-15T06:22:46.156 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-11-15T06:22:46.156 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-11-15T06:22:46.157 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-11-15T06:22:46.157 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-11-15T06:22:46.157 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-11-15T06:22:46.157 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-15T06:22:46.158 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-11-15T06:22:46.158 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-11-15T06:22:46.158 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-11-15T06:22:46.158 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-11-15T06:22:46.158 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-11-15T06:22:46.159 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-11-15T06:22:46.159 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-11-15T06:22:46.159 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-11-15T06:22:46.159 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-11-15T06:22:46.159 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-11-15T06:22:46.160 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-11-15T06:22:46.160 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-11-15T06:22:46.160 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-11-15T06:22:46.160 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-11-15T06:22:46.632 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:16 ago on Wed 15 Nov 2023 06:21:30 AM UTC. 2023-11-15T06:22:47.370 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-11-15T06:22:47.370 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:22:47.370 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2023-11-15T06:22:47.370 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-2:16.2.4-0.el8.x86_64 2023-11-15T06:22:47.371 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2023-11-15T06:22:47.371 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2023-11-15T06:22:47.371 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2023-11-15T06:22:47.371 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2023-11-15T06:22:47.372 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2023-11-15T06:22:47.372 INFO:teuthology.orchestra.run.smithi062.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-15T06:22:47.372 INFO:teuthology.orchestra.run.smithi062.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-15T06:22:47.372 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-11-15T06:22:47.372 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-15T06:22:47.373 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-15T06:22:47.373 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-11-15T06:22:47.373 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-15T06:22:47.373 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-15T06:22:47.373 INFO:teuthology.orchestra.run.smithi062.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-15T06:22:47.374 INFO:teuthology.orchestra.run.smithi062.stdout: python3-mako-1.0.6-14.el8.noarch 2023-11-15T06:22:47.374 INFO:teuthology.orchestra.run.smithi062.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-15T06:22:47.374 INFO:teuthology.orchestra.run.smithi062.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-15T06:22:47.374 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-15T06:22:47.374 INFO:teuthology.orchestra.run.smithi062.stdout: python3-portend-2.6-1.el8.noarch 2023-11-15T06:22:47.375 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-15T06:22:47.375 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-15T06:22:47.375 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pytz-2017.2-11.el8.noarch 2023-11-15T06:22:47.375 INFO:teuthology.orchestra.run.smithi062.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-15T06:22:47.375 INFO:teuthology.orchestra.run.smithi062.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-15T06:22:47.376 INFO:teuthology.orchestra.run.smithi062.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-15T06:22:47.376 INFO:teuthology.orchestra.run.smithi062.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-15T06:22:47.376 INFO:teuthology.orchestra.run.smithi062.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-15T06:22:47.376 INFO:teuthology.orchestra.run.smithi062.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-15T06:22:47.376 INFO:teuthology.orchestra.run.smithi062.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-15T06:22:47.377 INFO:teuthology.orchestra.run.smithi062.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-15T06:22:47.377 INFO:teuthology.orchestra.run.smithi062.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-15T06:22:47.377 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:22:47.377 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2023-11-15T06:22:47.512 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-base 2023-11-15T06:22:48.302 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2023-11-15T06:22:48.303 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:22:48.303 INFO:teuthology.orchestra.run.smithi081.stdout: Package Architecture Version Repository Size 2023-11-15T06:22:48.303 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:22:48.304 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2023-11-15T06:22:48.304 INFO:teuthology.orchestra.run.smithi081.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2023-11-15T06:22:48.304 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:22:48.304 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2023-11-15T06:22:48.304 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:22:48.305 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2023-11-15T06:22:48.305 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:22:48.305 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 71 k 2023-11-15T06:22:48.305 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 290 k 2023-11-15T06:22:48.305 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2023-11-15T06:22:48.357 INFO:teuthology.orchestra.run.smithi081.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.3 MB/s | 71 kB 00:00 2023-11-15T06:22:48.358 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:22:48.358 INFO:teuthology.orchestra.run.smithi081.stdout:Total 1.3 MB/s | 71 kB 00:00 2023-11-15T06:22:48.358 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2023-11-15T06:22:48.364 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2023-11-15T06:22:48.364 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2023-11-15T06:22:48.366 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2023-11-15T06:22:48.369 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2023-11-15T06:22:48.408 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:14 ago on Wed 15 Nov 2023 06:21:34 AM UTC. 2023-11-15T06:22:48.443 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2023-11-15T06:22:49.023 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T06:22:49.250 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T06:22:49.479 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T06:22:49.974 INFO:teuthology.orchestra.run.smithi062.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T06:22:50.209 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2023-11-15T06:22:50.210 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2023-11-15T06:22:50.210 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2023-11-15T06:22:50.316 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T06:22:50.317 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:22:50.317 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2023-11-15T06:22:50.317 INFO:teuthology.orchestra.run.smithi081.stdout: cephadm-2:16.2.4-0.el8.noarch 2023-11-15T06:22:50.317 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:22:50.317 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2023-11-15T06:22:50.326 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install cephadm 2023-11-15T06:22:51.213 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:17 ago on Wed 15 Nov 2023 06:21:34 AM UTC. 2023-11-15T06:22:51.284 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-immutable-object-cache 2023-11-15T06:22:52.152 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:22 ago on Wed 15 Nov 2023 06:21:30 AM UTC. 2023-11-15T06:22:52.994 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2023-11-15T06:22:52.995 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:22:52.996 INFO:teuthology.orchestra.run.smithi062.stdout: Package Architecture Version Repository Size 2023-11-15T06:22:52.996 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:22:52.996 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2023-11-15T06:22:52.996 INFO:teuthology.orchestra.run.smithi062.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2023-11-15T06:22:52.996 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:22:52.997 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2023-11-15T06:22:52.997 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:22:52.997 INFO:teuthology.orchestra.run.smithi062.stdout:Install 1 Package 2023-11-15T06:22:52.997 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:22:52.997 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 71 k 2023-11-15T06:22:52.998 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 290 k 2023-11-15T06:22:52.998 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2023-11-15T06:22:53.035 INFO:teuthology.orchestra.run.smithi062.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.8 MB/s | 71 kB 00:00 2023-11-15T06:22:53.036 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:22:53.036 INFO:teuthology.orchestra.run.smithi062.stdout:Total 1.7 MB/s | 71 kB 00:00 2023-11-15T06:22:53.036 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2023-11-15T06:22:53.043 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2023-11-15T06:22:53.043 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2023-11-15T06:22:53.045 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2023-11-15T06:22:53.048 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2023-11-15T06:22:53.094 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2023-11-15T06:22:53.604 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T06:22:53.775 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T06:22:53.820 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2023-11-15T06:22:53.820 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:22:53.821 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2023-11-15T06:22:53.821 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:22:53.821 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2023-11-15T06:22:53.821 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2023-11-15T06:22:53.822 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:22:53.822 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2023-11-15T06:22:53.822 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:22:53.822 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2023-11-15T06:22:53.822 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:22:53.822 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 163 k 2023-11-15T06:22:53.823 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 587 k 2023-11-15T06:22:53.823 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2023-11-15T06:22:53.871 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 3.3 MB/s | 163 kB 00:00 2023-11-15T06:22:53.871 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:22:53.872 INFO:teuthology.orchestra.run.smithi081.stdout:Total 3.2 MB/s | 163 kB 00:00 2023-11-15T06:22:53.872 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2023-11-15T06:22:53.882 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2023-11-15T06:22:53.882 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2023-11-15T06:22:53.930 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2023-11-15T06:22:53.933 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2023-11-15T06:22:54.005 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T06:22:54.050 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2023-11-15T06:22:54.266 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:22:54.687 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:22:54.690 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T06:22:54.690 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:22:54.690 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2023-11-15T06:22:54.690 INFO:teuthology.orchestra.run.smithi062.stdout: cephadm-2:16.2.4-0.el8.noarch 2023-11-15T06:22:54.690 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:22:54.691 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2023-11-15T06:22:55.434 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:22:55.434 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:22:55.435 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2023-11-15T06:22:55.435 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2023-11-15T06:22:55.435 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:22:55.435 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2023-11-15T06:22:55.658 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-immutable-object-cache 2023-11-15T06:22:55.673 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-mgr 2023-11-15T06:22:56.541 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:26 ago on Wed 15 Nov 2023 06:21:30 AM UTC. 2023-11-15T06:22:56.556 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:22 ago on Wed 15 Nov 2023 06:21:34 AM UTC. 2023-11-15T06:22:57.988 INFO:teuthology.orchestra.run.smithi081.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T06:22:58.205 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2023-11-15T06:22:58.206 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2023-11-15T06:22:58.206 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2023-11-15T06:22:58.310 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-mgr-dashboard 2023-11-15T06:22:58.340 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2023-11-15T06:22:58.341 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:22:58.342 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repo Size 2023-11-15T06:22:58.342 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:22:58.342 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2023-11-15T06:22:58.342 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2023-11-15T06:22:58.343 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:22:58.343 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2023-11-15T06:22:58.344 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:22:58.344 INFO:teuthology.orchestra.run.smithi062.stdout:Install 1 Package 2023-11-15T06:22:58.345 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:22:58.345 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 163 k 2023-11-15T06:22:58.345 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 587 k 2023-11-15T06:22:58.345 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2023-11-15T06:22:58.384 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 3.8 MB/s | 163 kB 00:00 2023-11-15T06:22:58.384 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:22:58.385 INFO:teuthology.orchestra.run.smithi062.stdout:Total 3.7 MB/s | 163 kB 00:00 2023-11-15T06:22:58.385 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2023-11-15T06:22:58.397 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2023-11-15T06:22:58.397 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2023-11-15T06:22:58.447 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2023-11-15T06:22:58.450 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2023-11-15T06:22:58.562 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2023-11-15T06:22:58.736 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:22:59.156 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:29 ago on Wed 15 Nov 2023 06:21:30 AM UTC. 2023-11-15T06:22:59.174 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:22:59.860 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:22:59.860 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:22:59.861 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2023-11-15T06:22:59.861 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2023-11-15T06:22:59.861 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:22:59.861 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2023-11-15T06:23:00.083 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-mgr 2023-11-15T06:23:00.810 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2023-11-15T06:23:00.811 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:23:00.812 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2023-11-15T06:23:00.812 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:23:00.812 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2023-11-15T06:23:00.812 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2023-11-15T06:23:00.812 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2023-11-15T06:23:00.813 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2023-11-15T06:23:00.813 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2023-11-15T06:23:00.813 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-11-15T06:23:00.813 INFO:teuthology.orchestra.run.smithi081.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-15T06:23:00.813 INFO:teuthology.orchestra.run.smithi081.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-15T06:23:00.814 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:23:00.814 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2023-11-15T06:23:00.814 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:23:00.814 INFO:teuthology.orchestra.run.smithi081.stdout:Install 6 Packages 2023-11-15T06:23:00.814 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:23:00.815 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 3.0 M 2023-11-15T06:23:00.815 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 15 M 2023-11-15T06:23:00.816 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2023-11-15T06:23:00.981 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:26 ago on Wed 15 Nov 2023 06:21:34 AM UTC. 2023-11-15T06:23:01.382 INFO:teuthology.orchestra.run.smithi081.stdout:(1/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 20 MB/s | 2.7 MB 00:00 2023-11-15T06:23:01.408 INFO:teuthology.orchestra.run.smithi081.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 57 kB/s | 8.9 kB 00:00 2023-11-15T06:23:01.433 INFO:teuthology.orchestra.run.smithi081.stdout:(3/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 132 kB/s | 24 kB 00:00 2023-11-15T06:23:01.666 INFO:teuthology.orchestra.run.smithi081.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 130 kB/s | 34 kB 00:00 2023-11-15T06:23:01.742 INFO:teuthology.orchestra.run.smithi081.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 636 kB/s | 196 kB 00:00 2023-11-15T06:23:01.767 INFO:teuthology.orchestra.run.smithi081.stdout:(6/6): python3-jwt-1.6.1-2.el8.noarch.rpm 111 kB/s | 43 kB 00:00 2023-11-15T06:23:01.770 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:23:01.770 INFO:teuthology.orchestra.run.smithi081.stdout:Total 3.1 MB/s | 3.0 MB 00:00 2023-11-15T06:23:01.780 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2023-11-15T06:23:01.790 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2023-11-15T06:23:01.791 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2023-11-15T06:23:01.962 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2023-11-15T06:23:01.966 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2023-11-15T06:23:02.308 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2023-11-15T06:23:02.522 INFO:teuthology.orchestra.run.smithi062.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T06:23:02.537 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-11-15T06:23:02.757 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2023-11-15T06:23:02.759 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2023-11-15T06:23:02.759 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2023-11-15T06:23:02.867 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-mgr-dashboard 2023-11-15T06:23:02.924 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-11-15T06:23:03.141 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-11-15T06:23:03.294 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2023-11-15T06:23:03.749 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:29 ago on Wed 15 Nov 2023 06:21:34 AM UTC. 2023-11-15T06:23:04.309 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2023-11-15T06:23:04.603 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-11-15T06:23:04.921 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-11-15T06:23:04.922 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2023-11-15T06:23:04.922 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2023-11-15T06:23:04.922 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2023-11-15T06:23:04.922 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-11-15T06:23:04.923 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-11-15T06:23:05.529 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2023-11-15T06:23:05.530 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:23:05.531 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2023-11-15T06:23:05.531 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:23:05.531 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2023-11-15T06:23:05.531 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2023-11-15T06:23:05.531 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2023-11-15T06:23:05.532 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2023-11-15T06:23:05.532 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2023-11-15T06:23:05.532 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-11-15T06:23:05.532 INFO:teuthology.orchestra.run.smithi062.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-15T06:23:05.532 INFO:teuthology.orchestra.run.smithi062.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-15T06:23:05.533 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:23:05.533 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2023-11-15T06:23:05.533 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:23:05.533 INFO:teuthology.orchestra.run.smithi062.stdout:Install 6 Packages 2023-11-15T06:23:05.533 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:23:05.534 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 3.0 M 2023-11-15T06:23:05.534 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 15 M 2023-11-15T06:23:05.534 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2023-11-15T06:23:05.658 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-11-15T06:23:05.658 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:23:05.658 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2023-11-15T06:23:05.659 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2023-11-15T06:23:05.659 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2023-11-15T06:23:05.659 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2023-11-15T06:23:05.659 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-15T06:23:05.659 INFO:teuthology.orchestra.run.smithi081.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-15T06:23:05.660 INFO:teuthology.orchestra.run.smithi081.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-15T06:23:05.660 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:23:05.660 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2023-11-15T06:23:05.754 INFO:teuthology.orchestra.run.smithi062.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 577 kB/s | 24 kB 00:00 2023-11-15T06:23:05.779 INFO:teuthology.orchestra.run.smithi062.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 134 kB/s | 8.9 kB 00:00 2023-11-15T06:23:05.791 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-mgr-diskprediction-local 2023-11-15T06:23:05.838 INFO:teuthology.orchestra.run.smithi062.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 21 MB/s | 2.7 MB 00:00 2023-11-15T06:23:06.030 INFO:teuthology.orchestra.run.smithi062.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 134 kB/s | 34 kB 00:00 2023-11-15T06:23:06.116 INFO:teuthology.orchestra.run.smithi062.stdout:(5/6): python3-jwt-1.6.1-2.el8.noarch.rpm 118 kB/s | 43 kB 00:00 2023-11-15T06:23:06.214 INFO:teuthology.orchestra.run.smithi062.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 523 kB/s | 196 kB 00:00 2023-11-15T06:23:06.216 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:23:06.216 INFO:teuthology.orchestra.run.smithi062.stdout:Total 4.3 MB/s | 3.0 MB 00:00 2023-11-15T06:23:06.227 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2023-11-15T06:23:06.238 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2023-11-15T06:23:06.238 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2023-11-15T06:23:06.401 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2023-11-15T06:23:06.404 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2023-11-15T06:23:06.641 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:36 ago on Wed 15 Nov 2023 06:21:30 AM UTC. 2023-11-15T06:23:06.683 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2023-11-15T06:23:06.877 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-11-15T06:23:07.121 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-11-15T06:23:07.329 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-11-15T06:23:07.466 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2023-11-15T06:23:08.304 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2023-11-15T06:23:08.305 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================================ 2023-11-15T06:23:08.305 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2023-11-15T06:23:08.306 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================================ 2023-11-15T06:23:08.306 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2023-11-15T06:23:08.306 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2023-11-15T06:23:08.306 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2023-11-15T06:23:08.306 INFO:teuthology.orchestra.run.smithi081.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-11-15T06:23:08.307 INFO:teuthology.orchestra.run.smithi081.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-11-15T06:23:08.307 INFO:teuthology.orchestra.run.smithi081.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-11-15T06:23:08.307 INFO:teuthology.orchestra.run.smithi081.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-11-15T06:23:08.307 INFO:teuthology.orchestra.run.smithi081.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-11-15T06:23:08.307 INFO:teuthology.orchestra.run.smithi081.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-11-15T06:23:08.308 INFO:teuthology.orchestra.run.smithi081.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-11-15T06:23:08.308 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:23:08.308 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2023-11-15T06:23:08.308 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================================ 2023-11-15T06:23:08.308 INFO:teuthology.orchestra.run.smithi081.stdout:Install 8 Packages 2023-11-15T06:23:08.309 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:23:08.309 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 37 M 2023-11-15T06:23:08.310 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 210 M 2023-11-15T06:23:08.310 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2023-11-15T06:23:08.493 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2023-11-15T06:23:08.734 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-11-15T06:23:09.031 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-11-15T06:23:09.032 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2023-11-15T06:23:09.032 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2023-11-15T06:23:09.032 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2023-11-15T06:23:09.032 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-11-15T06:23:09.033 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-11-15T06:23:09.751 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-11-15T06:23:09.751 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:23:09.752 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2023-11-15T06:23:09.752 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2023-11-15T06:23:09.752 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2023-11-15T06:23:09.752 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2023-11-15T06:23:09.752 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-15T06:23:09.753 INFO:teuthology.orchestra.run.smithi062.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-15T06:23:09.753 INFO:teuthology.orchestra.run.smithi062.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-15T06:23:09.753 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:23:09.753 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2023-11-15T06:23:09.876 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-mgr-diskprediction-local 2023-11-15T06:23:10.770 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:36 ago on Wed 15 Nov 2023 06:21:34 AM UTC. 2023-11-15T06:23:12.553 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2023-11-15T06:23:12.555 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================================ 2023-11-15T06:23:12.555 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2023-11-15T06:23:12.555 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================================ 2023-11-15T06:23:12.555 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2023-11-15T06:23:12.556 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2023-11-15T06:23:12.556 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2023-11-15T06:23:12.556 INFO:teuthology.orchestra.run.smithi062.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-11-15T06:23:12.556 INFO:teuthology.orchestra.run.smithi062.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-11-15T06:23:12.556 INFO:teuthology.orchestra.run.smithi062.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-11-15T06:23:12.557 INFO:teuthology.orchestra.run.smithi062.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-11-15T06:23:12.557 INFO:teuthology.orchestra.run.smithi062.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-11-15T06:23:12.557 INFO:teuthology.orchestra.run.smithi062.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-11-15T06:23:12.557 INFO:teuthology.orchestra.run.smithi062.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-11-15T06:23:12.557 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:23:12.558 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2023-11-15T06:23:12.558 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================================ 2023-11-15T06:23:12.558 INFO:teuthology.orchestra.run.smithi062.stdout:Install 8 Packages 2023-11-15T06:23:12.558 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:23:12.559 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 37 M 2023-11-15T06:23:12.559 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 210 M 2023-11-15T06:23:12.560 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2023-11-15T06:23:12.938 INFO:teuthology.orchestra.run.smithi062.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 56 MB/s | 8.6 MB 00:00 2023-11-15T06:23:12.970 INFO:teuthology.orchestra.run.smithi081.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 34 MB/s | 8.6 MB 00:00 2023-11-15T06:23:13.497 INFO:teuthology.orchestra.run.smithi062.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 6.8 MB/s | 4.9 MB 00:00 2023-11-15T06:23:13.555 INFO:teuthology.orchestra.run.smithi062.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 3.8 MB/s | 225 kB 00:00 2023-11-15T06:23:13.681 INFO:teuthology.orchestra.run.smithi062.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 5.0 MB/s | 3.7 MB 00:00 2023-11-15T06:23:14.100 INFO:teuthology.orchestra.run.smithi081.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 3.5 MB/s | 4.9 MB 00:01 2023-11-15T06:23:14.239 INFO:teuthology.orchestra.run.smithi062.stdout:(5/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.1 MB/s | 645 kB 00:00 2023-11-15T06:23:14.259 INFO:teuthology.orchestra.run.smithi081.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.4 MB/s | 225 kB 00:00 2023-11-15T06:23:14.364 INFO:teuthology.orchestra.run.smithi062.stdout:(6/8): openblas-0.3.15-4.el8.x86_64.rpm 3.0 MB/s | 4.7 MB 00:01 2023-11-15T06:23:14.390 INFO:teuthology.orchestra.run.smithi062.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 1.1 MB/s | 172 kB 00:00 2023-11-15T06:23:14.501 INFO:teuthology.orchestra.run.smithi081.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 2.4 MB/s | 3.7 MB 00:01 2023-11-15T06:23:14.532 INFO:teuthology.orchestra.run.smithi062.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 14 MB/s | 14 MB 00:00 2023-11-15T06:23:14.532 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:23:14.533 INFO:teuthology.orchestra.run.smithi062.stdout:Total 19 MB/s | 37 MB 00:01 2023-11-15T06:23:14.626 INFO:teuthology.orchestra.run.smithi081.stdout:(5/8): openblas-0.3.15-4.el8.x86_64.rpm 2.5 MB/s | 4.7 MB 00:01 2023-11-15T06:23:14.769 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2023-11-15T06:23:14.789 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2023-11-15T06:23:14.789 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2023-11-15T06:23:15.035 INFO:teuthology.orchestra.run.smithi081.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 422 kB/s | 172 kB 00:00 2023-11-15T06:23:15.129 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2023-11-15T06:23:15.133 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2023-11-15T06:23:15.135 INFO:teuthology.orchestra.run.smithi081.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.0 MB/s | 645 kB 00:00 2023-11-15T06:23:15.593 INFO:teuthology.orchestra.run.smithi081.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 10 MB/s | 14 MB 00:01 2023-11-15T06:23:15.594 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:23:15.595 INFO:teuthology.orchestra.run.smithi081.stdout:Total 5.0 MB/s | 37 MB 00:07 2023-11-15T06:23:15.757 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2023-11-15T06:23:15.831 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2023-11-15T06:23:15.849 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2023-11-15T06:23:15.849 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2023-11-15T06:23:15.909 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-15T06:23:16.047 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-15T06:23:16.177 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2023-11-15T06:23:16.180 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2023-11-15T06:23:16.185 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-15T06:23:16.815 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-15T06:23:16.825 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2023-11-15T06:23:16.975 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-11-15T06:23:17.004 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-15T06:23:17.150 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-15T06:23:17.312 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-15T06:23:17.627 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-11-15T06:23:17.890 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-15T06:23:18.104 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-11-15T06:23:18.567 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-15T06:23:18.701 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-11-15T06:23:18.904 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-15T06:23:19.312 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-15T06:23:19.570 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-15T06:23:19.633 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-11-15T06:23:19.854 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-15T06:23:20.065 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-11-15T06:23:20.350 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-15T06:23:21.736 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-15T06:23:22.130 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-15T06:23:23.067 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-11-15T06:23:23.262 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-11-15T06:23:23.451 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-11-15T06:23:23.600 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-11-15T06:23:23.600 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2023-11-15T06:23:23.601 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-11-15T06:23:23.601 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-11-15T06:23:23.601 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-11-15T06:23:23.602 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-11-15T06:23:23.602 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-11-15T06:23:23.603 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-11-15T06:23:23.649 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-11-15T06:23:23.970 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-11-15T06:23:23.970 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2023-11-15T06:23:23.970 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-11-15T06:23:23.971 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-11-15T06:23:23.971 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-11-15T06:23:23.971 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-11-15T06:23:23.971 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-11-15T06:23:23.971 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-11-15T06:23:24.382 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-11-15T06:23:24.382 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:23:24.382 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2023-11-15T06:23:24.383 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2023-11-15T06:23:24.383 INFO:teuthology.orchestra.run.smithi062.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-11-15T06:23:24.383 INFO:teuthology.orchestra.run.smithi062.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-11-15T06:23:24.383 INFO:teuthology.orchestra.run.smithi062.stdout: openblas-0.3.15-4.el8.x86_64 2023-11-15T06:23:24.383 INFO:teuthology.orchestra.run.smithi062.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-11-15T06:23:24.384 INFO:teuthology.orchestra.run.smithi062.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-15T06:23:24.384 INFO:teuthology.orchestra.run.smithi062.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-15T06:23:24.384 INFO:teuthology.orchestra.run.smithi062.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-11-15T06:23:24.384 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:23:24.384 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2023-11-15T06:23:24.500 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-mgr-rook 2023-11-15T06:23:24.743 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-11-15T06:23:24.743 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:23:24.744 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2023-11-15T06:23:24.744 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2023-11-15T06:23:24.744 INFO:teuthology.orchestra.run.smithi081.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-11-15T06:23:24.744 INFO:teuthology.orchestra.run.smithi081.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-11-15T06:23:24.744 INFO:teuthology.orchestra.run.smithi081.stdout: openblas-0.3.15-4.el8.x86_64 2023-11-15T06:23:24.745 INFO:teuthology.orchestra.run.smithi081.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-11-15T06:23:24.745 INFO:teuthology.orchestra.run.smithi081.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-15T06:23:24.745 INFO:teuthology.orchestra.run.smithi081.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-15T06:23:24.745 INFO:teuthology.orchestra.run.smithi081.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-11-15T06:23:24.745 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:23:24.746 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2023-11-15T06:23:24.845 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-mgr-rook 2023-11-15T06:23:25.396 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:51 ago on Wed 15 Nov 2023 06:21:34 AM UTC. 2023-11-15T06:23:25.695 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:55 ago on Wed 15 Nov 2023 06:21:30 AM UTC. 2023-11-15T06:23:27.180 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2023-11-15T06:23:27.182 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:23:27.182 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2023-11-15T06:23:27.183 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:23:27.183 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2023-11-15T06:23:27.183 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2023-11-15T06:23:27.183 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2023-11-15T06:23:27.184 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-15T06:23:27.184 INFO:teuthology.orchestra.run.smithi062.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-15T06:23:27.184 INFO:teuthology.orchestra.run.smithi062.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-15T06:23:27.184 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-11-15T06:23:27.184 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-11-15T06:23:27.185 INFO:teuthology.orchestra.run.smithi062.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-15T06:23:27.185 INFO:teuthology.orchestra.run.smithi062.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-11-15T06:23:27.185 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-11-15T06:23:27.185 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-11-15T06:23:27.185 INFO:teuthology.orchestra.run.smithi062.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-11-15T06:23:27.186 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-15T06:23:27.186 INFO:teuthology.orchestra.run.smithi062.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-15T06:23:27.186 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:23:27.186 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2023-11-15T06:23:27.186 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:23:27.187 INFO:teuthology.orchestra.run.smithi062.stdout:Install 13 Packages 2023-11-15T06:23:27.187 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:23:27.188 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 1.9 M 2023-11-15T06:23:27.188 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 24 M 2023-11-15T06:23:27.188 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2023-11-15T06:23:27.364 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2023-11-15T06:23:27.365 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:23:27.365 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2023-11-15T06:23:27.365 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:23:27.366 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2023-11-15T06:23:27.366 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2023-11-15T06:23:27.366 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2023-11-15T06:23:27.366 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-15T06:23:27.367 INFO:teuthology.orchestra.run.smithi081.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-15T06:23:27.367 INFO:teuthology.orchestra.run.smithi081.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-15T06:23:27.367 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-11-15T06:23:27.367 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-11-15T06:23:27.367 INFO:teuthology.orchestra.run.smithi081.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-15T06:23:27.368 INFO:teuthology.orchestra.run.smithi081.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-11-15T06:23:27.368 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-11-15T06:23:27.368 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-11-15T06:23:27.368 INFO:teuthology.orchestra.run.smithi081.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-11-15T06:23:27.368 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-15T06:23:27.369 INFO:teuthology.orchestra.run.smithi081.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-15T06:23:27.369 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:23:27.369 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2023-11-15T06:23:27.369 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:23:27.369 INFO:teuthology.orchestra.run.smithi081.stdout:Install 13 Packages 2023-11-15T06:23:27.370 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:23:27.370 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 1.9 M 2023-11-15T06:23:27.371 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 24 M 2023-11-15T06:23:27.371 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2023-11-15T06:23:27.497 INFO:teuthology.orchestra.run.smithi062.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 642 kB/s | 27 kB 00:00 2023-11-15T06:23:27.517 INFO:teuthology.orchestra.run.smithi081.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 637 kB/s | 27 kB 00:00 2023-11-15T06:23:27.606 INFO:teuthology.orchestra.run.smithi062.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 133 kB/s | 20 kB 00:00 2023-11-15T06:23:27.639 INFO:teuthology.orchestra.run.smithi062.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 144 kB/s | 27 kB 00:00 2023-11-15T06:23:27.676 INFO:teuthology.orchestra.run.smithi081.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 132 kB/s | 27 kB 00:00 2023-11-15T06:23:27.715 INFO:teuthology.orchestra.run.smithi062.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 1.0 MB/s | 110 kB 00:00 2023-11-15T06:23:27.809 INFO:teuthology.orchestra.run.smithi081.stdout:(3/13): python3-pyasn1-modules-0.3.7-6.el8.noar 824 kB/s | 110 kB 00:00 2023-11-15T06:23:27.943 INFO:teuthology.orchestra.run.smithi081.stdout:(4/13): python3-jsonpointer-1.10-11.el8.noarch. 43 kB/s | 20 kB 00:00 2023-11-15T06:23:28.040 INFO:teuthology.orchestra.run.smithi062.stdout:(5/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 233 kB/s | 126 kB 00:00 2023-11-15T06:23:28.065 INFO:teuthology.orchestra.run.smithi062.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 122 kB/s | 43 kB 00:00 2023-11-15T06:23:28.091 INFO:teuthology.orchestra.run.smithi062.stdout:(7/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 343 kB/s | 155 kB 00:00 2023-11-15T06:23:28.101 INFO:teuthology.orchestra.run.smithi081.stdout:(5/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 216 kB/s | 126 kB 00:00 2023-11-15T06:23:28.227 INFO:teuthology.orchestra.run.smithi081.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 371 kB/s | 155 kB 00:00 2023-11-15T06:23:28.260 INFO:teuthology.orchestra.run.smithi081.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 134 kB/s | 43 kB 00:00 2023-11-15T06:23:28.278 INFO:teuthology.orchestra.run.smithi081.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 188 kB/s | 33 kB 00:00 2023-11-15T06:23:28.386 INFO:teuthology.orchestra.run.smithi081.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 102 kB/s | 16 kB 00:00 2023-11-15T06:23:28.478 INFO:teuthology.orchestra.run.smithi081.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 374 kB/s | 81 kB 00:00 2023-11-15T06:23:28.503 INFO:teuthology.orchestra.run.smithi081.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 549 kB/s | 64 kB 00:00 2023-11-15T06:23:28.562 INFO:teuthology.orchestra.run.smithi081.stdout:(12/13): python3-kubernetes-11.0.0-6.el8.noarch 4.1 MB/s | 1.2 MB 00:00 2023-11-15T06:23:28.587 INFO:teuthology.orchestra.run.smithi081.stdout:(13/13): python3-websocket-client-0.56.0-5.el8. 564 kB/s | 61 kB 00:00 2023-11-15T06:23:28.588 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:23:28.588 INFO:teuthology.orchestra.run.smithi081.stdout:Total 1.6 MB/s | 1.9 MB 00:01 2023-11-15T06:23:28.646 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2023-11-15T06:23:28.662 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2023-11-15T06:23:28.662 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2023-11-15T06:23:28.749 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2023-11-15T06:23:28.752 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2023-11-15T06:23:29.033 INFO:teuthology.orchestra.run.smithi062.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 17 kB/s | 16 kB 00:00 2023-11-15T06:23:29.058 INFO:teuthology.orchestra.run.smithi062.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 32 kB/s | 33 kB 00:01 2023-11-15T06:23:29.272 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2023-11-15T06:23:29.317 INFO:teuthology.orchestra.run.smithi062.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 66 kB/s | 81 kB 00:01 2023-11-15T06:23:29.496 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-11-15T06:23:29.650 INFO:teuthology.orchestra.run.smithi062.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 184 kB/s | 61 kB 00:00 2023-11-15T06:23:29.722 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-11-15T06:23:29.800 INFO:teuthology.orchestra.run.smithi062.stdout:(12/13): python3-rsa-4.9-2.el8.noarch.rpm 86 kB/s | 64 kB 00:00 2023-11-15T06:23:29.988 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-11-15T06:23:30.250 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-11-15T06:23:30.473 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-11-15T06:23:30.525 INFO:teuthology.orchestra.run.smithi062.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 800 kB/s | 1.2 MB 00:01 2023-11-15T06:23:30.528 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:23:30.529 INFO:teuthology.orchestra.run.smithi062.stdout:Total 585 kB/s | 1.9 MB 00:03 2023-11-15T06:23:30.590 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2023-11-15T06:23:30.607 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2023-11-15T06:23:30.608 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2023-11-15T06:23:30.702 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2023-11-15T06:23:30.705 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2023-11-15T06:23:30.725 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-11-15T06:23:31.052 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-11-15T06:23:31.221 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2023-11-15T06:23:31.317 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-11-15T06:23:31.442 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-11-15T06:23:31.629 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-11-15T06:23:31.869 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-11-15T06:23:32.073 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-11-15T06:23:32.271 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-11-15T06:23:32.373 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-11-15T06:23:32.490 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-11-15T06:23:32.667 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-11-15T06:23:32.751 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-11-15T06:23:32.896 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-11-15T06:23:32.981 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-11-15T06:23:33.076 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-11-15T06:23:33.269 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-11-15T06:23:33.563 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-11-15T06:23:33.563 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2023-11-15T06:23:33.563 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-11-15T06:23:33.563 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-11-15T06:23:33.564 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-11-15T06:23:33.564 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-11-15T06:23:33.564 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-11-15T06:23:33.564 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-11-15T06:23:33.564 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-11-15T06:23:33.565 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-11-15T06:23:33.565 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-11-15T06:23:33.565 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-11-15T06:23:33.565 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-11-15T06:23:34.094 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-11-15T06:23:34.359 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-11-15T06:23:34.360 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:23:34.360 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2023-11-15T06:23:34.360 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2023-11-15T06:23:34.360 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-15T06:23:34.361 INFO:teuthology.orchestra.run.smithi081.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-15T06:23:34.361 INFO:teuthology.orchestra.run.smithi081.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-15T06:23:34.361 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-15T06:23:34.361 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-15T06:23:34.361 INFO:teuthology.orchestra.run.smithi081.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-15T06:23:34.362 INFO:teuthology.orchestra.run.smithi081.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-15T06:23:34.362 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-15T06:23:34.362 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-15T06:23:34.362 INFO:teuthology.orchestra.run.smithi081.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-15T06:23:34.362 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-15T06:23:34.363 INFO:teuthology.orchestra.run.smithi081.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-15T06:23:34.363 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:23:34.363 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2023-11-15T06:23:34.364 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-11-15T06:23:34.488 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-mgr-cephadm 2023-11-15T06:23:34.601 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-11-15T06:23:34.773 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-11-15T06:23:34.923 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-11-15T06:23:35.213 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-11-15T06:23:35.213 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2023-11-15T06:23:35.214 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-11-15T06:23:35.214 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-11-15T06:23:35.214 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-11-15T06:23:35.214 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-11-15T06:23:35.215 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-11-15T06:23:35.215 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-11-15T06:23:35.215 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-11-15T06:23:35.215 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-11-15T06:23:35.215 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-11-15T06:23:35.216 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-11-15T06:23:35.216 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-11-15T06:23:35.340 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:02:05 ago on Wed 15 Nov 2023 06:21:30 AM UTC. 2023-11-15T06:23:35.986 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-11-15T06:23:35.986 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:23:35.986 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2023-11-15T06:23:35.986 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2023-11-15T06:23:35.987 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-15T06:23:35.987 INFO:teuthology.orchestra.run.smithi062.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-15T06:23:35.987 INFO:teuthology.orchestra.run.smithi062.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-15T06:23:35.987 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-15T06:23:35.987 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-15T06:23:35.988 INFO:teuthology.orchestra.run.smithi062.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-15T06:23:35.988 INFO:teuthology.orchestra.run.smithi062.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-15T06:23:35.988 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-15T06:23:35.988 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-15T06:23:35.988 INFO:teuthology.orchestra.run.smithi062.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-15T06:23:35.989 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-15T06:23:35.989 INFO:teuthology.orchestra.run.smithi062.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-15T06:23:35.989 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:23:35.989 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2023-11-15T06:23:36.117 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-mgr-cephadm 2023-11-15T06:23:36.998 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2023-11-15T06:23:36.999 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:23:37.000 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2023-11-15T06:23:37.000 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:23:37.000 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2023-11-15T06:23:37.000 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2023-11-15T06:23:37.000 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2023-11-15T06:23:37.001 INFO:teuthology.orchestra.run.smithi081.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-11-15T06:23:37.001 INFO:teuthology.orchestra.run.smithi081.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-11-15T06:23:37.001 INFO:teuthology.orchestra.run.smithi081.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-11-15T06:23:37.001 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-11-15T06:23:37.001 INFO:teuthology.orchestra.run.smithi081.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-11-15T06:23:37.002 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:23:37.002 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2023-11-15T06:23:37.002 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:23:37.002 INFO:teuthology.orchestra.run.smithi081.stdout:Install 6 Packages 2023-11-15T06:23:37.002 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:23:37.003 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 5.8 M 2023-11-15T06:23:37.003 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 24 M 2023-11-15T06:23:37.003 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2023-11-15T06:23:37.011 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:02:03 ago on Wed 15 Nov 2023 06:21:34 AM UTC. 2023-11-15T06:23:37.489 INFO:teuthology.orchestra.run.smithi081.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.7 MB/s | 89 kB 00:00 2023-11-15T06:23:37.747 INFO:teuthology.orchestra.run.smithi081.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 82 kB/s | 21 kB 00:00 2023-11-15T06:23:38.089 INFO:teuthology.orchestra.run.smithi081.stdout:(3/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.0 MB/s | 355 kB 00:00 2023-11-15T06:23:38.190 INFO:teuthology.orchestra.run.smithi081.stdout:(4/6): python3-remoto-1.2.1-1.el8.noarch.rpm 489 kB/s | 49 kB 00:00 2023-11-15T06:23:38.215 INFO:teuthology.orchestra.run.smithi081.stdout:(5/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 692 kB/s | 538 kB 00:00 2023-11-15T06:23:38.498 INFO:teuthology.orchestra.run.smithi081.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 4.5 MB/s | 4.8 MB 00:01 2023-11-15T06:23:38.499 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:23:38.500 INFO:teuthology.orchestra.run.smithi081.stdout:Total 3.9 MB/s | 5.8 MB 00:01 2023-11-15T06:23:38.554 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2023-11-15T06:23:38.563 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2023-11-15T06:23:38.564 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2023-11-15T06:23:38.674 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2023-11-15T06:23:38.677 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2023-11-15T06:23:38.801 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2023-11-15T06:23:38.802 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:23:38.802 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2023-11-15T06:23:38.803 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:23:38.803 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2023-11-15T06:23:38.803 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2023-11-15T06:23:38.803 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2023-11-15T06:23:38.803 INFO:teuthology.orchestra.run.smithi062.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-11-15T06:23:38.804 INFO:teuthology.orchestra.run.smithi062.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-11-15T06:23:38.804 INFO:teuthology.orchestra.run.smithi062.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-11-15T06:23:38.804 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-11-15T06:23:38.804 INFO:teuthology.orchestra.run.smithi062.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-11-15T06:23:38.805 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:23:38.805 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2023-11-15T06:23:38.805 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:23:38.805 INFO:teuthology.orchestra.run.smithi062.stdout:Install 6 Packages 2023-11-15T06:23:38.806 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:23:38.806 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 5.8 M 2023-11-15T06:23:38.807 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 24 M 2023-11-15T06:23:38.807 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2023-11-15T06:23:38.946 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2023-11-15T06:23:39.184 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-11-15T06:23:39.310 INFO:teuthology.orchestra.run.smithi062.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 837 kB/s | 89 kB 00:00 2023-11-15T06:23:39.494 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-11-15T06:23:39.561 INFO:teuthology.orchestra.run.smithi062.stdout:(2/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.5 MB/s | 538 kB 00:00 2023-11-15T06:23:39.586 INFO:teuthology.orchestra.run.smithi062.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 77 kB/s | 21 kB 00:00 2023-11-15T06:23:39.795 INFO:teuthology.orchestra.run.smithi062.stdout:(4/6): python3-babel-2.5.1-7.el8.noarch.rpm 8.1 MB/s | 4.8 MB 00:00 2023-11-15T06:23:39.920 INFO:teuthology.orchestra.run.smithi062.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 146 kB/s | 49 kB 00:00 2023-11-15T06:23:39.978 INFO:teuthology.orchestra.run.smithi062.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 851 kB/s | 355 kB 00:00 2023-11-15T06:23:39.979 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:23:39.979 INFO:teuthology.orchestra.run.smithi062.stdout:Total 4.9 MB/s | 5.8 MB 00:01 2023-11-15T06:23:40.034 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2023-11-15T06:23:40.044 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2023-11-15T06:23:40.044 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2023-11-15T06:23:40.173 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2023-11-15T06:23:40.176 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2023-11-15T06:23:40.450 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-11-15T06:23:40.453 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2023-11-15T06:23:40.710 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-11-15T06:23:40.792 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-11-15T06:23:40.977 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-11-15T06:23:41.010 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-11-15T06:23:41.234 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-11-15T06:23:41.524 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-11-15T06:23:41.524 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2023-11-15T06:23:41.524 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-11-15T06:23:41.525 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-11-15T06:23:41.525 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-11-15T06:23:41.525 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-11-15T06:23:41.930 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-11-15T06:23:42.199 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-11-15T06:23:42.200 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:23:42.200 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2023-11-15T06:23:42.200 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2023-11-15T06:23:42.200 INFO:teuthology.orchestra.run.smithi081.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2023-11-15T06:23:42.200 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2023-11-15T06:23:42.201 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:23:42.201 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2023-11-15T06:23:42.297 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-11-15T06:23:42.328 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-fuse 2023-11-15T06:23:42.532 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-11-15T06:23:42.697 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-11-15T06:23:43.008 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-11-15T06:23:43.008 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2023-11-15T06:23:43.008 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-11-15T06:23:43.009 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-11-15T06:23:43.009 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-11-15T06:23:43.009 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-11-15T06:23:43.180 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:02:13 ago on Wed 15 Nov 2023 06:21:30 AM UTC. 2023-11-15T06:23:43.721 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-11-15T06:23:43.721 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:23:43.721 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2023-11-15T06:23:43.722 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2023-11-15T06:23:43.722 INFO:teuthology.orchestra.run.smithi062.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2023-11-15T06:23:43.722 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2023-11-15T06:23:43.722 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:23:43.723 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2023-11-15T06:23:43.859 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-fuse 2023-11-15T06:23:44.768 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:02:10 ago on Wed 15 Nov 2023 06:21:34 AM UTC. 2023-11-15T06:23:44.830 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2023-11-15T06:23:44.831 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:23:44.831 INFO:teuthology.orchestra.run.smithi081.stdout: Package Architecture Version Repository Size 2023-11-15T06:23:44.832 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:23:44.832 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2023-11-15T06:23:44.832 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2023-11-15T06:23:44.832 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2023-11-15T06:23:44.832 INFO:teuthology.orchestra.run.smithi081.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-11-15T06:23:44.833 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:23:44.833 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2023-11-15T06:23:44.833 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:23:44.833 INFO:teuthology.orchestra.run.smithi081.stdout:Install 2 Packages 2023-11-15T06:23:44.833 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:23:44.834 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 968 k 2023-11-15T06:23:44.834 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 3.2 M 2023-11-15T06:23:44.835 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2023-11-15T06:23:44.925 INFO:teuthology.orchestra.run.smithi081.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 16 MB/s | 885 kB 00:00 2023-11-15T06:23:45.450 INFO:teuthology.orchestra.run.smithi081.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 143 kB/s | 83 kB 00:00 2023-11-15T06:23:45.451 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:23:45.451 INFO:teuthology.orchestra.run.smithi081.stdout:Total 1.5 MB/s | 968 kB 00:00 2023-11-15T06:23:45.461 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2023-11-15T06:23:45.483 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2023-11-15T06:23:45.483 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2023-11-15T06:23:45.571 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2023-11-15T06:23:45.574 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2023-11-15T06:23:45.737 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2023-11-15T06:23:45.983 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-11-15T06:23:46.167 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-11-15T06:23:46.553 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-11-15T06:23:46.554 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2023-11-15T06:23:46.565 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2023-11-15T06:23:46.565 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:23:46.566 INFO:teuthology.orchestra.run.smithi062.stdout: Package Architecture Version Repository Size 2023-11-15T06:23:46.566 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:23:46.566 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2023-11-15T06:23:46.566 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2023-11-15T06:23:46.567 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2023-11-15T06:23:46.567 INFO:teuthology.orchestra.run.smithi062.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-11-15T06:23:46.567 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:23:46.567 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2023-11-15T06:23:46.567 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:23:46.568 INFO:teuthology.orchestra.run.smithi062.stdout:Install 2 Packages 2023-11-15T06:23:46.568 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:23:46.568 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 968 k 2023-11-15T06:23:46.569 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 3.2 M 2023-11-15T06:23:46.569 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2023-11-15T06:23:46.662 INFO:teuthology.orchestra.run.smithi062.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 15 MB/s | 885 kB 00:00 2023-11-15T06:23:46.946 INFO:teuthology.orchestra.run.smithi062.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 244 kB/s | 83 kB 00:00 2023-11-15T06:23:46.947 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:23:46.947 INFO:teuthology.orchestra.run.smithi062.stdout:Total 2.5 MB/s | 968 kB 00:00 2023-11-15T06:23:46.957 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2023-11-15T06:23:46.979 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2023-11-15T06:23:46.979 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2023-11-15T06:23:47.071 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2023-11-15T06:23:47.074 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2023-11-15T06:23:47.253 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2023-11-15T06:23:47.272 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-11-15T06:23:47.272 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:23:47.272 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2023-11-15T06:23:47.273 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-11-15T06:23:47.273 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:23:47.273 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2023-11-15T06:23:47.518 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-11-15T06:23:47.542 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install librados-devel 2023-11-15T06:23:47.705 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-11-15T06:23:48.102 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-11-15T06:23:48.102 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2023-11-15T06:23:48.411 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:02:18 ago on Wed 15 Nov 2023 06:21:30 AM UTC. 2023-11-15T06:23:48.786 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-11-15T06:23:48.786 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:23:48.786 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2023-11-15T06:23:48.787 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-11-15T06:23:48.787 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:23:48.787 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2023-11-15T06:23:48.992 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install librados-devel 2023-11-15T06:23:49.901 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:02:15 ago on Wed 15 Nov 2023 06:21:34 AM UTC. 2023-11-15T06:23:50.062 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2023-11-15T06:23:50.063 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:23:50.063 INFO:teuthology.orchestra.run.smithi081.stdout: Package Architecture Version Repository Size 2023-11-15T06:23:50.063 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:23:50.064 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2023-11-15T06:23:50.064 INFO:teuthology.orchestra.run.smithi081.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2023-11-15T06:23:50.064 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:23:50.064 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2023-11-15T06:23:50.065 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:23:50.065 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2023-11-15T06:23:50.065 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:23:50.065 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 131 k 2023-11-15T06:23:50.066 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 439 k 2023-11-15T06:23:50.066 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2023-11-15T06:23:50.137 INFO:teuthology.orchestra.run.smithi081.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 1.7 MB/s | 131 kB 00:00 2023-11-15T06:23:50.138 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:23:50.138 INFO:teuthology.orchestra.run.smithi081.stdout:Total 1.7 MB/s | 131 kB 00:00 2023-11-15T06:23:50.139 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2023-11-15T06:23:50.144 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2023-11-15T06:23:50.144 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2023-11-15T06:23:50.192 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2023-11-15T06:23:50.195 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2023-11-15T06:23:50.303 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2023-11-15T06:23:51.687 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2023-11-15T06:23:51.687 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:23:51.688 INFO:teuthology.orchestra.run.smithi062.stdout: Package Architecture Version Repository Size 2023-11-15T06:23:51.688 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:23:51.688 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2023-11-15T06:23:51.688 INFO:teuthology.orchestra.run.smithi062.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2023-11-15T06:23:51.688 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:23:51.689 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2023-11-15T06:23:51.689 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:23:51.689 INFO:teuthology.orchestra.run.smithi062.stdout:Install 1 Package 2023-11-15T06:23:51.689 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:23:51.690 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 131 k 2023-11-15T06:23:51.690 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 439 k 2023-11-15T06:23:51.690 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2023-11-15T06:23:51.754 INFO:teuthology.orchestra.run.smithi062.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.0 MB/s | 131 kB 00:00 2023-11-15T06:23:51.755 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:23:51.755 INFO:teuthology.orchestra.run.smithi062.stdout:Total 1.9 MB/s | 131 kB 00:00 2023-11-15T06:23:51.755 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2023-11-15T06:23:51.760 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2023-11-15T06:23:51.761 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2023-11-15T06:23:51.815 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2023-11-15T06:23:51.818 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2023-11-15T06:23:51.935 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2023-11-15T06:23:52.112 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:23:52.145 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:23:52.447 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:23:52.588 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:23:53.141 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:23:53.141 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:23:53.141 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2023-11-15T06:23:53.142 INFO:teuthology.orchestra.run.smithi062.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2023-11-15T06:23:53.142 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:23:53.142 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2023-11-15T06:23:53.284 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:23:53.285 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:23:53.285 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2023-11-15T06:23:53.285 INFO:teuthology.orchestra.run.smithi081.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2023-11-15T06:23:53.285 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:23:53.285 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2023-11-15T06:23:53.353 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install libcephfs2 2023-11-15T06:23:53.564 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install libcephfs2 2023-11-15T06:23:54.260 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:02:20 ago on Wed 15 Nov 2023 06:21:34 AM UTC. 2023-11-15T06:23:54.429 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:02:24 ago on Wed 15 Nov 2023 06:21:30 AM UTC. 2023-11-15T06:23:55.808 INFO:teuthology.orchestra.run.smithi062.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T06:23:55.863 INFO:teuthology.orchestra.run.smithi081.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T06:23:56.042 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2023-11-15T06:23:56.043 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2023-11-15T06:23:56.044 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2023-11-15T06:23:56.077 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2023-11-15T06:23:56.078 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2023-11-15T06:23:56.079 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2023-11-15T06:23:56.210 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install libcephfs-devel 2023-11-15T06:23:56.272 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install libcephfs-devel 2023-11-15T06:23:57.066 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:02:27 ago on Wed 15 Nov 2023 06:21:30 AM UTC. 2023-11-15T06:23:57.177 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:02:23 ago on Wed 15 Nov 2023 06:21:34 AM UTC. 2023-11-15T06:23:58.723 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2023-11-15T06:23:58.724 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:23:58.725 INFO:teuthology.orchestra.run.smithi081.stdout: Package Architecture Version Repository Size 2023-11-15T06:23:58.725 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:23:58.725 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2023-11-15T06:23:58.725 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2023-11-15T06:23:58.726 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:23:58.726 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2023-11-15T06:23:58.726 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:23:58.726 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2023-11-15T06:23:58.727 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:23:58.727 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 24 k 2023-11-15T06:23:58.727 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 94 k 2023-11-15T06:23:58.728 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2023-11-15T06:23:58.778 INFO:teuthology.orchestra.run.smithi081.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 458 kB/s | 24 kB 00:00 2023-11-15T06:23:58.779 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:23:58.779 INFO:teuthology.orchestra.run.smithi081.stdout:Total 448 kB/s | 24 kB 00:00 2023-11-15T06:23:58.779 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2023-11-15T06:23:58.782 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2023-11-15T06:23:58.782 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2023-11-15T06:23:58.784 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2023-11-15T06:23:58.787 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2023-11-15T06:23:58.836 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2023-11-15T06:23:58.954 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2023-11-15T06:23:58.955 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:23:58.955 INFO:teuthology.orchestra.run.smithi062.stdout: Package Architecture Version Repository Size 2023-11-15T06:23:58.955 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:23:58.956 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2023-11-15T06:23:58.956 INFO:teuthology.orchestra.run.smithi062.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2023-11-15T06:23:58.957 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:23:58.957 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2023-11-15T06:23:58.957 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:23:58.957 INFO:teuthology.orchestra.run.smithi062.stdout:Install 1 Package 2023-11-15T06:23:58.958 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:23:58.958 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 24 k 2023-11-15T06:23:58.958 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 94 k 2023-11-15T06:23:58.959 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2023-11-15T06:23:58.990 INFO:teuthology.orchestra.run.smithi062.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 715 kB/s | 24 kB 00:00 2023-11-15T06:23:58.991 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:23:58.991 INFO:teuthology.orchestra.run.smithi062.stdout:Total 690 kB/s | 24 kB 00:00 2023-11-15T06:23:58.992 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2023-11-15T06:23:58.994 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2023-11-15T06:23:58.995 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2023-11-15T06:23:58.997 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2023-11-15T06:23:59.000 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2023-11-15T06:23:59.046 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2023-11-15T06:23:59.073 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:23:59.202 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:23:59.341 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:23:59.484 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:23:59.973 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:23:59.974 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:23:59.974 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2023-11-15T06:23:59.974 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2023-11-15T06:23:59.974 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:23:59.975 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2023-11-15T06:24:00.126 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install librados2 2023-11-15T06:24:00.169 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:24:00.169 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:24:00.170 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2023-11-15T06:24:00.170 INFO:teuthology.orchestra.run.smithi062.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2023-11-15T06:24:00.170 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:24:00.170 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2023-11-15T06:24:00.290 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install librados2 2023-11-15T06:24:00.982 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:02:30 ago on Wed 15 Nov 2023 06:21:30 AM UTC. 2023-11-15T06:24:01.193 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:02:27 ago on Wed 15 Nov 2023 06:21:34 AM UTC. 2023-11-15T06:24:02.435 INFO:teuthology.orchestra.run.smithi081.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T06:24:02.650 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2023-11-15T06:24:02.651 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2023-11-15T06:24:02.651 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2023-11-15T06:24:02.757 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install librbd1 2023-11-15T06:24:02.822 INFO:teuthology.orchestra.run.smithi062.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T06:24:03.052 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2023-11-15T06:24:03.053 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2023-11-15T06:24:03.053 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2023-11-15T06:24:03.150 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install librbd1 2023-11-15T06:24:03.616 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:02:33 ago on Wed 15 Nov 2023 06:21:30 AM UTC. 2023-11-15T06:24:04.056 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:02:30 ago on Wed 15 Nov 2023 06:21:34 AM UTC. 2023-11-15T06:24:05.066 INFO:teuthology.orchestra.run.smithi081.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T06:24:05.285 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2023-11-15T06:24:05.286 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2023-11-15T06:24:05.286 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2023-11-15T06:24:05.384 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install python3-rados 2023-11-15T06:24:05.611 INFO:teuthology.orchestra.run.smithi062.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T06:24:05.842 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2023-11-15T06:24:05.843 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2023-11-15T06:24:05.843 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2023-11-15T06:24:05.949 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install python3-rados 2023-11-15T06:24:06.237 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:02:36 ago on Wed 15 Nov 2023 06:21:30 AM UTC. 2023-11-15T06:24:06.844 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:02:32 ago on Wed 15 Nov 2023 06:21:34 AM UTC. 2023-11-15T06:24:07.689 INFO:teuthology.orchestra.run.smithi081.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T06:24:07.903 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2023-11-15T06:24:07.904 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2023-11-15T06:24:07.904 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2023-11-15T06:24:08.009 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install python3-rgw 2023-11-15T06:24:08.406 INFO:teuthology.orchestra.run.smithi062.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T06:24:08.636 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2023-11-15T06:24:08.638 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2023-11-15T06:24:08.638 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2023-11-15T06:24:08.747 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install python3-rgw 2023-11-15T06:24:08.867 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:02:38 ago on Wed 15 Nov 2023 06:21:30 AM UTC. 2023-11-15T06:24:09.648 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:02:35 ago on Wed 15 Nov 2023 06:21:34 AM UTC. 2023-11-15T06:24:10.313 INFO:teuthology.orchestra.run.smithi081.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T06:24:10.526 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2023-11-15T06:24:10.527 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2023-11-15T06:24:10.528 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2023-11-15T06:24:10.632 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install python3-cephfs 2023-11-15T06:24:11.205 INFO:teuthology.orchestra.run.smithi062.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T06:24:11.435 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2023-11-15T06:24:11.437 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2023-11-15T06:24:11.437 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2023-11-15T06:24:11.484 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:02:41 ago on Wed 15 Nov 2023 06:21:30 AM UTC. 2023-11-15T06:24:11.531 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install python3-cephfs 2023-11-15T06:24:12.433 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:02:38 ago on Wed 15 Nov 2023 06:21:34 AM UTC. 2023-11-15T06:24:12.928 INFO:teuthology.orchestra.run.smithi081.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T06:24:13.143 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2023-11-15T06:24:13.144 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2023-11-15T06:24:13.144 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2023-11-15T06:24:13.268 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install python3-rbd 2023-11-15T06:24:13.989 INFO:teuthology.orchestra.run.smithi062.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T06:24:14.126 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:02:44 ago on Wed 15 Nov 2023 06:21:30 AM UTC. 2023-11-15T06:24:14.222 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2023-11-15T06:24:14.223 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2023-11-15T06:24:14.223 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2023-11-15T06:24:14.323 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install python3-rbd 2023-11-15T06:24:15.220 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:02:41 ago on Wed 15 Nov 2023 06:21:34 AM UTC. 2023-11-15T06:24:15.576 INFO:teuthology.orchestra.run.smithi081.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T06:24:15.789 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2023-11-15T06:24:15.790 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2023-11-15T06:24:15.790 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2023-11-15T06:24:15.896 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install rbd-fuse 2023-11-15T06:24:16.752 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:02:46 ago on Wed 15 Nov 2023 06:21:30 AM UTC. 2023-11-15T06:24:16.765 INFO:teuthology.orchestra.run.smithi062.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T06:24:16.997 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2023-11-15T06:24:16.998 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2023-11-15T06:24:16.998 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2023-11-15T06:24:17.238 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install rbd-fuse 2023-11-15T06:24:18.142 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:02:44 ago on Wed 15 Nov 2023 06:21:34 AM UTC. 2023-11-15T06:24:18.416 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2023-11-15T06:24:18.418 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:24:18.418 INFO:teuthology.orchestra.run.smithi081.stdout: Package Architecture Version Repository Size 2023-11-15T06:24:18.418 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:24:18.418 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2023-11-15T06:24:18.418 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2023-11-15T06:24:18.419 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:24:18.419 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2023-11-15T06:24:18.419 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:24:18.419 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2023-11-15T06:24:18.419 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:24:18.420 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 89 k 2023-11-15T06:24:18.420 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 298 k 2023-11-15T06:24:18.420 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2023-11-15T06:24:18.460 INFO:teuthology.orchestra.run.smithi081.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 2.1 MB/s | 89 kB 00:00 2023-11-15T06:24:18.461 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:24:18.461 INFO:teuthology.orchestra.run.smithi081.stdout:Total 2.0 MB/s | 89 kB 00:00 2023-11-15T06:24:18.461 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2023-11-15T06:24:18.470 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2023-11-15T06:24:18.471 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2023-11-15T06:24:18.520 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2023-11-15T06:24:18.523 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2023-11-15T06:24:18.652 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2023-11-15T06:24:18.837 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:24:19.144 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:24:19.902 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:24:19.903 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:24:19.903 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2023-11-15T06:24:19.903 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2023-11-15T06:24:19.904 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:24:19.904 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2023-11-15T06:24:19.920 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2023-11-15T06:24:19.921 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:24:19.922 INFO:teuthology.orchestra.run.smithi062.stdout: Package Architecture Version Repository Size 2023-11-15T06:24:19.922 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:24:19.922 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2023-11-15T06:24:19.922 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2023-11-15T06:24:19.922 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:24:19.923 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2023-11-15T06:24:19.923 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:24:19.923 INFO:teuthology.orchestra.run.smithi062.stdout:Install 1 Package 2023-11-15T06:24:19.923 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:24:19.923 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 89 k 2023-11-15T06:24:19.924 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 298 k 2023-11-15T06:24:19.924 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2023-11-15T06:24:20.056 INFO:teuthology.orchestra.run.smithi062.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 664 kB/s | 89 kB 00:00 2023-11-15T06:24:20.057 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:24:20.057 INFO:teuthology.orchestra.run.smithi062.stdout:Total 658 kB/s | 89 kB 00:00 2023-11-15T06:24:20.058 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2023-11-15T06:24:20.068 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2023-11-15T06:24:20.068 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2023-11-15T06:24:20.121 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2023-11-15T06:24:20.124 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2023-11-15T06:24:20.151 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install rbd-mirror 2023-11-15T06:24:20.239 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2023-11-15T06:24:20.516 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:24:20.801 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:24:21.021 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:02:51 ago on Wed 15 Nov 2023 06:21:30 AM UTC. 2023-11-15T06:24:21.500 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:24:21.501 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:24:21.501 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2023-11-15T06:24:21.502 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2023-11-15T06:24:21.502 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:24:21.502 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2023-11-15T06:24:21.672 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install rbd-mirror 2023-11-15T06:24:22.572 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:02:48 ago on Wed 15 Nov 2023 06:21:34 AM UTC. 2023-11-15T06:24:22.706 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2023-11-15T06:24:22.707 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:24:22.707 INFO:teuthology.orchestra.run.smithi081.stdout: Package Architecture Version Repository Size 2023-11-15T06:24:22.707 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:24:22.708 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2023-11-15T06:24:22.708 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-11-15T06:24:22.708 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:24:22.708 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2023-11-15T06:24:22.708 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:24:22.709 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2023-11-15T06:24:22.709 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:24:22.710 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 3.7 M 2023-11-15T06:24:22.710 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 18 M 2023-11-15T06:24:22.710 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2023-11-15T06:24:22.805 INFO:teuthology.orchestra.run.smithi081.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 38 MB/s | 3.7 MB 00:00 2023-11-15T06:24:22.806 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:24:22.806 INFO:teuthology.orchestra.run.smithi081.stdout:Total 38 MB/s | 3.7 MB 00:00 2023-11-15T06:24:22.807 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2023-11-15T06:24:22.818 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2023-11-15T06:24:22.819 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2023-11-15T06:24:22.881 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2023-11-15T06:24:22.883 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2023-11-15T06:24:23.345 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2023-11-15T06:24:23.546 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:24:23.953 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:24:24.348 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2023-11-15T06:24:24.349 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:24:24.349 INFO:teuthology.orchestra.run.smithi062.stdout: Package Architecture Version Repository Size 2023-11-15T06:24:24.350 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:24:24.350 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2023-11-15T06:24:24.350 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-11-15T06:24:24.350 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:24:24.351 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2023-11-15T06:24:24.351 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:24:24.351 INFO:teuthology.orchestra.run.smithi062.stdout:Install 1 Package 2023-11-15T06:24:24.351 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:24:24.351 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 3.7 M 2023-11-15T06:24:24.352 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 18 M 2023-11-15T06:24:24.352 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2023-11-15T06:24:24.509 INFO:teuthology.orchestra.run.smithi062.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 23 MB/s | 3.7 MB 00:00 2023-11-15T06:24:24.509 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:24:24.510 INFO:teuthology.orchestra.run.smithi062.stdout:Total 23 MB/s | 3.7 MB 00:00 2023-11-15T06:24:24.510 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2023-11-15T06:24:24.523 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2023-11-15T06:24:24.524 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2023-11-15T06:24:24.592 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2023-11-15T06:24:24.595 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2023-11-15T06:24:24.608 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:24:24.609 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:24:24.609 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2023-11-15T06:24:24.609 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2023-11-15T06:24:24.609 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:24:24.610 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2023-11-15T06:24:24.852 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install rbd-nbd 2023-11-15T06:24:25.093 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2023-11-15T06:24:25.300 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:24:25.716 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:02:55 ago on Wed 15 Nov 2023 06:21:30 AM UTC. 2023-11-15T06:24:25.727 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:24:26.433 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:24:26.433 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:24:26.433 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2023-11-15T06:24:26.434 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2023-11-15T06:24:26.434 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:24:26.434 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2023-11-15T06:24:26.656 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install rbd-nbd 2023-11-15T06:24:27.389 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2023-11-15T06:24:27.390 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:24:27.391 INFO:teuthology.orchestra.run.smithi081.stdout: Package Architecture Version Repository Size 2023-11-15T06:24:27.391 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:24:27.391 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2023-11-15T06:24:27.391 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2023-11-15T06:24:27.391 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:24:27.392 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2023-11-15T06:24:27.392 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2023-11-15T06:24:27.392 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2023-11-15T06:24:27.392 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:24:27.392 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 173 k 2023-11-15T06:24:27.392 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 649 k 2023-11-15T06:24:27.393 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2023-11-15T06:24:27.465 INFO:teuthology.orchestra.run.smithi081.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 2.3 MB/s | 173 kB 00:00 2023-11-15T06:24:27.465 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:24:27.466 INFO:teuthology.orchestra.run.smithi081.stdout:Total 2.3 MB/s | 173 kB 00:00 2023-11-15T06:24:27.466 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2023-11-15T06:24:27.476 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2023-11-15T06:24:27.476 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2023-11-15T06:24:27.525 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2023-11-15T06:24:27.528 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2023-11-15T06:24:27.562 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:02:53 ago on Wed 15 Nov 2023 06:21:34 AM UTC. 2023-11-15T06:24:27.645 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2023-11-15T06:24:27.877 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:24:28.160 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:24:28.830 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:24:28.830 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:24:28.830 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2023-11-15T06:24:28.830 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2023-11-15T06:24:28.831 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:24:28.831 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2023-11-15T06:24:29.073 DEBUG:teuthology.parallel:result is None 2023-11-15T06:24:29.338 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2023-11-15T06:24:29.339 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:24:29.340 INFO:teuthology.orchestra.run.smithi062.stdout: Package Architecture Version Repository Size 2023-11-15T06:24:29.340 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:24:29.340 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2023-11-15T06:24:29.340 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2023-11-15T06:24:29.340 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:24:29.341 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2023-11-15T06:24:29.341 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2023-11-15T06:24:29.341 INFO:teuthology.orchestra.run.smithi062.stdout:Install 1 Package 2023-11-15T06:24:29.341 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:24:29.341 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 173 k 2023-11-15T06:24:29.342 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 649 k 2023-11-15T06:24:29.342 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2023-11-15T06:24:29.387 INFO:teuthology.orchestra.run.smithi062.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.6 MB/s | 173 kB 00:00 2023-11-15T06:24:29.387 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:24:29.388 INFO:teuthology.orchestra.run.smithi062.stdout:Total 3.5 MB/s | 173 kB 00:00 2023-11-15T06:24:29.388 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2023-11-15T06:24:29.400 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2023-11-15T06:24:29.400 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2023-11-15T06:24:29.453 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2023-11-15T06:24:29.456 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2023-11-15T06:24:29.575 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2023-11-15T06:24:29.732 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:24:30.032 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:24:30.727 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:24:30.728 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:24:30.728 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2023-11-15T06:24:30.728 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2023-11-15T06:24:30.728 INFO:teuthology.orchestra.run.smithi062.stdout: 2023-11-15T06:24:30.729 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2023-11-15T06:24:30.957 DEBUG:teuthology.parallel:result is None 2023-11-15T06:24:30.958 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T06:24:30.958 INFO:teuthology.packaging:ref: None 2023-11-15T06:24:30.959 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T06:24:30.959 INFO:teuthology.packaging:branch: None 2023-11-15T06:24:30.959 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T06:24:31.150 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T06:24:31.151 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T06:24:31.373 DEBUG:teuthology.orchestra.run.smithi062:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-15T06:24:31.399 INFO:teuthology.orchestra.run.smithi062.stdout:16.2.4-0.el8 2023-11-15T06:24:31.399 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2023-11-15T06:24:31.400 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2023-11-15T06:24:31.400 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T06:24:31.401 INFO:teuthology.packaging:ref: None 2023-11-15T06:24:31.401 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T06:24:31.401 INFO:teuthology.packaging:branch: None 2023-11-15T06:24:31.402 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T06:24:31.591 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T06:24:31.592 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T06:24:31.843 DEBUG:teuthology.orchestra.run.smithi081:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-15T06:24:31.871 INFO:teuthology.orchestra.run.smithi081.stdout:16.2.4-0.el8 2023-11-15T06:24:31.871 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2023-11-15T06:24:31.871 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2023-11-15T06:24:31.872 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-11-15T06:24:31.873 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-11-15T06:24:31.873 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-15T06:24:31.908 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2023-11-15T06:24:31.908 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-15T06:24:31.943 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-11-15T06:24:31.944 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-11-15T06:24:31.944 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/usr/bin/daemon-helper 2023-11-15T06:24:31.979 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-15T06:24:32.048 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2023-11-15T06:24:32.048 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/usr/bin/daemon-helper 2023-11-15T06:24:32.080 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-15T06:24:32.150 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-11-15T06:24:32.151 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-11-15T06:24:32.151 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-15T06:24:32.180 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-15T06:24:32.248 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2023-11-15T06:24:32.248 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-15T06:24:32.278 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-15T06:24:32.347 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-11-15T06:24:32.347 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-11-15T06:24:32.348 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/usr/bin/stdin-killer 2023-11-15T06:24:32.377 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-15T06:24:32.445 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2023-11-15T06:24:32.445 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/usr/bin/stdin-killer 2023-11-15T06:24:32.474 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-15T06:24:32.542 INFO:teuthology.run_tasks:Running task print... 2023-11-15T06:24:32.554 INFO:teuthology.task.print:**** done install task... 2023-11-15T06:24:32.555 INFO:teuthology.run_tasks:Running task cephadm... 2023-11-15T06:24:32.659 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '620f25c1794f62207424fc9319427943fb8d1fae'} 2023-11-15T06:24:32.660 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2023-11-15T06:24:32.660 INFO:tasks.cephadm:Cluster fsid is a3c2da76-837f-11ee-8dbe-212e2dc638e7 2023-11-15T06:24:32.660 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-11-15T06:24:32.661 INFO:tasks.cephadm:No mon roles; fabricating mons 2023-11-15T06:24:32.661 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi062': '172.21.15.62', 'mon.smithi081': '172.21.15.81'} 2023-11-15T06:24:32.661 INFO:tasks.cephadm:Normalizing hostnames... 2023-11-15T06:24:32.662 DEBUG:teuthology.orchestra.run.smithi062:> sudo hostname $(hostname -s) 2023-11-15T06:24:32.692 DEBUG:teuthology.orchestra.run.smithi081:> sudo hostname $(hostname -s) 2023-11-15T06:24:32.721 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2023-11-15T06:24:32.722 DEBUG:teuthology.orchestra.run.smithi062:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-11-15T06:24:32.941 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Nov 15 06:24 /home/ubuntu/cephtest/cephadm 2023-11-15T06:24:32.941 DEBUG:teuthology.orchestra.run.smithi081:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-11-15T06:24:33.081 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Nov 15 06:24 /home/ubuntu/cephtest/cephadm 2023-11-15T06:24:33.082 DEBUG:teuthology.orchestra.run.smithi062:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-11-15T06:24:33.103 DEBUG:teuthology.orchestra.run.smithi081:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-11-15T06:24:33.136 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /etc/ceph 2023-11-15T06:24:33.173 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /etc/ceph 2023-11-15T06:24:33.204 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 777 /etc/ceph 2023-11-15T06:24:33.243 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 777 /etc/ceph 2023-11-15T06:24:33.273 INFO:tasks.cephadm:Writing seed config... 2023-11-15T06:24:33.275 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-11-15T06:24:33.275 INFO:tasks.cephadm: override: [client] debug client = 20 2023-11-15T06:24:33.275 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-11-15T06:24:33.275 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-11-15T06:24:33.276 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-11-15T06:24:33.276 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2023-11-15T06:24:33.276 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-11-15T06:24:33.277 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2023-11-15T06:24:33.277 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-11-15T06:24:33.277 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-11-15T06:24:33.278 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-11-15T06:24:33.278 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-11-15T06:24:33.278 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-11-15T06:24:33.279 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-11-15T06:24:33.279 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-11-15T06:24:33.279 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-11-15T06:24:33.279 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-11-15T06:24:33.280 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-11-15T06:24:33.280 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-11-15T06:24:33.280 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-11-15T06:24:33.281 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-11-15T06:24:33.281 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-11-15T06:24:33.281 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2023-11-15T06:24:33.282 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2023-11-15T06:24:33.282 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2023-11-15T06:24:33.282 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-11-15T06:24:33.283 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-11-15T06:24:33.283 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2023-11-15T06:24:33.283 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2023-11-15T06:24:33.283 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-11-15T06:24:33.284 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-11-15T06:24:33.284 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2023-11-15T06:24:33.284 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-11-15T06:24:33.285 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-11-15T06:24:33.285 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-11-15T06:24:33.285 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-11-15T06:24:33.286 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-11-15T06:24:33.286 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-11-15T06:24:33.287 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-11-15T06:24:33.287 DEBUG:teuthology.orchestra.run.smithi062:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-11-15T06:24:33.306 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = a3c2da76-837f-11ee-8dbe-212e2dc638e7 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-11-15T06:24:33.308 DEBUG:teuthology.orchestra.run.smithi062:mon.smithi062> sudo journalctl -f -n 0 -u ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7@mon.smithi062.service 2023-11-15T06:24:33.349 INFO:tasks.cephadm:Bootstrapping... 2023-11-15T06:24:33.349 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.62 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-11-15T06:24:33.378 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:-- Logs begin at Wed 2023-11-15 06:09:10 UTC. -- 2023-11-15T06:24:33.614 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-11-15T06:24:34.430 INFO:teuthology.orchestra.run.smithi062.stderr:/bin/podman: 4.6.1 2023-11-15T06:24:34.442 INFO:teuthology.orchestra.run.smithi062.stderr:Verifying podman|docker is present... 2023-11-15T06:24:34.442 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-11-15T06:24:34.468 INFO:teuthology.orchestra.run.smithi062.stderr:/bin/podman: 4.6.1 2023-11-15T06:24:34.479 INFO:teuthology.orchestra.run.smithi062.stderr:Verifying lvm2 is present... 2023-11-15T06:24:34.479 INFO:teuthology.orchestra.run.smithi062.stderr:Verifying time synchronization is in place... 2023-11-15T06:24:34.480 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: systemctl is-enabled chrony.service 2023-11-15T06:24:34.487 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-11-15T06:24:34.488 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: systemctl is-active chrony.service 2023-11-15T06:24:34.494 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: inactive 2023-11-15T06:24:34.495 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: systemctl is-enabled chronyd.service 2023-11-15T06:24:34.501 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: enabled 2023-11-15T06:24:34.502 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: systemctl is-active chronyd.service 2023-11-15T06:24:34.508 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: active 2023-11-15T06:24:34.508 INFO:teuthology.orchestra.run.smithi062.stderr:Unit chronyd.service is enabled and running 2023-11-15T06:24:34.509 INFO:teuthology.orchestra.run.smithi062.stderr:Repeating the final host check... 2023-11-15T06:24:34.509 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-11-15T06:24:34.530 INFO:teuthology.orchestra.run.smithi062.stderr:/bin/podman: 4.6.1 2023-11-15T06:24:34.536 INFO:teuthology.orchestra.run.smithi062.stderr:podman|docker (/bin/podman) is present 2023-11-15T06:24:34.537 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl is present 2023-11-15T06:24:34.537 INFO:teuthology.orchestra.run.smithi062.stderr:lvcreate is present 2023-11-15T06:24:34.537 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: systemctl is-enabled chrony.service 2023-11-15T06:24:34.543 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-11-15T06:24:34.543 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: systemctl is-active chrony.service 2023-11-15T06:24:34.549 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: inactive 2023-11-15T06:24:34.549 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: systemctl is-enabled chronyd.service 2023-11-15T06:24:34.555 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: enabled 2023-11-15T06:24:34.556 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: systemctl is-active chronyd.service 2023-11-15T06:24:34.561 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: active 2023-11-15T06:24:34.562 INFO:teuthology.orchestra.run.smithi062.stderr:Unit chronyd.service is enabled and running 2023-11-15T06:24:34.563 INFO:teuthology.orchestra.run.smithi062.stderr:Host looks OK 2023-11-15T06:24:34.563 INFO:teuthology.orchestra.run.smithi062.stderr:Cluster fsid: a3c2da76-837f-11ee-8dbe-212e2dc638e7 2023-11-15T06:24:34.563 INFO:teuthology.orchestra.run.smithi062.stderr:Acquiring lock 140428132735464 on /run/cephadm/a3c2da76-837f-11ee-8dbe-212e2dc638e7.lock 2023-11-15T06:24:34.563 INFO:teuthology.orchestra.run.smithi062.stderr:Lock 140428132735464 acquired on /run/cephadm/a3c2da76-837f-11ee-8dbe-212e2dc638e7.lock 2023-11-15T06:24:34.564 INFO:teuthology.orchestra.run.smithi062.stderr:Verifying IP 172.21.15.62 port 3300 ... 2023-11-15T06:24:34.565 INFO:teuthology.orchestra.run.smithi062.stderr:Verifying IP 172.21.15.62 port 6789 ... 2023-11-15T06:24:34.565 INFO:teuthology.orchestra.run.smithi062.stderr:Base mon IP is 172.21.15.62, final addrv is [v2:172.21.15.62:3300,v1:172.21.15.62:6789] 2023-11-15T06:24:34.565 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /sbin/ip route ls 2023-11-15T06:24:34.567 INFO:teuthology.orchestra.run.smithi062.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.62 metric 100 2023-11-15T06:24:34.567 INFO:teuthology.orchestra.run.smithi062.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.62 metric 100 2023-11-15T06:24:34.567 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /sbin/ip -6 route ls 2023-11-15T06:24:34.570 INFO:teuthology.orchestra.run.smithi062.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2023-11-15T06:24:34.570 INFO:teuthology.orchestra.run.smithi062.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2023-11-15T06:24:34.570 INFO:teuthology.orchestra.run.smithi062.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1785sec hoplimit 64 pref medium 2023-11-15T06:24:34.571 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /sbin/ip -6 addr ls 2023-11-15T06:24:34.573 INFO:teuthology.orchestra.run.smithi062.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-11-15T06:24:34.573 INFO:teuthology.orchestra.run.smithi062.stderr:/sbin/ip: inet6 ::1/128 scope host 2023-11-15T06:24:34.573 INFO:teuthology.orchestra.run.smithi062.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2023-11-15T06:24:34.573 INFO:teuthology.orchestra.run.smithi062.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2023-11-15T06:24:34.574 INFO:teuthology.orchestra.run.smithi062.stderr:/sbin/ip: inet6 fe80::ec4:7aff:febb:d1c3/64 scope link 2023-11-15T06:24:34.574 INFO:teuthology.orchestra.run.smithi062.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2023-11-15T06:24:34.574 INFO:teuthology.orchestra.run.smithi062.stderr:Mon IP 172.21.15.62 is in CIDR network 172.21.0.0/20 2023-11-15T06:24:34.575 INFO:teuthology.orchestra.run.smithi062.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-11-15T06:24:34.575 INFO:teuthology.orchestra.run.smithi062.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2023-11-15T06:24:34.576 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2023-11-15T06:24:34.669 INFO:teuthology.orchestra.run.smithi062.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2023-11-15T06:24:41.191 INFO:teuthology.orchestra.run.smithi062.stderr:/bin/podman: Getting image source signatures 2023-11-15T06:24:41.198 INFO:teuthology.orchestra.run.smithi062.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2023-11-15T06:24:41.199 INFO:teuthology.orchestra.run.smithi062.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2023-11-15T06:24:53.436 INFO:teuthology.orchestra.run.smithi062.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2023-11-15T06:24:53.493 INFO:teuthology.orchestra.run.smithi062.stderr:/bin/podman: Writing manifest to image destination 2023-11-15T06:24:53.828 INFO:teuthology.orchestra.run.smithi062.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2023-11-15T06:24:53.840 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi062 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2023-11-15T06:24:54.368 INFO:teuthology.orchestra.run.smithi062.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2023-11-15T06:24:54.729 INFO:teuthology.orchestra.run.smithi062.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2023-11-15T06:24:54.729 INFO:teuthology.orchestra.run.smithi062.stderr:Extracting ceph user uid/gid from container image... 2023-11-15T06:24:54.729 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi062 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2023-11-15T06:24:55.014 INFO:teuthology.orchestra.run.smithi062.stderr:stat: 167 167 2023-11-15T06:24:55.335 INFO:teuthology.orchestra.run.smithi062.stderr:Creating initial keys... 2023-11-15T06:24:55.335 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi062 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-11-15T06:24:55.635 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-authtool: AQA3ZFRlHYPTIhAAzXQFr4H7f9P8KOI8ePBuZw== 2023-11-15T06:24:55.965 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi062 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-11-15T06:24:56.235 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-authtool: AQA4ZFRl0EwmDRAADqQPhdX4x3xKRZafHF4rVw== 2023-11-15T06:24:56.568 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi062 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-11-15T06:24:56.829 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-authtool: AQA4ZFRl4bs/MBAA88U5knmqlCDesNdFLrFHGg== 2023-11-15T06:24:57.161 INFO:teuthology.orchestra.run.smithi062.stderr:Creating initial monmap... 2023-11-15T06:24:57.161 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi062 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmp_flrxb5u:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 --addv smithi062 [v2:172.21.15.62:3300,v1:172.21.15.62:6789] /tmp/monmap 2023-11-15T06:24:57.435 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2023-11-15T06:24:57.436 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to a3c2da76-837f-11ee-8dbe-212e2dc638e7 2023-11-15T06:24:57.436 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-11-15T06:24:57.779 INFO:teuthology.orchestra.run.smithi062.stderr:monmaptool for smithi062 [v2:172.21.15.62:3300,v1:172.21.15.62:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-11-15T06:24:57.780 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/monmaptool: set fsid to a3c2da76-837f-11ee-8dbe-212e2dc638e7 2023-11-15T06:24:57.780 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-11-15T06:24:57.780 INFO:teuthology.orchestra.run.smithi062.stderr: 2023-11-15T06:24:57.780 INFO:teuthology.orchestra.run.smithi062.stderr:Creating mon... 2023-11-15T06:24:57.781 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi062 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /var/lib/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/mon.smithi062:/var/lib/ceph/mon/ceph-smithi062:z -v /tmp/ceph-tmpwv3e3xf0:/tmp/keyring:z -v /tmp/ceph-tmp_flrxb5u:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi062 --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2023-11-15T06:24:58.085 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.054+0000 7fd729e3f700 0 set uid:gid to 167:167 (ceph:ceph) 2023-11-15T06:24:58.085 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 1 imported monmap: 2023-11-15T06:24:58.086 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: epoch 0 2023-11-15T06:24:58.086 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 2023-11-15T06:24:58.086 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: last_changed 2023-11-15T06:24:57.422579+0000 2023-11-15T06:24:58.086 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: created 2023-11-15T06:24:57.422579+0000 2023-11-15T06:24:58.087 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2023-11-15T06:24:58.087 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: election_strategy: 1 2023-11-15T06:24:58.087 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.smithi062 2023-11-15T06:24:58.087 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: 2023-11-15T06:24:58.088 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 0 /usr/bin/ceph-mon: set fsid to a3c2da76-837f-11ee-8dbe-212e2dc638e7 2023-11-15T06:24:58.088 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: RocksDB version: 6.8.1 2023-11-15T06:24:58.088 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: 2023-11-15T06:24:58.088 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-11-15T06:24:58.089 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Compile date May 13 2021 2023-11-15T06:24:58.089 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: DB SUMMARY 2023-11-15T06:24:58.089 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: 2023-11-15T06:24:58.090 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi062/store.db dir, Total Num: 0, files: 2023-11-15T06:24:58.090 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: 2023-11-15T06:24:58.090 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi062/store.db: 2023-11-15T06:24:58.090 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: 2023-11-15T06:24:58.091 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.error_if_exists: 0 2023-11-15T06:24:58.091 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.create_if_missing: 1 2023-11-15T06:24:58.091 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.paranoid_checks: 1 2023-11-15T06:24:58.091 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.env: 0x561b4b3ef1c0 2023-11-15T06:24:58.091 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.fs: Posix File System 2023-11-15T06:24:58.091 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.info_log: 0x561b4d5beec0 2023-11-15T06:24:58.092 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.max_file_opening_threads: 16 2023-11-15T06:24:58.092 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.statistics: (nil) 2023-11-15T06:24:58.092 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.use_fsync: 0 2023-11-15T06:24:58.092 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.max_log_file_size: 0 2023-11-15T06:24:58.092 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-15T06:24:58.093 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.log_file_time_to_roll: 0 2023-11-15T06:24:58.093 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.keep_log_file_num: 1000 2023-11-15T06:24:58.093 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.recycle_log_file_num: 0 2023-11-15T06:24:58.093 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.allow_fallocate: 1 2023-11-15T06:24:58.093 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.allow_mmap_reads: 0 2023-11-15T06:24:58.094 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.allow_mmap_writes: 0 2023-11-15T06:24:58.094 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.use_direct_reads: 0 2023-11-15T06:24:58.094 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-15T06:24:58.098 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.create_missing_column_families: 0 2023-11-15T06:24:58.098 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.db_log_dir: 2023-11-15T06:24:58.098 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi062/store.db 2023-11-15T06:24:58.098 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.table_cache_numshardbits: 6 2023-11-15T06:24:58.099 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.max_subcompactions: 1 2023-11-15T06:24:58.099 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.max_background_flushes: -1 2023-11-15T06:24:58.100 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-11-15T06:24:58.100 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-11-15T06:24:58.100 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-15T06:24:58.100 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-15T06:24:58.100 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-11-15T06:24:58.101 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.advise_random_on_open: 1 2023-11-15T06:24:58.101 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.db_write_buffer_size: 0 2023-11-15T06:24:58.101 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.write_buffer_manager: 0x561b4d5c5530 2023-11-15T06:24:58.101 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-15T06:24:58.101 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-11-15T06:24:58.102 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-15T06:24:58.102 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.use_adaptive_mutex: 0 2023-11-15T06:24:58.102 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.rate_limiter: (nil) 2023-11-15T06:24:58.102 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-15T06:24:58.102 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.wal_recovery_mode: 2 2023-11-15T06:24:58.102 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.enable_thread_tracking: 0 2023-11-15T06:24:58.103 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.enable_pipelined_write: 0 2023-11-15T06:24:58.103 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.unordered_write: 0 2023-11-15T06:24:58.103 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-15T06:24:58.103 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-15T06:24:58.103 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-15T06:24:58.104 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-15T06:24:58.104 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.row_cache: None 2023-11-15T06:24:58.104 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.wal_filter: None 2023-11-15T06:24:58.104 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-15T06:24:58.104 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.allow_ingest_behind: 0 2023-11-15T06:24:58.105 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.preserve_deletes: 0 2023-11-15T06:24:58.105 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.two_write_queues: 0 2023-11-15T06:24:58.105 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.manual_wal_flush: 0 2023-11-15T06:24:58.105 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.atomic_flush: 0 2023-11-15T06:24:58.105 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-15T06:24:58.106 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.persist_stats_to_disk: 0 2023-11-15T06:24:58.106 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-11-15T06:24:58.106 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.log_readahead_size: 0 2023-11-15T06:24:58.106 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-11-15T06:24:58.106 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.max_background_jobs: 2 2023-11-15T06:24:58.107 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.max_background_compactions: -1 2023-11-15T06:24:58.107 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-15T06:24:58.107 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-15T06:24:58.107 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.delayed_write_rate : 16777216 2023-11-15T06:24:58.107 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.max_total_wal_size: 0 2023-11-15T06:24:58.107 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-15T06:24:58.108 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.stats_dump_period_sec: 600 2023-11-15T06:24:58.108 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.stats_persist_period_sec: 600 2023-11-15T06:24:58.108 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-15T06:24:58.108 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.max_open_files: -1 2023-11-15T06:24:58.108 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.bytes_per_sync: 0 2023-11-15T06:24:58.109 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-11-15T06:24:58.109 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-11-15T06:24:58.109 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Options.compaction_readahead_size: 0 2023-11-15T06:24:58.109 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Compression algorithms supported: 2023-11-15T06:24:58.109 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-15T06:24:58.109 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: kZSTD supported: 0 2023-11-15T06:24:58.110 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: kXpressCompression supported: 0 2023-11-15T06:24:58.110 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: kLZ4HCCompression supported: 1 2023-11-15T06:24:58.110 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: kLZ4Compression supported: 1 2023-11-15T06:24:58.110 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: kBZip2Compression supported: 0 2023-11-15T06:24:58.110 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: kZlibCompression supported: 1 2023-11-15T06:24:58.111 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: kSnappyCompression supported: 1 2023-11-15T06:24:58.111 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.055+0000 7fd729e3f700 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-11-15T06:24:58.111 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2023-11-15T06:24:58.111 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: 2023-11-15T06:24:58.111 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi062/store.db/MANIFEST-000001 2023-11-15T06:24:58.112 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: 2023-11-15T06:24:58.112 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-11-15T06:24:58.112 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: 2023-11-15T06:24:58.112 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-15T06:24:58.112 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.merge_operator: 2023-11-15T06:24:58.113 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.compaction_filter: None 2023-11-15T06:24:58.113 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.compaction_filter_factory: None 2023-11-15T06:24:58.113 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-11-15T06:24:58.113 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.table_factory: BlockBasedTable 2023-11-15T06:24:58.113 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x561b4d4d6120) 2023-11-15T06:24:58.113 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2023-11-15T06:24:58.114 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2023-11-15T06:24:58.114 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2023-11-15T06:24:58.114 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2023-11-15T06:24:58.114 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: index_type: 0 2023-11-15T06:24:58.114 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2023-11-15T06:24:58.115 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: index_shortening: 1 2023-11-15T06:24:58.115 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2023-11-15T06:24:58.115 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2023-11-15T06:24:58.115 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: checksum: 1 2023-11-15T06:24:58.115 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2023-11-15T06:24:58.116 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: block_cache: 0x561b4d50cd10 2023-11-15T06:24:58.116 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2023-11-15T06:24:58.116 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: block_cache_options: 2023-11-15T06:24:58.116 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: capacity : 536870912 2023-11-15T06:24:58.116 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2023-11-15T06:24:58.116 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2023-11-15T06:24:58.117 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2023-11-15T06:24:58.117 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2023-11-15T06:24:58.117 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2023-11-15T06:24:58.117 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: block_size: 4096 2023-11-15T06:24:58.117 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2023-11-15T06:24:58.118 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2023-11-15T06:24:58.118 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2023-11-15T06:24:58.118 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2023-11-15T06:24:58.118 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: partition_filters: 0 2023-11-15T06:24:58.119 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2023-11-15T06:24:58.119 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2023-11-15T06:24:58.119 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2023-11-15T06:24:58.119 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: verify_compression: 0 2023-11-15T06:24:58.119 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2023-11-15T06:24:58.120 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: format_version: 2 2023-11-15T06:24:58.120 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2023-11-15T06:24:58.120 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: block_align: 0 2023-11-15T06:24:58.120 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: 2023-11-15T06:24:58.120 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.write_buffer_size: 33554432 2023-11-15T06:24:58.120 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.max_write_buffer_number: 2 2023-11-15T06:24:58.121 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.compression: NoCompression 2023-11-15T06:24:58.121 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.bottommost_compression: Disabled 2023-11-15T06:24:58.121 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.prefix_extractor: nullptr 2023-11-15T06:24:58.121 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-15T06:24:58.121 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.num_levels: 7 2023-11-15T06:24:58.122 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-15T06:24:58.122 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-15T06:24:58.122 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-15T06:24:58.122 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-15T06:24:58.122 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-15T06:24:58.123 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-15T06:24:58.123 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-15T06:24:58.123 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-15T06:24:58.123 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-15T06:24:58.123 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.compression_opts.window_bits: -14 2023-11-15T06:24:58.124 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.compression_opts.level: 32767 2023-11-15T06:24:58.124 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.compression_opts.strategy: 0 2023-11-15T06:24:58.124 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-15T06:24:58.124 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-15T06:24:58.124 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.compression_opts.enabled: false 2023-11-15T06:24:58.125 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-15T06:24:58.125 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-15T06:24:58.125 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-15T06:24:58.125 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.target_file_size_base: 67108864 2023-11-15T06:24:58.125 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.target_file_size_multiplier: 1 2023-11-15T06:24:58.125 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-15T06:24:58.126 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-15T06:24:58.126 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-15T06:24:58.126 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-15T06:24:58.126 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-15T06:24:58.126 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-15T06:24:58.127 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-15T06:24:58.127 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-15T06:24:58.127 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-15T06:24:58.127 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-15T06:24:58.127 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-15T06:24:58.128 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-15T06:24:58.128 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.arena_block_size: 4194304 2023-11-15T06:24:58.128 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-15T06:24:58.128 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-15T06:24:58.128 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-11-15T06:24:58.128 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.disable_auto_compactions: 0 2023-11-15T06:24:58.129 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-15T06:24:58.129 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-15T06:24:58.129 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-15T06:24:58.129 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-15T06:24:58.129 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-15T06:24:58.130 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-15T06:24:58.130 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-15T06:24:58.130 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-15T06:24:58.130 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-15T06:24:58.130 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-15T06:24:58.130 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.table_properties_collectors: 2023-11-15T06:24:58.131 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.inplace_update_support: 0 2023-11-15T06:24:58.131 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-11-15T06:24:58.131 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-15T06:24:58.131 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-15T06:24:58.131 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.memtable_huge_page_size: 0 2023-11-15T06:24:58.132 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.bloom_locality: 0 2023-11-15T06:24:58.132 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.max_successive_merges: 0 2023-11-15T06:24:58.132 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-11-15T06:24:58.132 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.paranoid_file_checks: 0 2023-11-15T06:24:58.132 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.force_consistency_checks: 0 2023-11-15T06:24:58.133 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.report_bg_io_stats: 0 2023-11-15T06:24:58.133 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.ttl: 2592000 2023-11-15T06:24:58.133 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.056+0000 7fd729e3f700 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-11-15T06:24:58.133 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.057+0000 7fd729e3f700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi062/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-11-15T06:24:58.133 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: 2023-11-15T06:24:58.134 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.057+0000 7fd729e3f700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2023-11-15T06:24:58.134 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: 2023-11-15T06:24:58.134 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.057+0000 7fd729e3f700 4 rocksdb: DB pointer 0x561b4d5d1800 2023-11-15T06:24:58.134 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.057+0000 7fd712cd8700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-11-15T06:24:58.134 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.057+0000 7fd712cd8700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-11-15T06:24:58.135 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2023-11-15T06:24:58.135 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T06:24:58.135 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-11-15T06:24:58.135 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-15T06:24:58.135 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-15T06:24:58.135 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-11-15T06:24:58.136 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-11-15T06:24:58.136 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-15T06:24:58.136 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: 2023-11-15T06:24:58.136 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-15T06:24:58.136 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T06:24:58.137 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T06:24:58.137 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-15T06:24:58.137 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-15T06:24:58.137 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: 2023-11-15T06:24:58.137 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-15T06:24:58.137 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T06:24:58.138 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T06:24:58.138 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T06:24:58.138 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-11-15T06:24:58.138 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-15T06:24:58.138 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-11-15T06:24:58.139 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-11-15T06:24:58.139 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-11-15T06:24:58.139 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T06:24:58.139 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T06:24:58.139 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-15T06:24:58.140 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: 2023-11-15T06:24:58.140 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-11-15T06:24:58.140 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: 2023-11-15T06:24:58.140 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-15T06:24:58.140 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T06:24:58.141 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T06:24:58.141 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-15T06:24:58.141 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-15T06:24:58.141 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: 2023-11-15T06:24:58.141 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-15T06:24:58.141 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T06:24:58.142 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T06:24:58.142 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T06:24:58.142 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-11-15T06:24:58.142 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-15T06:24:58.142 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-11-15T06:24:58.143 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-11-15T06:24:58.143 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-11-15T06:24:58.143 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T06:24:58.143 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T06:24:58.143 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-15T06:24:58.143 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: 2023-11-15T06:24:58.144 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-11-15T06:24:58.144 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: 2023-11-15T06:24:58.144 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.059+0000 7fd729e3f700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2023-11-15T06:24:58.144 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.059+0000 7fd729e3f700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2023-11-15T06:24:58.144 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:24:58.059+0000 7fd729e3f700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi062 for mon.smithi062 2023-11-15T06:24:58.411 INFO:teuthology.orchestra.run.smithi062.stderr:create mon.smithi062 on 2023-11-15T06:24:58.412 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7 2023-11-15T06:24:58.417 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: systemctl disable ceph.target 2023-11-15T06:24:58.423 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-11-15T06:24:58.510 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: systemctl enable ceph.target 2023-11-15T06:24:58.516 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-11-15T06:24:58.598 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: systemctl start ceph.target 2023-11-15T06:24:58.606 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: systemctl enable ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7.target 2023-11-15T06:24:58.612 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7.target → /etc/systemd/system/ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7.target. 2023-11-15T06:24:58.613 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7.target → /etc/systemd/system/ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7.target. 2023-11-15T06:24:58.695 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: systemctl start ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7.target 2023-11-15T06:24:58.703 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: systemctl daemon-reload 2023-11-15T06:24:58.791 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: systemctl stop ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7@mon.smithi062 2023-11-15T06:24:58.818 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: systemctl reset-failed ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7@mon.smithi062 2023-11-15T06:24:58.824 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: Failed to reset failed state of unit ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7@mon.smithi062.service: Unit ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7@mon.smithi062.service not loaded. 2023-11-15T06:24:58.824 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: systemctl enable ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7@mon.smithi062 2023-11-15T06:24:58.831 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: Created symlink /etc/systemd/system/ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7.target.wants/ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7@mon.smithi062.service → /etc/systemd/system/ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7@.service. 2023-11-15T06:24:58.921 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: systemctl start ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7@mon.smithi062 2023-11-15T06:24:58.961 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:24:58 smithi062 systemd[1]: Starting Ceph mon.smithi062 for a3c2da76-837f-11ee-8dbe-212e2dc638e7... 2023-11-15T06:24:59.248 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: systemctl is-enabled firewalld.service 2023-11-15T06:24:59.256 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: disabled 2023-11-15T06:24:59.257 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: systemctl is-active firewalld.service 2023-11-15T06:24:59.264 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: inactive 2023-11-15T06:24:59.265 INFO:teuthology.orchestra.run.smithi062.stderr:firewalld.service is not enabled 2023-11-15T06:24:59.266 INFO:teuthology.orchestra.run.smithi062.stderr:Not possible to enable service . firewalld.service is not available 2023-11-15T06:24:59.266 INFO:teuthology.orchestra.run.smithi062.stderr:Not possible to open ports <[]>. firewalld.service is not available 2023-11-15T06:24:59.266 INFO:teuthology.orchestra.run.smithi062.stderr:Waiting for mon to start... 2023-11-15T06:24:59.266 INFO:teuthology.orchestra.run.smithi062.stderr:Waiting for mon... 2023-11-15T06:24:59.267 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi062 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/mon.smithi062:/var/lib/ceph/mon/ceph-smithi062:z -v /tmp/ceph-tmptvq9ksio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpf12kfgcg:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2023-11-15T06:24:59.659 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.658+0000 7fba97a66700 1 Processor -- start 2023-11-15T06:24:59.660 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.658+0000 7fba97a66700 1 -- start start 2023-11-15T06:24:59.660 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.658+0000 7fba97a66700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba900703f0 0x7fba900fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:24:59.660 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.658+0000 7fba97a66700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba900fee10 con 0x7fba900703f0 2023-11-15T06:24:59.662 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.659+0000 7fba95802700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba900703f0 0x7fba900fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:24:59.662 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.659+0000 7fba95802700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba900703f0 0x7fba900fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38712/0 (socket says 172.21.15.62:38712) 2023-11-15T06:24:59.663 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.659+0000 7fba95802700 1 -- 172.21.15.62:0/2246620254 learned_addr learned my addr 172.21.15.62:0/2246620254 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:24:59.663 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.659+0000 7fba95802700 1 -- 172.21.15.62:0/2246620254 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba900fef50 con 0x7fba900703f0 2023-11-15T06:24:59.663 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.659+0000 7fba95802700 1 --2- 172.21.15.62:0/2246620254 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba900703f0 0x7fba900fe920 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fba8c00b5f0 tx=0x7fba8c006ee0).ready entity=mon.0 client_cookie=9c593ddd5c76a672 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:24:59.663 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.659+0000 7fba94800700 1 -- 172.21.15.62:0/2246620254 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba8c00d600 con 0x7fba900703f0 2023-11-15T06:24:59.664 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.659+0000 7fba94800700 1 -- 172.21.15.62:0/2246620254 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba8c00d760 con 0x7fba900703f0 2023-11-15T06:24:59.664 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.659+0000 7fba94800700 1 -- 172.21.15.62:0/2246620254 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba8c00da30 con 0x7fba900703f0 2023-11-15T06:24:59.664 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.660+0000 7fba97a66700 1 -- 172.21.15.62:0/2246620254 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba900703f0 msgr2=0x7fba900fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:24:59.664 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.660+0000 7fba97a66700 1 --2- 172.21.15.62:0/2246620254 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba900703f0 0x7fba900fe920 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fba8c00b5f0 tx=0x7fba8c006ee0).stop 2023-11-15T06:24:59.664 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.660+0000 7fba97a66700 1 -- 172.21.15.62:0/2246620254 shutdown_connections 2023-11-15T06:24:59.665 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.660+0000 7fba97a66700 1 --2- 172.21.15.62:0/2246620254 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba900703f0 0x7fba900fe920 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:24:59.665 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.660+0000 7fba97a66700 1 -- 172.21.15.62:0/2246620254 >> 172.21.15.62:0/2246620254 conn(0x7fba900f63c0 msgr2=0x7fba900f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:24:59.665 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.660+0000 7fba97a66700 1 -- 172.21.15.62:0/2246620254 shutdown_connections 2023-11-15T06:24:59.665 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.660+0000 7fba97a66700 1 -- 172.21.15.62:0/2246620254 wait complete. 2023-11-15T06:24:59.665 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.661+0000 7fba97a66700 1 Processor -- start 2023-11-15T06:24:59.665 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.661+0000 7fba97a66700 1 -- start start 2023-11-15T06:24:59.666 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.661+0000 7fba97a66700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba900703f0 0x7fba900fc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:24:59.666 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.661+0000 7fba97a66700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba900fa820 con 0x7fba900703f0 2023-11-15T06:24:59.666 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.661+0000 7fba95802700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba900703f0 0x7fba900fc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:24:59.667 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.661+0000 7fba95802700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba900703f0 0x7fba900fc3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38722/0 (socket says 172.21.15.62:38722) 2023-11-15T06:24:59.667 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.661+0000 7fba95802700 1 -- 172.21.15.62:0/3450450930 learned_addr learned my addr 172.21.15.62:0/3450450930 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:24:59.668 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.662+0000 7fba95802700 1 -- 172.21.15.62:0/3450450930 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba8c00b200 con 0x7fba900703f0 2023-11-15T06:24:59.668 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.662+0000 7fba95802700 1 --2- 172.21.15.62:0/3450450930 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba900703f0 0x7fba900fc3a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fba8c006c70 tx=0x7fba8c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:24:59.668 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.662+0000 7fba7e7fc700 1 -- 172.21.15.62:0/3450450930 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba8c011b40 con 0x7fba900703f0 2023-11-15T06:24:59.668 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.662+0000 7fba7e7fc700 1 -- 172.21.15.62:0/3450450930 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba8c011ca0 con 0x7fba900703f0 2023-11-15T06:24:59.669 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.662+0000 7fba97a66700 1 -- 172.21.15.62:0/3450450930 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba900fa9c0 con 0x7fba900703f0 2023-11-15T06:24:59.669 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.662+0000 7fba7e7fc700 1 -- 172.21.15.62:0/3450450930 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba8c01d430 con 0x7fba900703f0 2023-11-15T06:24:59.669 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.662+0000 7fba97a66700 1 -- 172.21.15.62:0/3450450930 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba900fadd0 con 0x7fba900703f0 2023-11-15T06:24:59.669 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.662+0000 7fba7e7fc700 1 -- 172.21.15.62:0/3450450930 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fba8c00f070 con 0x7fba900703f0 2023-11-15T06:24:59.669 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.662+0000 7fba7e7fc700 1 -- 172.21.15.62:0/3450450930 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fba8c027460 con 0x7fba900703f0 2023-11-15T06:24:59.669 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.663+0000 7fba97a66700 1 -- 172.21.15.62:0/3450450930 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba90059100 con 0x7fba900703f0 2023-11-15T06:24:59.670 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.665+0000 7fba7e7fc700 1 -- 172.21.15.62:0/3450450930 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fba8c016070 con 0x7fba900703f0 2023-11-15T06:24:59.670 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:24:59 smithi062 bash[111939]: 9a330bb785805eb6e3c5e3a5978ec90ffb7f9ed863b3525673ea5e50ba256539 2023-11-15T06:24:59.670 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:24:59 smithi062 systemd[1]: Started Ceph mon.smithi062 for a3c2da76-837f-11ee-8dbe-212e2dc638e7. 2023-11-15T06:24:59.670 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:24:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[111953]: cluster 2023-11-15T06:24:59.271352+0000 mon.smithi062 (mon.0) 0 : [INF] mkfs a3c2da76-837f-11ee-8dbe-212e2dc638e7 2023-11-15T06:24:59.671 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:24:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[111953]: cluster 2023-11-15T06:24:59.242537+0000 mon.smithi062 (mon.0) 1 : cluster [INF] mon.smithi062 is new leader, mons smithi062 in quorum (ranks 0) 2023-11-15T06:24:59.739 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.735+0000 7fba97a66700 1 -- 172.21.15.62:0/3450450930 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fba90103df0 con 0x7fba900703f0 2023-11-15T06:24:59.739 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.735+0000 7fba7e7fc700 1 -- 172.21.15.62:0/3450450930 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7fba8c036070 con 0x7fba900703f0 2023-11-15T06:24:59.739 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: cluster: 2023-11-15T06:24:59.739 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: id: a3c2da76-837f-11ee-8dbe-212e2dc638e7 2023-11-15T06:24:59.740 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: health: HEALTH_OK 2023-11-15T06:24:59.740 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.740 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: services: 2023-11-15T06:24:59.740 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi062 (age 0.467873s) 2023-11-15T06:24:59.740 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: mgr: no daemons active 2023-11-15T06:24:59.741 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2023-11-15T06:24:59.741 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.741 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: data: 2023-11-15T06:24:59.741 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2023-11-15T06:24:59.742 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2023-11-15T06:24:59.742 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2023-11-15T06:24:59.742 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: pgs: 2023-11-15T06:24:59.742 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.742 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.736+0000 7fba97a66700 1 -- 172.21.15.62:0/3450450930 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba900703f0 msgr2=0x7fba900fc3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:24:59.743 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.736+0000 7fba97a66700 1 --2- 172.21.15.62:0/3450450930 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba900703f0 0x7fba900fc3a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fba8c006c70 tx=0x7fba8c016000).stop 2023-11-15T06:24:59.743 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.736+0000 7fba97a66700 1 -- 172.21.15.62:0/3450450930 shutdown_connections 2023-11-15T06:24:59.743 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.736+0000 7fba97a66700 1 --2- 172.21.15.62:0/3450450930 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba900703f0 0x7fba900fc3a0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:24:59.743 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.736+0000 7fba97a66700 1 -- 172.21.15.62:0/3450450930 >> 172.21.15.62:0/3450450930 conn(0x7fba900f63c0 msgr2=0x7fba900f8100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:24:59.743 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.736+0000 7fba97a66700 1 -- 172.21.15.62:0/3450450930 shutdown_connections 2023-11-15T06:24:59.743 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:24:59.736+0000 7fba97a66700 1 -- 172.21.15.62:0/3450450930 wait complete. 2023-11-15T06:25:00.087 INFO:teuthology.orchestra.run.smithi062.stderr:mon is available 2023-11-15T06:25:00.087 INFO:teuthology.orchestra.run.smithi062.stderr:Assimilating anything we can from ceph.conf... 2023-11-15T06:25:00.087 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi062 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptvq9ksio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpf12kfgcg:/etc/ceph/ceph.conf:z -v /var/lib/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/mon.smithi062:/var/lib/ceph/mon/ceph-smithi062:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi062/config 2023-11-15T06:25:00.486 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.476+0000 7fa1e1c52700 1 Processor -- start 2023-11-15T06:25:00.486 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.476+0000 7fa1e1c52700 1 -- start start 2023-11-15T06:25:00.486 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.477+0000 7fa1e1c52700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1dc0703f0 0x7fa1dc0fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:00.487 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.477+0000 7fa1e1c52700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1dc0fee10 con 0x7fa1dc0703f0 2023-11-15T06:25:00.487 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.477+0000 7fa1db7fe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1dc0703f0 0x7fa1dc0fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:00.488 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.477+0000 7fa1db7fe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1dc0703f0 0x7fa1dc0fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38726/0 (socket says 172.21.15.62:38726) 2023-11-15T06:25:00.488 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.477+0000 7fa1db7fe700 1 -- 172.21.15.62:0/2220241666 learned_addr learned my addr 172.21.15.62:0/2220241666 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:00.488 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.477+0000 7fa1db7fe700 1 -- 172.21.15.62:0/2220241666 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1dc0fef50 con 0x7fa1dc0703f0 2023-11-15T06:25:00.489 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.478+0000 7fa1db7fe700 1 --2- 172.21.15.62:0/2220241666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1dc0703f0 0x7fa1dc0fe920 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fa1cc00b5f0 tx=0x7fa1cc006ee0).ready entity=mon.0 client_cookie=b413590cf42c2aa9 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:00.489 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.478+0000 7fa1da7fc700 1 -- 172.21.15.62:0/2220241666 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1cc00d600 con 0x7fa1dc0703f0 2023-11-15T06:25:00.490 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.478+0000 7fa1da7fc700 1 -- 172.21.15.62:0/2220241666 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1cc00d760 con 0x7fa1dc0703f0 2023-11-15T06:25:00.490 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.478+0000 7fa1da7fc700 1 -- 172.21.15.62:0/2220241666 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1cc00da30 con 0x7fa1dc0703f0 2023-11-15T06:25:00.490 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.478+0000 7fa1e1c52700 1 -- 172.21.15.62:0/2220241666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1dc0703f0 msgr2=0x7fa1dc0fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:00.491 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.478+0000 7fa1e1c52700 1 --2- 172.21.15.62:0/2220241666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1dc0703f0 0x7fa1dc0fe920 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fa1cc00b5f0 tx=0x7fa1cc006ee0).stop 2023-11-15T06:25:00.491 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.479+0000 7fa1e1c52700 1 -- 172.21.15.62:0/2220241666 shutdown_connections 2023-11-15T06:25:00.491 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.479+0000 7fa1e1c52700 1 --2- 172.21.15.62:0/2220241666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1dc0703f0 0x7fa1dc0fe920 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:00.492 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.479+0000 7fa1e1c52700 1 -- 172.21.15.62:0/2220241666 >> 172.21.15.62:0/2220241666 conn(0x7fa1dc0f63c0 msgr2=0x7fa1dc0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:00.492 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.479+0000 7fa1e1c52700 1 -- 172.21.15.62:0/2220241666 shutdown_connections 2023-11-15T06:25:00.493 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.479+0000 7fa1e1c52700 1 -- 172.21.15.62:0/2220241666 wait complete. 2023-11-15T06:25:00.493 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.479+0000 7fa1e1c52700 1 Processor -- start 2023-11-15T06:25:00.493 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.480+0000 7fa1e1c52700 1 -- start start 2023-11-15T06:25:00.494 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.480+0000 7fa1e1c52700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1dc0703f0 0x7fa1dc10a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:00.494 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.480+0000 7fa1e1c52700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1dc10a770 con 0x7fa1dc0703f0 2023-11-15T06:25:00.494 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.480+0000 7fa1db7fe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1dc0703f0 0x7fa1dc10a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:00.495 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.480+0000 7fa1db7fe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1dc0703f0 0x7fa1dc10a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38732/0 (socket says 172.21.15.62:38732) 2023-11-15T06:25:00.495 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.480+0000 7fa1db7fe700 1 -- 172.21.15.62:0/1884538676 learned_addr learned my addr 172.21.15.62:0/1884538676 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:00.496 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.480+0000 7fa1db7fe700 1 -- 172.21.15.62:0/1884538676 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1cc00b200 con 0x7fa1dc0703f0 2023-11-15T06:25:00.496 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.480+0000 7fa1db7fe700 1 --2- 172.21.15.62:0/1884538676 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1dc0703f0 0x7fa1dc10a280 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fa1cc006c70 tx=0x7fa1cc016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:00.496 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.480+0000 7fa1c3fff700 1 -- 172.21.15.62:0/1884538676 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1cc011b40 con 0x7fa1dc0703f0 2023-11-15T06:25:00.497 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.480+0000 7fa1c3fff700 1 -- 172.21.15.62:0/1884538676 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1cc011ca0 con 0x7fa1dc0703f0 2023-11-15T06:25:00.497 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.481+0000 7fa1e1c52700 1 -- 172.21.15.62:0/1884538676 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1dc10a970 con 0x7fa1dc0703f0 2023-11-15T06:25:00.498 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.481+0000 7fa1c3fff700 1 -- 172.21.15.62:0/1884538676 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1cc01d430 con 0x7fa1dc0703f0 2023-11-15T06:25:00.498 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.481+0000 7fa1e1c52700 1 -- 172.21.15.62:0/1884538676 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1dc10ad80 con 0x7fa1dc0703f0 2023-11-15T06:25:00.498 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.481+0000 7fa1c3fff700 1 -- 172.21.15.62:0/1884538676 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fa1cc00f070 con 0x7fa1dc0703f0 2023-11-15T06:25:00.499 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.481+0000 7fa1c3fff700 1 -- 172.21.15.62:0/1884538676 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fa1cc027460 con 0x7fa1dc0703f0 2023-11-15T06:25:00.499 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.482+0000 7fa1e1c52700 1 -- 172.21.15.62:0/1884538676 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1dc045ad0 con 0x7fa1dc0703f0 2023-11-15T06:25:00.500 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.484+0000 7fa1c3fff700 1 -- 172.21.15.62:0/1884538676 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fa1cc016070 con 0x7fa1dc0703f0 2023-11-15T06:25:00.552 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.551+0000 7fa1e1c52700 1 -- 172.21.15.62:0/1884538676 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fa1dc0ff1a0 con 0x7fa1dc0703f0 2023-11-15T06:25:00.560 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.554+0000 7fa1c3fff700 1 -- 172.21.15.62:0/1884538676 <== mon.0 v2:172.21.15.62:3300/0 7 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa1cc01d5e0 con 0x7fa1dc0703f0 2023-11-15T06:25:00.560 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.554+0000 7fa1c3fff700 1 -- 172.21.15.62:0/1884538676 <== mon.0 v2:172.21.15.62:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+464 (secure 0 0 0) 0x7fa1cc036020 con 0x7fa1dc0703f0 2023-11-15T06:25:00.561 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.561 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: [global] 2023-11-15T06:25:00.562 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: fsid = a3c2da76-837f-11ee-8dbe-212e2dc638e7 2023-11-15T06:25:00.562 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.62:3300,v1:172.21.15.62:6789] 2023-11-15T06:25:00.562 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2023-11-15T06:25:00.563 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2023-11-15T06:25:00.563 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2023-11-15T06:25:00.563 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2023-11-15T06:25:00.564 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.564 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: [mgr] 2023-11-15T06:25:00.564 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2023-11-15T06:25:00.565 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.565 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: [osd] 2023-11-15T06:25:00.566 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2023-11-15T06:25:00.566 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2023-11-15T06:25:00.567 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2023-11-15T06:25:00.567 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2023-11-15T06:25:00.567 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2023-11-15T06:25:00.568 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2023-11-15T06:25:00.568 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.555+0000 7fa1e1c52700 1 -- 172.21.15.62:0/1884538676 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1dc0703f0 msgr2=0x7fa1dc10a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:00.569 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.555+0000 7fa1e1c52700 1 --2- 172.21.15.62:0/1884538676 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1dc0703f0 0x7fa1dc10a280 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fa1cc006c70 tx=0x7fa1cc016000).stop 2023-11-15T06:25:00.569 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.555+0000 7fa1e1c52700 1 -- 172.21.15.62:0/1884538676 shutdown_connections 2023-11-15T06:25:00.569 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.555+0000 7fa1e1c52700 1 --2- 172.21.15.62:0/1884538676 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1dc0703f0 0x7fa1dc10a280 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:00.570 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.555+0000 7fa1e1c52700 1 -- 172.21.15.62:0/1884538676 >> 172.21.15.62:0/1884538676 conn(0x7fa1dc0f63c0 msgr2=0x7fa1dc0f8100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:00.570 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.555+0000 7fa1e1c52700 1 -- 172.21.15.62:0/1884538676 shutdown_connections 2023-11-15T06:25:00.570 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:00.556+0000 7fa1e1c52700 1 -- 172.21.15.62:0/1884538676 wait complete. 2023-11-15T06:25:00.583 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:00 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[111953]: cluster 2023-11-15T06:24:59.267863+0000 mon.smithi062 (mon.0) 2 : cluster [INF] mon.smithi062 is new leader, mons smithi062 in quorum (ranks 0) 2023-11-15T06:25:00.584 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:00 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[111953]: cluster 2023-11-15T06:24:59.268091+0000 mon.smithi062 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi062=[v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0]} 2023-11-15T06:25:00.584 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:00 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[111953]: cluster 2023-11-15T06:24:59.275529+0000 mon.smithi062 (mon.0) 4 : cluster [DBG] fsmap 2023-11-15T06:25:00.584 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:00 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[111953]: cluster 2023-11-15T06:24:59.277404+0000 mon.smithi062 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-11-15T06:25:00.585 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:00 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[111953]: cluster 2023-11-15T06:24:59.277811+0000 mon.smithi062 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2023-11-15T06:25:00.585 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:00 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[111953]: audit 2023-11-15T06:24:59.735700+0000 mon.smithi062 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.62:0/3450450930' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-11-15T06:25:00.912 INFO:teuthology.orchestra.run.smithi062.stderr:Generating new minimal ceph.conf... 2023-11-15T06:25:00.912 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi062 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptvq9ksio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpf12kfgcg:/etc/ceph/ceph.conf:z -v /var/lib/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/mon.smithi062:/var/lib/ceph/mon/ceph-smithi062:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi062/config 2023-11-15T06:25:01.267 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.264+0000 7fd7c5228700 1 Processor -- start 2023-11-15T06:25:01.267 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.264+0000 7fd7c5228700 1 -- start start 2023-11-15T06:25:01.268 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.264+0000 7fd7c5228700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7c00703d0 0x7fd7c00fe790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:01.268 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.264+0000 7fd7c5228700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c00fec80 con 0x7fd7c00703d0 2023-11-15T06:25:01.268 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.264+0000 7fd7bffff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7c00703d0 0x7fd7c00fe790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:01.268 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.264+0000 7fd7bffff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7c00703d0 0x7fd7c00fe790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38734/0 (socket says 172.21.15.62:38734) 2023-11-15T06:25:01.268 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.264+0000 7fd7bffff700 1 -- 172.21.15.62:0/2059786942 learned_addr learned my addr 172.21.15.62:0/2059786942 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:01.269 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.265+0000 7fd7bffff700 1 -- 172.21.15.62:0/2059786942 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c00fedc0 con 0x7fd7c00703d0 2023-11-15T06:25:01.269 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.265+0000 7fd7bffff700 1 --2- 172.21.15.62:0/2059786942 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7c00703d0 0x7fd7c00fe790 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fd7b000b5f0 tx=0x7fd7b0006ee0).ready entity=mon.0 client_cookie=5849ca5e47e43bfa server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:01.269 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.265+0000 7fd7beffd700 1 -- 172.21.15.62:0/2059786942 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7b000d600 con 0x7fd7c00703d0 2023-11-15T06:25:01.269 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.265+0000 7fd7beffd700 1 -- 172.21.15.62:0/2059786942 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd7b000dbd0 con 0x7fd7c00703d0 2023-11-15T06:25:01.269 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.265+0000 7fd7beffd700 1 -- 172.21.15.62:0/2059786942 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7b0011c00 con 0x7fd7c00703d0 2023-11-15T06:25:01.269 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.265+0000 7fd7c5228700 1 -- 172.21.15.62:0/2059786942 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7c00703d0 msgr2=0x7fd7c00fe790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:01.270 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.265+0000 7fd7c5228700 1 --2- 172.21.15.62:0/2059786942 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7c00703d0 0x7fd7c00fe790 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fd7b000b5f0 tx=0x7fd7b0006ee0).stop 2023-11-15T06:25:01.270 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.265+0000 7fd7c5228700 1 -- 172.21.15.62:0/2059786942 shutdown_connections 2023-11-15T06:25:01.270 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.265+0000 7fd7c5228700 1 --2- 172.21.15.62:0/2059786942 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7c00703d0 0x7fd7c00fe790 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:01.270 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.265+0000 7fd7c5228700 1 -- 172.21.15.62:0/2059786942 >> 172.21.15.62:0/2059786942 conn(0x7fd7c00f6210 msgr2=0x7fd7c00f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:01.270 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.265+0000 7fd7c5228700 1 -- 172.21.15.62:0/2059786942 shutdown_connections 2023-11-15T06:25:01.271 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.265+0000 7fd7c5228700 1 -- 172.21.15.62:0/2059786942 wait complete. 2023-11-15T06:25:01.272 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.266+0000 7fd7c5228700 1 Processor -- start 2023-11-15T06:25:01.273 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.266+0000 7fd7c5228700 1 -- start start 2023-11-15T06:25:01.273 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.266+0000 7fd7c5228700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7c00703d0 0x7fd7c00fc210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:01.273 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.266+0000 7fd7c5228700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c00fa690 con 0x7fd7c00703d0 2023-11-15T06:25:01.273 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.266+0000 7fd7bffff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7c00703d0 0x7fd7c00fc210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:01.274 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.266+0000 7fd7bffff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7c00703d0 0x7fd7c00fc210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38744/0 (socket says 172.21.15.62:38744) 2023-11-15T06:25:01.274 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.267+0000 7fd7bffff700 1 -- 172.21.15.62:0/3642596673 learned_addr learned my addr 172.21.15.62:0/3642596673 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:01.274 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.267+0000 7fd7bffff700 1 -- 172.21.15.62:0/3642596673 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7b000b200 con 0x7fd7c00703d0 2023-11-15T06:25:01.274 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.267+0000 7fd7bffff700 1 --2- 172.21.15.62:0/3642596673 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7c00703d0 0x7fd7c00fc210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fd7b000b190 tx=0x7fd7b0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:01.274 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.267+0000 7fd7bcff9700 1 -- 172.21.15.62:0/3642596673 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7b001b410 con 0x7fd7c00703d0 2023-11-15T06:25:01.274 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.267+0000 7fd7c5228700 1 -- 172.21.15.62:0/3642596673 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7c00fa830 con 0x7fd7c00703d0 2023-11-15T06:25:01.275 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.267+0000 7fd7bcff9700 1 -- 172.21.15.62:0/3642596673 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd7b001b9e0 con 0x7fd7c00703d0 2023-11-15T06:25:01.275 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.267+0000 7fd7bcff9700 1 -- 172.21.15.62:0/3642596673 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7b001a880 con 0x7fd7c00703d0 2023-11-15T06:25:01.275 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.267+0000 7fd7c5228700 1 -- 172.21.15.62:0/3642596673 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7c00fac40 con 0x7fd7c00703d0 2023-11-15T06:25:01.275 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.267+0000 7fd7bcff9700 1 -- 172.21.15.62:0/3642596673 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fd7b000f070 con 0x7fd7c00703d0 2023-11-15T06:25:01.275 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.267+0000 7fd7bcff9700 1 -- 172.21.15.62:0/3642596673 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fd7b0025c90 con 0x7fd7c00703d0 2023-11-15T06:25:01.276 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.268+0000 7fd7c5228700 1 -- 172.21.15.62:0/3642596673 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7c0045ad0 con 0x7fd7c00703d0 2023-11-15T06:25:01.276 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.271+0000 7fd7bcff9700 1 -- 172.21.15.62:0/3642596673 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fd7b0016070 con 0x7fd7c00703d0 2023-11-15T06:25:01.341 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.337+0000 7fd7c5228700 1 -- 172.21.15.62:0/3642596673 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fd7c00008d0 con 0x7fd7c00703d0 2023-11-15T06:25:01.341 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.338+0000 7fd7bcff9700 1 -- 172.21.15.62:0/3642596673 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7fd7b001bb50 con 0x7fd7c00703d0 2023-11-15T06:25:01.342 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.338+0000 7fd7c5228700 1 -- 172.21.15.62:0/3642596673 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7c00703d0 msgr2=0x7fd7c00fc210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:01.342 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.338+0000 7fd7c5228700 1 --2- 172.21.15.62:0/3642596673 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7c00703d0 0x7fd7c00fc210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fd7b000b190 tx=0x7fd7b0006ee0).stop 2023-11-15T06:25:01.343 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.338+0000 7fd7c5228700 1 -- 172.21.15.62:0/3642596673 shutdown_connections 2023-11-15T06:25:01.343 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.338+0000 7fd7c5228700 1 --2- 172.21.15.62:0/3642596673 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7c00703d0 0x7fd7c00fc210 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:01.344 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.338+0000 7fd7c5228700 1 -- 172.21.15.62:0/3642596673 >> 172.21.15.62:0/3642596673 conn(0x7fd7c00f6210 msgr2=0x7fd7c00f7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:01.344 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.339+0000 7fd7c5228700 1 -- 172.21.15.62:0/3642596673 shutdown_connections 2023-11-15T06:25:01.344 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:01.339+0000 7fd7c5228700 1 -- 172.21.15.62:0/3642596673 wait complete. 2023-11-15T06:25:01.585 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:01 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[111953]: audit 2023-11-15T06:25:00.551586+0000 mon.smithi062 (mon.0) 8 : audit [INF] from='client.? 172.21.15.62:0/1884538676' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-11-15T06:25:01.586 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:01 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[111953]: audit 2023-11-15T06:25:00.554720+0000 mon.smithi062 (mon.0) 9 : audit [INF] from='client.? 172.21.15.62:0/1884538676' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-11-15T06:25:01.737 INFO:teuthology.orchestra.run.smithi062.stderr:Restarting the monitor... 2023-11-15T06:25:01.738 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: systemctl restart ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7@mon.smithi062 2023-11-15T06:25:01.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:01 smithi062 systemd[1]: Stopping Ceph mon.smithi062 for a3c2da76-837f-11ee-8dbe-212e2dc638e7... 2023-11-15T06:25:01.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:01 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[111953]: debug 2023-11-15T06:25:01.817+0000 7f7a3dc8b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi062 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-11-15T06:25:01.919 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:01 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[111953]: debug 2023-11-15T06:25:01.817+0000 7f7a3dc8b700 -1 mon.smithi062@0(leader) e1 *** Got Signal Terminated *** 2023-11-15T06:25:02.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 podman[112276]: ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062 2023-11-15T06:25:02.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 systemd[1]: ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7@mon.smithi062.service: Succeeded. 2023-11-15T06:25:02.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 systemd[1]: Stopped Ceph mon.smithi062 for a3c2da76-837f-11ee-8dbe-212e2dc638e7. 2023-11-15T06:25:02.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 systemd[1]: Starting Ceph mon.smithi062 for a3c2da76-837f-11ee-8dbe-212e2dc638e7... 2023-11-15T06:25:02.562 INFO:teuthology.orchestra.run.smithi062.stderr:Setting mon public_network to 172.21.0.0/20 2023-11-15T06:25:02.563 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi062 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptvq9ksio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpf12kfgcg:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2023-11-15T06:25:02.913 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.911+0000 7f26a1811700 1 Processor -- start 2023-11-15T06:25:02.917 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.911+0000 7f26a1811700 1 -- start start 2023-11-15T06:25:02.918 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.911+0000 7f26a1811700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f269c0fab90 0x7f269c0fafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:02.918 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.911+0000 7f26a1811700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f269c0fb4a0 con 0x7f269c0fab90 2023-11-15T06:25:02.918 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.912+0000 7f269affd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f269c0fab90 0x7f269c0fafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:02.918 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.912+0000 7f269affd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f269c0fab90 0x7f269c0fafb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38754/0 (socket says 172.21.15.62:38754) 2023-11-15T06:25:02.918 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.912+0000 7f269affd700 1 -- 172.21.15.62:0/3436185926 learned_addr learned my addr 172.21.15.62:0/3436185926 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:02.919 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.912+0000 7f269affd700 1 -- 172.21.15.62:0/3436185926 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f269c0fb5e0 con 0x7f269c0fab90 2023-11-15T06:25:02.919 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.912+0000 7f269affd700 1 --2- 172.21.15.62:0/3436185926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f269c0fab90 0x7f269c0fafb0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f268400b5f0 tx=0x7f2684006ee0).ready entity=mon.0 client_cookie=3bb8a3ecbf426a11 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:02.919 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.912+0000 7f2699ffb700 1 -- 172.21.15.62:0/3436185926 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f268400d600 con 0x7f269c0fab90 2023-11-15T06:25:02.919 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.912+0000 7f2699ffb700 1 -- 172.21.15.62:0/3436185926 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f268400dbd0 con 0x7f269c0fab90 2023-11-15T06:25:02.919 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.912+0000 7f2699ffb700 1 -- 172.21.15.62:0/3436185926 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2684011c00 con 0x7f269c0fab90 2023-11-15T06:25:02.920 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.913+0000 7f26a1811700 1 -- 172.21.15.62:0/3436185926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f269c0fab90 msgr2=0x7f269c0fafb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:02.920 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.913+0000 7f26a1811700 1 --2- 172.21.15.62:0/3436185926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f269c0fab90 0x7f269c0fafb0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f268400b5f0 tx=0x7f2684006ee0).stop 2023-11-15T06:25:02.920 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.913+0000 7f26a1811700 1 -- 172.21.15.62:0/3436185926 shutdown_connections 2023-11-15T06:25:02.920 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.913+0000 7f26a1811700 1 --2- 172.21.15.62:0/3436185926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f269c0fab90 0x7f269c0fafb0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:02.921 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.913+0000 7f26a1811700 1 -- 172.21.15.62:0/3436185926 >> 172.21.15.62:0/3436185926 conn(0x7f269c0f63b0 msgr2=0x7f269c0f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:02.921 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.913+0000 7f26a1811700 1 -- 172.21.15.62:0/3436185926 shutdown_connections 2023-11-15T06:25:02.921 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.913+0000 7f26a1811700 1 -- 172.21.15.62:0/3436185926 wait complete. 2023-11-15T06:25:02.921 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.914+0000 7f26a1811700 1 Processor -- start 2023-11-15T06:25:02.921 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.914+0000 7f26a1811700 1 -- start start 2023-11-15T06:25:02.922 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.914+0000 7f26a1811700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f269c0fab90 0x7f269c100e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:02.922 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.914+0000 7f26a1811700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f269c0ff2a0 con 0x7f269c0fab90 2023-11-15T06:25:02.922 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.914+0000 7f269affd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f269c0fab90 0x7f269c100e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:02.922 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.914+0000 7f269affd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f269c0fab90 0x7f269c100e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38768/0 (socket says 172.21.15.62:38768) 2023-11-15T06:25:02.922 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.914+0000 7f269affd700 1 -- 172.21.15.62:0/913990642 learned_addr learned my addr 172.21.15.62:0/913990642 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:02.923 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.914+0000 7f269affd700 1 -- 172.21.15.62:0/913990642 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f268400b200 con 0x7f269c0fab90 2023-11-15T06:25:02.923 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.915+0000 7f269affd700 1 --2- 172.21.15.62:0/913990642 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f269c0fab90 0x7f269c100e20 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f268400b1d0 tx=0x7f2684006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:02.923 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.915+0000 7f26937fe700 1 -- 172.21.15.62:0/913990642 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f268401b410 con 0x7f269c0fab90 2023-11-15T06:25:02.923 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.915+0000 7f26937fe700 1 -- 172.21.15.62:0/913990642 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f268401b9e0 con 0x7f269c0fab90 2023-11-15T06:25:02.924 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.915+0000 7f26a1811700 1 -- 172.21.15.62:0/913990642 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f269c0ff440 con 0x7f269c0fab90 2023-11-15T06:25:02.924 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.915+0000 7f26937fe700 1 -- 172.21.15.62:0/913990642 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f268401a840 con 0x7f269c0fab90 2023-11-15T06:25:02.924 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.915+0000 7f26a1811700 1 -- 172.21.15.62:0/913990642 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f269c0ff850 con 0x7f269c0fab90 2023-11-15T06:25:02.924 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.916+0000 7f26937fe700 1 -- 172.21.15.62:0/913990642 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f268400f070 con 0x7f269c0fab90 2023-11-15T06:25:02.924 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.916+0000 7f26a1811700 1 -- 172.21.15.62:0/913990642 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f269c045ad0 con 0x7f269c0fab90 2023-11-15T06:25:02.925 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.916+0000 7f26937fe700 1 -- 172.21.15.62:0/913990642 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f26840257d0 con 0x7f269c0fab90 2023-11-15T06:25:02.927 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 bash[112348]: 5f0a428b48a453e7bca02f5613a5f62e61d9da5ec43e3f787d6153fdb58191c3 2023-11-15T06:25:02.927 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.558+0000 7fc758064700 0 set uid:gid to 167:167 (ceph:ceph) 2023-11-15T06:25:02.927 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.558+0000 7fc758064700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2023-11-15T06:25:02.927 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.558+0000 7fc758064700 0 pidfile_write: ignore empty --pid-file 2023-11-15T06:25:02.928 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 systemd[1]: Started Ceph mon.smithi062 for a3c2da76-837f-11ee-8dbe-212e2dc638e7. 2023-11-15T06:25:02.928 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.563+0000 7fc758064700 0 load: jerasure load: lrc load: isa 2023-11-15T06:25:02.928 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: RocksDB version: 6.8.1 2023-11-15T06:25:02.928 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: 2023-11-15T06:25:02.928 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-11-15T06:25:02.928 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Compile date May 13 2021 2023-11-15T06:25:02.929 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: DB SUMMARY 2023-11-15T06:25:02.929 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: 2023-11-15T06:25:02.929 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: CURRENT file: CURRENT 2023-11-15T06:25:02.929 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: 2023-11-15T06:25:02.930 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: IDENTITY file: IDENTITY 2023-11-15T06:25:02.930 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: 2023-11-15T06:25:02.930 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2023-11-15T06:25:02.931 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: 2023-11-15T06:25:02.931 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi062/store.db dir, Total Num: 1, files: 000004.sst 2023-11-15T06:25:02.931 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: 2023-11-15T06:25:02.931 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi062/store.db: 000006.log size: 92918 ; 2023-11-15T06:25:02.931 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: 2023-11-15T06:25:02.932 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.error_if_exists: 0 2023-11-15T06:25:02.932 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.create_if_missing: 0 2023-11-15T06:25:02.932 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.paranoid_checks: 1 2023-11-15T06:25:02.932 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.env: 0x562d34ee81c0 2023-11-15T06:25:02.932 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.fs: Posix File System 2023-11-15T06:25:02.933 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.info_log: 0x562d37665960 2023-11-15T06:25:02.933 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.max_file_opening_threads: 16 2023-11-15T06:25:02.933 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.statistics: (nil) 2023-11-15T06:25:02.933 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.use_fsync: 0 2023-11-15T06:25:02.933 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.max_log_file_size: 0 2023-11-15T06:25:02.933 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-15T06:25:02.934 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.log_file_time_to_roll: 0 2023-11-15T06:25:02.934 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.keep_log_file_num: 1000 2023-11-15T06:25:02.934 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.recycle_log_file_num: 0 2023-11-15T06:25:02.934 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.allow_fallocate: 1 2023-11-15T06:25:02.935 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.allow_mmap_reads: 0 2023-11-15T06:25:02.935 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.allow_mmap_writes: 0 2023-11-15T06:25:02.935 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.use_direct_reads: 0 2023-11-15T06:25:02.935 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-15T06:25:02.935 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.create_missing_column_families: 0 2023-11-15T06:25:02.936 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.db_log_dir: 2023-11-15T06:25:02.936 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi062/store.db 2023-11-15T06:25:02.936 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.table_cache_numshardbits: 6 2023-11-15T06:25:02.936 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.max_subcompactions: 1 2023-11-15T06:25:02.937 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.max_background_flushes: -1 2023-11-15T06:25:02.937 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-11-15T06:25:02.937 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-11-15T06:25:02.937 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-15T06:25:02.938 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-15T06:25:02.938 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-11-15T06:25:02.938 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.advise_random_on_open: 1 2023-11-15T06:25:02.938 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.db_write_buffer_size: 0 2023-11-15T06:25:02.938 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.write_buffer_manager: 0x562d382da030 2023-11-15T06:25:02.939 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-15T06:25:02.939 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-11-15T06:25:02.939 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-15T06:25:02.939 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.use_adaptive_mutex: 0 2023-11-15T06:25:02.939 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.rate_limiter: (nil) 2023-11-15T06:25:02.940 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-15T06:25:02.940 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.wal_recovery_mode: 2 2023-11-15T06:25:02.940 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.enable_thread_tracking: 0 2023-11-15T06:25:02.940 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.enable_pipelined_write: 0 2023-11-15T06:25:02.940 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.unordered_write: 0 2023-11-15T06:25:02.940 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-15T06:25:02.941 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-15T06:25:02.941 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-15T06:25:02.941 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-15T06:25:02.941 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.row_cache: None 2023-11-15T06:25:02.941 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.wal_filter: None 2023-11-15T06:25:02.942 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-15T06:25:02.942 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.allow_ingest_behind: 0 2023-11-15T06:25:02.942 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.preserve_deletes: 0 2023-11-15T06:25:02.942 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.two_write_queues: 0 2023-11-15T06:25:02.942 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.manual_wal_flush: 0 2023-11-15T06:25:02.943 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.atomic_flush: 0 2023-11-15T06:25:02.943 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-15T06:25:02.943 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.persist_stats_to_disk: 0 2023-11-15T06:25:02.943 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-11-15T06:25:02.943 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.log_readahead_size: 0 2023-11-15T06:25:02.944 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-11-15T06:25:02.944 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.max_background_jobs: 2 2023-11-15T06:25:02.944 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.max_background_compactions: -1 2023-11-15T06:25:02.944 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-15T06:25:02.944 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-15T06:25:02.945 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.delayed_write_rate : 16777216 2023-11-15T06:25:02.945 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.max_total_wal_size: 0 2023-11-15T06:25:02.945 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-15T06:25:02.945 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.stats_dump_period_sec: 600 2023-11-15T06:25:02.945 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.stats_persist_period_sec: 600 2023-11-15T06:25:02.946 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-15T06:25:02.946 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.max_open_files: -1 2023-11-15T06:25:02.946 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.bytes_per_sync: 0 2023-11-15T06:25:02.946 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-11-15T06:25:02.947 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-11-15T06:25:02.947 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.compaction_readahead_size: 0 2023-11-15T06:25:02.947 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Compression algorithms supported: 2023-11-15T06:25:02.947 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-15T06:25:02.948 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: kZSTD supported: 0 2023-11-15T06:25:02.948 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: kXpressCompression supported: 0 2023-11-15T06:25:02.948 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: kLZ4HCCompression supported: 1 2023-11-15T06:25:02.948 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: kLZ4Compression supported: 1 2023-11-15T06:25:02.948 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: kBZip2Compression supported: 0 2023-11-15T06:25:02.949 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: kZlibCompression supported: 1 2023-11-15T06:25:02.949 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: kSnappyCompression supported: 1 2023-11-15T06:25:02.949 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-11-15T06:25:02.949 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi062/store.db/MANIFEST-000005 2023-11-15T06:25:02.949 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: 2023-11-15T06:25:02.950 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-11-15T06:25:02.950 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: 2023-11-15T06:25:02.950 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-15T06:25:02.950 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.merge_operator: 2023-11-15T06:25:02.950 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.compaction_filter: None 2023-11-15T06:25:02.951 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.compaction_filter_factory: None 2023-11-15T06:25:02.951 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-11-15T06:25:02.951 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.table_factory: BlockBasedTable 2023-11-15T06:25:02.951 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562d3757c128) 2023-11-15T06:25:02.951 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cache_index_and_filter_blocks: 1 2023-11-15T06:25:02.952 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cache_index_and_filter_blocks_with_high_priority: 0 2023-11-15T06:25:02.952 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: pin_l0_filter_and_index_blocks_in_cache: 0 2023-11-15T06:25:02.952 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: pin_top_level_index_and_filter: 1 2023-11-15T06:25:02.952 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: index_type: 0 2023-11-15T06:25:02.952 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: data_block_index_type: 0 2023-11-15T06:25:02.953 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: index_shortening: 1 2023-11-15T06:25:02.953 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: data_block_hash_table_util_ratio: 0.750000 2023-11-15T06:25:02.953 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: hash_index_allow_collision: 1 2023-11-15T06:25:02.953 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: checksum: 1 2023-11-15T06:25:02.953 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: no_block_cache: 0 2023-11-15T06:25:02.953 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: block_cache: 0x562d375b2f10 2023-11-15T06:25:02.954 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: block_cache_name: BinnedLRUCache 2023-11-15T06:25:02.954 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: block_cache_options: 2023-11-15T06:25:02.954 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: capacity : 536870912 2023-11-15T06:25:02.954 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: num_shard_bits : 4 2023-11-15T06:25:02.954 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: strict_capacity_limit : 0 2023-11-15T06:25:02.955 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: high_pri_pool_ratio: 0.000 2023-11-15T06:25:02.955 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: block_cache_compressed: (nil) 2023-11-15T06:25:02.955 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: persistent_cache: (nil) 2023-11-15T06:25:02.955 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: block_size: 4096 2023-11-15T06:25:02.955 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: block_size_deviation: 10 2023-11-15T06:25:02.956 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: block_restart_interval: 16 2023-11-15T06:25:02.956 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: index_block_restart_interval: 1 2023-11-15T06:25:02.956 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: metadata_block_size: 4096 2023-11-15T06:25:02.956 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: partition_filters: 0 2023-11-15T06:25:02.956 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: use_delta_encoding: 1 2023-11-15T06:25:02.957 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: filter_policy: rocksdb.BuiltinBloomFilter 2023-11-15T06:25:02.957 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: whole_key_filtering: 1 2023-11-15T06:25:02.957 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: verify_compression: 0 2023-11-15T06:25:02.957 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: read_amp_bytes_per_bit: 0 2023-11-15T06:25:02.957 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: format_version: 2 2023-11-15T06:25:02.958 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: enable_index_compression: 1 2023-11-15T06:25:02.958 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: block_align: 0 2023-11-15T06:25:02.958 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: 2023-11-15T06:25:02.958 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.write_buffer_size: 33554432 2023-11-15T06:25:02.959 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.max_write_buffer_number: 2 2023-11-15T06:25:02.959 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.compression: NoCompression 2023-11-15T06:25:02.959 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.bottommost_compression: Disabled 2023-11-15T06:25:02.959 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.prefix_extractor: nullptr 2023-11-15T06:25:02.959 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-15T06:25:02.959 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.num_levels: 7 2023-11-15T06:25:02.960 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-15T06:25:02.960 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-15T06:25:02.960 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-15T06:25:02.960 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-15T06:25:02.960 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-15T06:25:02.961 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-15T06:25:02.961 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-15T06:25:02.961 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-15T06:25:02.961 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-15T06:25:02.961 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.compression_opts.window_bits: -14 2023-11-15T06:25:02.962 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.compression_opts.level: 32767 2023-11-15T06:25:02.962 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.compression_opts.strategy: 0 2023-11-15T06:25:02.962 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-15T06:25:02.962 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-15T06:25:02.962 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.compression_opts.enabled: false 2023-11-15T06:25:02.963 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-15T06:25:02.963 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-15T06:25:02.963 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-15T06:25:02.963 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.target_file_size_base: 67108864 2023-11-15T06:25:02.963 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.target_file_size_multiplier: 1 2023-11-15T06:25:02.964 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-15T06:25:02.964 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-15T06:25:02.964 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-15T06:25:02.964 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-15T06:25:02.965 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-15T06:25:02.965 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-15T06:25:02.965 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-15T06:25:02.965 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-15T06:25:02.966 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-15T06:25:02.966 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-15T06:25:02.966 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-15T06:25:02.966 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-15T06:25:02.966 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.arena_block_size: 4194304 2023-11-15T06:25:02.967 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-15T06:25:02.967 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-15T06:25:02.967 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-11-15T06:25:02.967 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.disable_auto_compactions: 0 2023-11-15T06:25:02.968 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-15T06:25:02.968 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-15T06:25:02.968 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-15T06:25:02.968 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-15T06:25:02.968 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-15T06:25:02.969 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-15T06:25:02.969 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-15T06:25:02.969 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-15T06:25:02.969 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-15T06:25:02.970 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-15T06:25:02.970 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.table_properties_collectors: 2023-11-15T06:25:02.970 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.inplace_update_support: 0 2023-11-15T06:25:02.970 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-11-15T06:25:02.971 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-15T06:25:02.971 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-15T06:25:02.971 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.memtable_huge_page_size: 0 2023-11-15T06:25:02.971 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.bloom_locality: 0 2023-11-15T06:25:02.971 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.max_successive_merges: 0 2023-11-15T06:25:02.972 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-11-15T06:25:02.972 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.paranoid_file_checks: 0 2023-11-15T06:25:02.972 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.force_consistency_checks: 0 2023-11-15T06:25:02.972 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.report_bg_io_stats: 0 2023-11-15T06:25:02.972 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.ttl: 2592000 2023-11-15T06:25:02.972 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.564+0000 7fc758064700 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-11-15T06:25:02.973 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.565+0000 7fc758064700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi062/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-11-15T06:25:02.973 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: 2023-11-15T06:25:02.973 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.565+0000 7fc758064700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2023-11-15T06:25:02.973 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: 2023-11-15T06:25:02.973 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.565+0000 7fc758064700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700029502566091, "job": 1, "event": "recovery_started", "log_files": [6]} 2023-11-15T06:25:02.974 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.565+0000 7fc758064700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2023-11-15T06:25:02.974 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.566+0000 7fc758064700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2023-11-15T06:25:02.974 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.566+0000 7fc758064700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700029502567076, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 88698, "table_properties": {"data_size": 86813, "index_size": 362, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12384, "raw_average_key_size": 49, "raw_value_size": 79954, "raw_average_value_size": 321, "num_data_blocks": 11, "num_entries": 249, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1700029502, "oldest_key_time": 3, "file_creation_time": 0}} 2023-11-15T06:25:02.974 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.566+0000 7fc758064700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2023-11-15T06:25:02.975 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: 2023-11-15T06:25:02.975 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.567+0000 7fc758064700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700029502568175, "job": 1, "event": "recovery_finished"} 2023-11-15T06:25:02.975 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.568+0000 7fc758064700 4 rocksdb: DB pointer 0x562d37677800 2023-11-15T06:25:02.975 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.568+0000 7fc73ee7c700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-11-15T06:25:02.975 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.568+0000 7fc73ee7c700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-11-15T06:25:02.976 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: ** DB Stats ** 2023-11-15T06:25:02.976 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T06:25:02.976 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-11-15T06:25:02.976 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-15T06:25:02.976 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-15T06:25:02.976 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-11-15T06:25:02.977 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-11-15T06:25:02.977 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-15T06:25:02.977 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: 2023-11-15T06:25:02.977 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: ** Compaction Stats [default] ** 2023-11-15T06:25:02.977 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T06:25:02.978 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T06:25:02.978 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: L0 2/0 88.31 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.6 0.00 0.00 1 0.001 0 0 2023-11-15T06:25:02.978 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: Sum 2/0 88.31 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.6 0.00 0.00 1 0.001 0 0 2023-11-15T06:25:02.978 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.6 0.00 0.00 1 0.001 0 0 2023-11-15T06:25:02.979 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: 2023-11-15T06:25:02.979 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: ** Compaction Stats [default] ** 2023-11-15T06:25:02.979 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T06:25:02.979 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T06:25:02.979 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 143.6 0.00 0.00 1 0.001 0 0 2023-11-15T06:25:02.979 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T06:25:02.980 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: Flush(GB): cumulative 0.000, interval 0.000 2023-11-15T06:25:02.980 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-15T06:25:02.980 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: AddFile(Total Files): cumulative 0, interval 0 2023-11-15T06:25:02.981 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.918+0000 7f26937fe700 1 -- 172.21.15.62:0/913990642 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f2684016070 con 0x7f269c0fab90 2023-11-15T06:25:02.982 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: AddFile(L0 Files): cumulative 0, interval 0 2023-11-15T06:25:02.982 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: AddFile(Keys): cumulative 0, interval 0 2023-11-15T06:25:02.982 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: Cumulative compaction: 0.00 GB write, 21.25 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T06:25:02.982 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: Interval compaction: 0.00 GB write, 21.25 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T06:25:02.983 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-15T06:25:02.983 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: 2023-11-15T06:25:02.983 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: ** File Read Latency Histogram By Level [default] ** 2023-11-15T06:25:02.983 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: 2023-11-15T06:25:02.983 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: ** Compaction Stats [default] ** 2023-11-15T06:25:02.984 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T06:25:02.984 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T06:25:02.984 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: L0 2/0 88.31 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.6 0.00 0.00 1 0.001 0 0 2023-11-15T06:25:02.984 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: Sum 2/0 88.31 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.6 0.00 0.00 1 0.001 0 0 2023-11-15T06:25:02.984 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-15T06:25:02.985 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: 2023-11-15T06:25:02.985 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: ** Compaction Stats [default] ** 2023-11-15T06:25:02.985 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T06:25:02.985 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T06:25:02.985 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 143.6 0.00 0.00 1 0.001 0 0 2023-11-15T06:25:02.985 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T06:25:02.986 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: Flush(GB): cumulative 0.000, interval 0.000 2023-11-15T06:25:02.986 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-15T06:25:02.986 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: AddFile(Total Files): cumulative 0, interval 0 2023-11-15T06:25:02.986 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: AddFile(L0 Files): cumulative 0, interval 0 2023-11-15T06:25:02.986 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: AddFile(Keys): cumulative 0, interval 0 2023-11-15T06:25:02.987 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: Cumulative compaction: 0.00 GB write, 21.08 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T06:25:02.987 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T06:25:02.987 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-15T06:25:02.987 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: 2023-11-15T06:25:02.987 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: ** File Read Latency Histogram By Level [default] ** 2023-11-15T06:25:02.988 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: 2023-11-15T06:25:02.988 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.569+0000 7fc758064700 0 starting mon.smithi062 rank 0 at public addrs [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] at bind addrs [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi062 fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 2023-11-15T06:25:02.988 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.570+0000 7fc758064700 1 mon.smithi062@-1(???) e1 preinit fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 2023-11-15T06:25:02.989 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.570+0000 7fc758064700 0 mon.smithi062@-1(???).mds e1 new map 2023-11-15T06:25:02.989 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.570+0000 7fc758064700 0 mon.smithi062@-1(???).mds e1 print_map 2023-11-15T06:25:02.989 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: e1 2023-11-15T06:25:02.989 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: enable_multiple, ever_enabled_multiple: 1,1 2023-11-15T06:25:02.989 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-11-15T06:25:02.990 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: legacy client fscid: -1 2023-11-15T06:25:02.990 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: 2023-11-15T06:25:02.990 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: No filesystems configured 2023-11-15T06:25:02.990 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: 2023-11-15T06:25:02.990 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.570+0000 7fc758064700 0 mon.smithi062@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-11-15T06:25:02.991 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.570+0000 7fc758064700 0 mon.smithi062@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-15T06:25:02.991 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.570+0000 7fc758064700 0 mon.smithi062@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-15T06:25:02.991 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.570+0000 7fc758064700 0 mon.smithi062@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-15T06:25:02.991 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:00.551586+0000 mon.smithi062 (mon.0) 8 : audit [INF] from='client.? 172.21.15.62:0/1884538676' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-11-15T06:25:02.991 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:00.554720+0000 mon.smithi062 (mon.0) 9 : audit [INF] from='client.? 172.21.15.62:0/1884538676' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-11-15T06:25:02.992 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:25:02.570+0000 7fc758064700 1 mon.smithi062@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-11-15T06:25:02.992 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:25:02.587051+0000 mon.smithi062 (mon.0) 1 : cluster [INF] mon.smithi062 is new leader, mons smithi062 in quorum (ranks 0) 2023-11-15T06:25:02.992 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:25:02.587114+0000 mon.smithi062 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi062=[v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0]} 2023-11-15T06:25:02.992 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:25:02.587153+0000 mon.smithi062 (mon.0) 3 : cluster [DBG] fsmap 2023-11-15T06:25:02.992 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:25:02.587178+0000 mon.smithi062 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-11-15T06:25:02.993 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:25:02.587603+0000 mon.smithi062 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2023-11-15T06:25:02.994 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.985+0000 7f26a1811700 1 -- 172.21.15.62:0/913990642 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f269c0fc8a0 con 0x7f269c0fab90 2023-11-15T06:25:02.994 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.988+0000 7f26937fe700 1 -- 172.21.15.62:0/913990642 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f269c0fc8a0 con 0x7f269c0fab90 2023-11-15T06:25:02.994 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.988+0000 7f26a1811700 1 -- 172.21.15.62:0/913990642 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f269c0fab90 msgr2=0x7f269c100e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:02.995 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.988+0000 7f26a1811700 1 --2- 172.21.15.62:0/913990642 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f269c0fab90 0x7f269c100e20 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f268400b1d0 tx=0x7f2684006ee0).stop 2023-11-15T06:25:02.995 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.988+0000 7f26a1811700 1 -- 172.21.15.62:0/913990642 shutdown_connections 2023-11-15T06:25:02.995 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.988+0000 7f26a1811700 1 --2- 172.21.15.62:0/913990642 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f269c0fab90 0x7f269c100e20 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:02.995 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.988+0000 7f26a1811700 1 -- 172.21.15.62:0/913990642 >> 172.21.15.62:0/913990642 conn(0x7f269c0f63b0 msgr2=0x7f269c0700c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:02.995 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.988+0000 7f26a1811700 1 -- 172.21.15.62:0/913990642 shutdown_connections 2023-11-15T06:25:02.996 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:02.989+0000 7f26a1811700 1 -- 172.21.15.62:0/913990642 wait complete. 2023-11-15T06:25:03.345 INFO:teuthology.orchestra.run.smithi062.stderr:Wrote config to /etc/ceph/ceph.conf 2023-11-15T06:25:03.345 INFO:teuthology.orchestra.run.smithi062.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-11-15T06:25:03.345 INFO:teuthology.orchestra.run.smithi062.stderr:Creating mgr... 2023-11-15T06:25:03.346 INFO:teuthology.orchestra.run.smithi062.stderr:Verifying port 9283 ... 2023-11-15T06:25:03.347 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: systemctl daemon-reload 2023-11-15T06:25:03.449 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: systemctl stop ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7@mgr.smithi062.rwurtk 2023-11-15T06:25:03.483 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: systemctl reset-failed ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7@mgr.smithi062.rwurtk 2023-11-15T06:25:03.489 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: Failed to reset failed state of unit ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7@mgr.smithi062.rwurtk.service: Unit ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7@mgr.smithi062.rwurtk.service not loaded. 2023-11-15T06:25:03.489 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: systemctl enable ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7@mgr.smithi062.rwurtk 2023-11-15T06:25:03.495 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: Created symlink /etc/systemd/system/ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7.target.wants/ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7@mgr.smithi062.rwurtk.service → /etc/systemd/system/ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7@.service. 2023-11-15T06:25:03.589 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: systemctl start ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7@mgr.smithi062.rwurtk 2023-11-15T06:25:03.930 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: systemctl is-enabled firewalld.service 2023-11-15T06:25:03.939 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: disabled 2023-11-15T06:25:03.940 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: systemctl is-active firewalld.service 2023-11-15T06:25:03.947 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: inactive 2023-11-15T06:25:03.948 INFO:teuthology.orchestra.run.smithi062.stderr:firewalld.service is not enabled 2023-11-15T06:25:03.949 INFO:teuthology.orchestra.run.smithi062.stderr:Not possible to enable service . firewalld.service is not available 2023-11-15T06:25:03.949 INFO:teuthology.orchestra.run.smithi062.stderr:Not possible to open ports <[]>. firewalld.service is not available 2023-11-15T06:25:03.949 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: systemctl is-enabled firewalld.service 2023-11-15T06:25:03.957 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: disabled 2023-11-15T06:25:03.957 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: systemctl is-active firewalld.service 2023-11-15T06:25:03.965 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: inactive 2023-11-15T06:25:03.966 INFO:teuthology.orchestra.run.smithi062.stderr:firewalld.service is not enabled 2023-11-15T06:25:03.966 INFO:teuthology.orchestra.run.smithi062.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2023-11-15T06:25:03.966 INFO:teuthology.orchestra.run.smithi062.stderr:Waiting for mgr to start... 2023-11-15T06:25:03.966 INFO:teuthology.orchestra.run.smithi062.stderr:Waiting for mgr... 2023-11-15T06:25:03.967 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi062 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptvq9ksio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpf12kfgcg:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-11-15T06:25:04.328 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.326+0000 7f9ce4839700 1 Processor -- start 2023-11-15T06:25:04.329 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.326+0000 7f9ce4839700 1 -- start start 2023-11-15T06:25:04.329 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.326+0000 7f9ce4839700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ce00703b0 0x7f9ce0100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:04.329 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.326+0000 7f9ce4839700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce00fc750 con 0x7f9ce00703b0 2023-11-15T06:25:04.330 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.327+0000 7f9cde59c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ce00703b0 0x7f9ce0100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:04.331 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.327+0000 7f9cde59c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ce00703b0 0x7f9ce0100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38794/0 (socket says 172.21.15.62:38794) 2023-11-15T06:25:04.331 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.327+0000 7f9cde59c700 1 -- 172.21.15.62:0/2995139980 learned_addr learned my addr 172.21.15.62:0/2995139980 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:04.331 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.327+0000 7f9cde59c700 1 -- 172.21.15.62:0/2995139980 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce0101250 con 0x7f9ce00703b0 2023-11-15T06:25:04.331 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.327+0000 7f9cde59c700 1 --2- 172.21.15.62:0/2995139980 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ce00703b0 0x7f9ce0100d60 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f9cd000b5f0 tx=0x7f9cd0006ee0).ready entity=mon.0 client_cookie=bf7f792b15dd1ab2 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:04.331 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.327+0000 7f9cdd59a700 1 -- 172.21.15.62:0/2995139980 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cd000d600 con 0x7f9ce00703b0 2023-11-15T06:25:04.332 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.327+0000 7f9cdd59a700 1 -- 172.21.15.62:0/2995139980 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9cd000dbd0 con 0x7f9ce00703b0 2023-11-15T06:25:04.332 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.327+0000 7f9cdd59a700 1 -- 172.21.15.62:0/2995139980 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cd0011c00 con 0x7f9ce00703b0 2023-11-15T06:25:04.332 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.328+0000 7f9ce4839700 1 -- 172.21.15.62:0/2995139980 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ce00703b0 msgr2=0x7f9ce0100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:04.333 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.328+0000 7f9ce4839700 1 --2- 172.21.15.62:0/2995139980 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ce00703b0 0x7f9ce0100d60 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f9cd000b5f0 tx=0x7f9cd0006ee0).stop 2023-11-15T06:25:04.333 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.328+0000 7f9ce4839700 1 -- 172.21.15.62:0/2995139980 shutdown_connections 2023-11-15T06:25:04.333 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.328+0000 7f9ce4839700 1 --2- 172.21.15.62:0/2995139980 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ce00703b0 0x7f9ce0100d60 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:04.334 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.328+0000 7f9ce4839700 1 -- 172.21.15.62:0/2995139980 >> 172.21.15.62:0/2995139980 conn(0x7f9ce00f63c0 msgr2=0x7f9ce00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:04.334 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.328+0000 7f9ce4839700 1 -- 172.21.15.62:0/2995139980 shutdown_connections 2023-11-15T06:25:04.334 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.328+0000 7f9ce4839700 1 -- 172.21.15.62:0/2995139980 wait complete. 2023-11-15T06:25:04.334 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.329+0000 7f9ce4839700 1 Processor -- start 2023-11-15T06:25:04.334 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.329+0000 7f9ce4839700 1 -- start start 2023-11-15T06:25:04.335 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.329+0000 7f9ce4839700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ce00703b0 0x7f9ce010ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:04.335 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.329+0000 7f9ce4839700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce010d1e0 con 0x7f9ce00703b0 2023-11-15T06:25:04.335 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.329+0000 7f9cde59c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ce00703b0 0x7f9ce010ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:04.335 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.329+0000 7f9cde59c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ce00703b0 0x7f9ce010ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38798/0 (socket says 172.21.15.62:38798) 2023-11-15T06:25:04.335 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.329+0000 7f9cde59c700 1 -- 172.21.15.62:0/1751984188 learned_addr learned my addr 172.21.15.62:0/1751984188 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:04.336 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.329+0000 7f9cde59c700 1 -- 172.21.15.62:0/1751984188 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd000b200 con 0x7f9ce00703b0 2023-11-15T06:25:04.336 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.329+0000 7f9cde59c700 1 --2- 172.21.15.62:0/1751984188 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ce00703b0 0x7f9ce010ccf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f9cd000e040 tx=0x7f9cd0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:04.336 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.329+0000 7f9cceffd700 1 -- 172.21.15.62:0/1751984188 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cd000d9f0 con 0x7f9ce00703b0 2023-11-15T06:25:04.336 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.329+0000 7f9cceffd700 1 -- 172.21.15.62:0/1751984188 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9cd001a9f0 con 0x7f9ce00703b0 2023-11-15T06:25:04.337 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.330+0000 7f9ce4839700 1 -- 172.21.15.62:0/1751984188 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ce010d3e0 con 0x7f9ce00703b0 2023-11-15T06:25:04.337 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.330+0000 7f9cceffd700 1 -- 172.21.15.62:0/1751984188 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cd001b8c0 con 0x7f9ce00703b0 2023-11-15T06:25:04.337 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.330+0000 7f9ce4839700 1 -- 172.21.15.62:0/1751984188 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ce010d7f0 con 0x7f9ce00703b0 2023-11-15T06:25:04.337 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.330+0000 7f9cceffd700 1 -- 172.21.15.62:0/1751984188 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f9cd000f070 con 0x7f9ce00703b0 2023-11-15T06:25:04.337 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.330+0000 7f9cceffd700 1 -- 172.21.15.62:0/1751984188 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f9cd0025ae0 con 0x7f9ce00703b0 2023-11-15T06:25:04.338 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.331+0000 7f9ce4839700 1 -- 172.21.15.62:0/1751984188 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ce0045ad0 con 0x7f9ce00703b0 2023-11-15T06:25:04.338 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.333+0000 7f9cceffd700 1 -- 172.21.15.62:0/1751984188 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f9cd0016070 con 0x7f9ce00703b0 2023-11-15T06:25:04.415 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.403+0000 7f9ce4839700 1 -- 172.21.15.62:0/1751984188 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f9ce010ec40 con 0x7f9ce00703b0 2023-11-15T06:25:04.416 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.403+0000 7f9cceffd700 1 -- 172.21.15.62:0/1751984188 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f9cd001bdc0 con 0x7f9ce00703b0 2023-11-15T06:25:04.416 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.416 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: { 2023-11-15T06:25:04.416 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "fsid": "a3c2da76-837f-11ee-8dbe-212e2dc638e7", 2023-11-15T06:25:04.417 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "health": { 2023-11-15T06:25:04.417 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-15T06:25:04.417 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "checks": {}, 2023-11-15T06:25:04.418 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "mutes": [] 2023-11-15T06:25:04.418 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2023-11-15T06:25:04.418 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-15T06:25:04.418 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "quorum": [ 2023-11-15T06:25:04.419 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 0 2023-11-15T06:25:04.419 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: ], 2023-11-15T06:25:04.419 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-15T06:25:04.420 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "smithi062" 2023-11-15T06:25:04.420 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: ], 2023-11-15T06:25:04.420 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "quorum_age": 1, 2023-11-15T06:25:04.420 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "monmap": { 2023-11-15T06:25:04.421 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T06:25:04.421 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-15T06:25:04.421 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-15T06:25:04.421 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2023-11-15T06:25:04.422 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "osdmap": { 2023-11-15T06:25:04.423 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T06:25:04.423 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-15T06:25:04.423 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-15T06:25:04.423 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-15T06:25:04.424 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-15T06:25:04.424 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-15T06:25:04.424 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-15T06:25:04.425 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2023-11-15T06:25:04.425 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "pgmap": { 2023-11-15T06:25:04.425 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-15T06:25:04.425 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-15T06:25:04.426 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-15T06:25:04.426 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-15T06:25:04.426 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-15T06:25:04.426 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-15T06:25:04.427 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-15T06:25:04.427 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-15T06:25:04.427 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2023-11-15T06:25:04.427 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "fsmap": { 2023-11-15T06:25:04.428 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T06:25:04.428 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-15T06:25:04.428 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-15T06:25:04.429 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2023-11-15T06:25:04.429 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-15T06:25:04.429 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "available": false, 2023-11-15T06:25:04.430 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-15T06:25:04.430 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "modules": [ 2023-11-15T06:25:04.430 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "iostat", 2023-11-15T06:25:04.430 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "restful" 2023-11-15T06:25:04.431 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: ], 2023-11-15T06:25:04.431 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "services": {} 2023-11-15T06:25:04.432 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2023-11-15T06:25:04.432 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "servicemap": { 2023-11-15T06:25:04.432 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T06:25:04.432 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "modified": "2023-11-15T06:24:59.271791+0000", 2023-11-15T06:25:04.433 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "services": {} 2023-11-15T06:25:04.433 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2023-11-15T06:25:04.433 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-15T06:25:04.433 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: } 2023-11-15T06:25:04.434 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.405+0000 7f9ce4839700 1 -- 172.21.15.62:0/1751984188 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ce00703b0 msgr2=0x7f9ce010ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:04.434 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.405+0000 7f9ce4839700 1 --2- 172.21.15.62:0/1751984188 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ce00703b0 0x7f9ce010ccf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f9cd000e040 tx=0x7f9cd0006ee0).stop 2023-11-15T06:25:04.434 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.405+0000 7f9ce4839700 1 -- 172.21.15.62:0/1751984188 shutdown_connections 2023-11-15T06:25:04.434 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.405+0000 7f9ce4839700 1 --2- 172.21.15.62:0/1751984188 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ce00703b0 0x7f9ce010ccf0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:04.434 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.405+0000 7f9ce4839700 1 -- 172.21.15.62:0/1751984188 >> 172.21.15.62:0/1751984188 conn(0x7f9ce00f63c0 msgr2=0x7f9ce00f7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:04.435 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.405+0000 7f9ce4839700 1 -- 172.21.15.62:0/1751984188 shutdown_connections 2023-11-15T06:25:04.435 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:04.405+0000 7f9ce4839700 1 -- 172.21.15.62:0/1751984188 wait complete. 2023-11-15T06:25:04.436 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:03 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:02.988275+0000 mon.smithi062 (mon.0) 6 : audit [INF] from='client.? 172.21.15.62:0/913990642' entity='client.admin' 2023-11-15T06:25:04.770 INFO:teuthology.orchestra.run.smithi062.stderr:mgr not available, waiting (1/15)... 2023-11-15T06:25:05.417 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:04 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:04.403729+0000 mon.smithi062 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.62:0/1751984188' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-15T06:25:06.772 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi062 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptvq9ksio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpf12kfgcg:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-11-15T06:25:07.147 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.145+0000 7fce73be7700 1 Processor -- start 2023-11-15T06:25:07.148 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.145+0000 7fce73be7700 1 -- start start 2023-11-15T06:25:07.148 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.145+0000 7fce73be7700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce6c070440 0x7fce6c100d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:07.148 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.145+0000 7fce73be7700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce6c0fc740 con 0x7fce6c070440 2023-11-15T06:25:07.148 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.146+0000 7fce71983700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce6c070440 0x7fce6c100d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:07.148 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.146+0000 7fce71983700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce6c070440 0x7fce6c100d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38802/0 (socket says 172.21.15.62:38802) 2023-11-15T06:25:07.149 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.146+0000 7fce71983700 1 -- 172.21.15.62:0/165934442 learned_addr learned my addr 172.21.15.62:0/165934442 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:07.149 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.146+0000 7fce71983700 1 -- 172.21.15.62:0/165934442 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce6c101240 con 0x7fce6c070440 2023-11-15T06:25:07.151 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.146+0000 7fce71983700 1 --2- 172.21.15.62:0/165934442 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce6c070440 0x7fce6c100d50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fce6800b5f0 tx=0x7fce68006ee0).ready entity=mon.0 client_cookie=991498ce4a283fc2 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:07.151 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.146+0000 7fce70981700 1 -- 172.21.15.62:0/165934442 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce6800d600 con 0x7fce6c070440 2023-11-15T06:25:07.151 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.146+0000 7fce70981700 1 -- 172.21.15.62:0/165934442 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fce6800dbd0 con 0x7fce6c070440 2023-11-15T06:25:07.151 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.146+0000 7fce70981700 1 -- 172.21.15.62:0/165934442 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce68011c00 con 0x7fce6c070440 2023-11-15T06:25:07.151 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.147+0000 7fce73be7700 1 -- 172.21.15.62:0/165934442 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce6c070440 msgr2=0x7fce6c100d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:07.152 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.147+0000 7fce73be7700 1 --2- 172.21.15.62:0/165934442 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce6c070440 0x7fce6c100d50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fce6800b5f0 tx=0x7fce68006ee0).stop 2023-11-15T06:25:07.152 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.147+0000 7fce73be7700 1 -- 172.21.15.62:0/165934442 shutdown_connections 2023-11-15T06:25:07.152 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.147+0000 7fce73be7700 1 --2- 172.21.15.62:0/165934442 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce6c070440 0x7fce6c100d50 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:07.152 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.147+0000 7fce73be7700 1 -- 172.21.15.62:0/165934442 >> 172.21.15.62:0/165934442 conn(0x7fce6c0f63b0 msgr2=0x7fce6c0f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:07.152 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.147+0000 7fce73be7700 1 -- 172.21.15.62:0/165934442 shutdown_connections 2023-11-15T06:25:07.153 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.147+0000 7fce73be7700 1 -- 172.21.15.62:0/165934442 wait complete. 2023-11-15T06:25:07.153 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.148+0000 7fce73be7700 1 Processor -- start 2023-11-15T06:25:07.153 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.148+0000 7fce73be7700 1 -- start start 2023-11-15T06:25:07.153 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.148+0000 7fce73be7700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce6c070440 0x7fce6c10cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:07.153 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.148+0000 7fce73be7700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce6c10d100 con 0x7fce6c070440 2023-11-15T06:25:07.154 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.148+0000 7fce71983700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce6c070440 0x7fce6c10cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:07.154 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.148+0000 7fce71983700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce6c070440 0x7fce6c10cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38806/0 (socket says 172.21.15.62:38806) 2023-11-15T06:25:07.154 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.148+0000 7fce71983700 1 -- 172.21.15.62:0/4140772507 learned_addr learned my addr 172.21.15.62:0/4140772507 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:07.154 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.149+0000 7fce71983700 1 -- 172.21.15.62:0/4140772507 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce6800b200 con 0x7fce6c070440 2023-11-15T06:25:07.155 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.149+0000 7fce71983700 1 --2- 172.21.15.62:0/4140772507 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce6c070440 0x7fce6c10cc10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fce6800e040 tx=0x7fce68006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:07.155 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.149+0000 7fce5a7fc700 1 -- 172.21.15.62:0/4140772507 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce6800d9f0 con 0x7fce6c070440 2023-11-15T06:25:07.155 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.149+0000 7fce5a7fc700 1 -- 172.21.15.62:0/4140772507 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fce6801a9f0 con 0x7fce6c070440 2023-11-15T06:25:07.155 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.149+0000 7fce5a7fc700 1 -- 172.21.15.62:0/4140772507 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce6801b8c0 con 0x7fce6c070440 2023-11-15T06:25:07.156 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.149+0000 7fce73be7700 1 -- 172.21.15.62:0/4140772507 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce6c10d300 con 0x7fce6c070440 2023-11-15T06:25:07.156 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.149+0000 7fce73be7700 1 -- 172.21.15.62:0/4140772507 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce6c10d710 con 0x7fce6c070440 2023-11-15T06:25:07.156 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.149+0000 7fce5a7fc700 1 -- 172.21.15.62:0/4140772507 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fce6800f070 con 0x7fce6c070440 2023-11-15T06:25:07.156 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.150+0000 7fce5a7fc700 1 -- 172.21.15.62:0/4140772507 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fce6801bd90 con 0x7fce6c070440 2023-11-15T06:25:07.156 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.150+0000 7fce73be7700 1 -- 172.21.15.62:0/4140772507 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce6c045ad0 con 0x7fce6c070440 2023-11-15T06:25:07.157 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.152+0000 7fce5a7fc700 1 -- 172.21.15.62:0/4140772507 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fce68029060 con 0x7fce6c070440 2023-11-15T06:25:07.230 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.223+0000 7fce73be7700 1 -- 172.21.15.62:0/4140772507 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fce6c101da0 con 0x7fce6c070440 2023-11-15T06:25:07.230 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.223+0000 7fce5a7fc700 1 -- 172.21.15.62:0/4140772507 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7fce68032090 con 0x7fce6c070440 2023-11-15T06:25:07.230 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.231 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: { 2023-11-15T06:25:07.231 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "fsid": "a3c2da76-837f-11ee-8dbe-212e2dc638e7", 2023-11-15T06:25:07.231 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "health": { 2023-11-15T06:25:07.231 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-15T06:25:07.231 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "checks": {}, 2023-11-15T06:25:07.232 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "mutes": [] 2023-11-15T06:25:07.232 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2023-11-15T06:25:07.232 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-15T06:25:07.232 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "quorum": [ 2023-11-15T06:25:07.232 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 0 2023-11-15T06:25:07.232 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: ], 2023-11-15T06:25:07.233 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-15T06:25:07.233 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "smithi062" 2023-11-15T06:25:07.233 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: ], 2023-11-15T06:25:07.234 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "quorum_age": 4, 2023-11-15T06:25:07.234 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "monmap": { 2023-11-15T06:25:07.234 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T06:25:07.235 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-15T06:25:07.235 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-15T06:25:07.235 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2023-11-15T06:25:07.235 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "osdmap": { 2023-11-15T06:25:07.236 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T06:25:07.237 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-15T06:25:07.237 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-15T06:25:07.238 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-15T06:25:07.238 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-15T06:25:07.238 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-15T06:25:07.238 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-15T06:25:07.239 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2023-11-15T06:25:07.239 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "pgmap": { 2023-11-15T06:25:07.239 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-15T06:25:07.239 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-15T06:25:07.239 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-15T06:25:07.240 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-15T06:25:07.240 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-15T06:25:07.240 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-15T06:25:07.240 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-15T06:25:07.240 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-15T06:25:07.241 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2023-11-15T06:25:07.241 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "fsmap": { 2023-11-15T06:25:07.241 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T06:25:07.241 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-15T06:25:07.241 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-15T06:25:07.241 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2023-11-15T06:25:07.242 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-15T06:25:07.242 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "available": false, 2023-11-15T06:25:07.242 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-15T06:25:07.242 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "modules": [ 2023-11-15T06:25:07.242 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "iostat", 2023-11-15T06:25:07.243 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "restful" 2023-11-15T06:25:07.243 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: ], 2023-11-15T06:25:07.243 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "services": {} 2023-11-15T06:25:07.243 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2023-11-15T06:25:07.243 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "servicemap": { 2023-11-15T06:25:07.244 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T06:25:07.244 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "modified": "2023-11-15T06:24:59.271791+0000", 2023-11-15T06:25:07.244 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "services": {} 2023-11-15T06:25:07.244 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2023-11-15T06:25:07.244 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-15T06:25:07.245 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: } 2023-11-15T06:25:07.245 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.224+0000 7fce73be7700 1 -- 172.21.15.62:0/4140772507 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce6c070440 msgr2=0x7fce6c10cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:07.245 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.224+0000 7fce73be7700 1 --2- 172.21.15.62:0/4140772507 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce6c070440 0x7fce6c10cc10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fce6800e040 tx=0x7fce68006ee0).stop 2023-11-15T06:25:07.245 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.224+0000 7fce73be7700 1 -- 172.21.15.62:0/4140772507 shutdown_connections 2023-11-15T06:25:07.246 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.224+0000 7fce73be7700 1 --2- 172.21.15.62:0/4140772507 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce6c070440 0x7fce6c10cc10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:07.246 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.224+0000 7fce73be7700 1 -- 172.21.15.62:0/4140772507 >> 172.21.15.62:0/4140772507 conn(0x7fce6c0f63b0 msgr2=0x7fce6c0f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:07.246 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.224+0000 7fce73be7700 1 -- 172.21.15.62:0/4140772507 shutdown_connections 2023-11-15T06:25:07.246 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:07.224+0000 7fce73be7700 1 -- 172.21.15.62:0/4140772507 wait complete. 2023-11-15T06:25:07.513 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:07 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:07.223759+0000 mon.smithi062 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.62:0/4140772507' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-15T06:25:07.586 INFO:teuthology.orchestra.run.smithi062.stderr:mgr not available, waiting (2/15)... 2023-11-15T06:25:09.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:09 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:25:09.093592+0000 mon.smithi062 (mon.0) 9 : cluster [INF] Activating manager daemon smithi062.rwurtk 2023-11-15T06:25:09.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:09 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:25:09.096230+0000 mon.smithi062 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi062.rwurtk(active, starting, since 0.00275752s) 2023-11-15T06:25:09.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:09 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:09.097130+0000 mon.smithi062 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.62:0/281290441' entity='mgr.smithi062.rwurtk' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-15T06:25:09.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:09 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:09.097316+0000 mon.smithi062 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.62:0/281290441' entity='mgr.smithi062.rwurtk' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-15T06:25:09.419 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:09 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:09.097469+0000 mon.smithi062 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.62:0/281290441' entity='mgr.smithi062.rwurtk' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-15T06:25:09.419 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:09 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:09.097608+0000 mon.smithi062 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.62:0/281290441' entity='mgr.smithi062.rwurtk' cmd=[{"prefix": "mon metadata", "id": "smithi062"}]: dispatch 2023-11-15T06:25:09.419 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:09 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:09.097883+0000 mon.smithi062 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.62:0/281290441' entity='mgr.smithi062.rwurtk' cmd=[{"prefix": "mgr metadata", "who": "smithi062.rwurtk", "id": "smithi062.rwurtk"}]: dispatch 2023-11-15T06:25:09.419 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:09 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:25:09.111797+0000 mon.smithi062 (mon.0) 16 : cluster [INF] Manager daemon smithi062.rwurtk is now available 2023-11-15T06:25:09.420 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:09 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:09.119240+0000 mon.smithi062 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.62:0/281290441' entity='mgr.smithi062.rwurtk' 2023-11-15T06:25:09.420 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:09 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:09.124215+0000 mon.smithi062 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.62:0/281290441' entity='mgr.smithi062.rwurtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi062.rwurtk/mirror_snapshot_schedule"}]: dispatch 2023-11-15T06:25:09.420 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:09 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:09.125450+0000 mon.smithi062 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.62:0/281290441' entity='mgr.smithi062.rwurtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi062.rwurtk/trash_purge_schedule"}]: dispatch 2023-11-15T06:25:09.420 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:09 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:09.130240+0000 mon.smithi062 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.62:0/281290441' entity='mgr.smithi062.rwurtk' 2023-11-15T06:25:09.420 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:09 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:09.132259+0000 mon.smithi062 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.62:0/281290441' entity='mgr.smithi062.rwurtk' 2023-11-15T06:25:09.589 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi062 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptvq9ksio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpf12kfgcg:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-11-15T06:25:09.953 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:09.948+0000 7f4d97272700 1 Processor -- start 2023-11-15T06:25:09.953 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:09.948+0000 7f4d97272700 1 -- start start 2023-11-15T06:25:09.953 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:09.948+0000 7f4d97272700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d900703b0 0x7f4d90100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:09.954 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:09.948+0000 7f4d97272700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d900fc750 con 0x7f4d900703b0 2023-11-15T06:25:09.954 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:09.949+0000 7f4d9500e700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d900703b0 0x7f4d90100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:09.954 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:09.949+0000 7f4d9500e700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d900703b0 0x7f4d90100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33830/0 (socket says 172.21.15.62:33830) 2023-11-15T06:25:09.954 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:09.949+0000 7f4d9500e700 1 -- 172.21.15.62:0/3437817506 learned_addr learned my addr 172.21.15.62:0/3437817506 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:09.955 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:09.949+0000 7f4d9500e700 1 -- 172.21.15.62:0/3437817506 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d90101250 con 0x7f4d900703b0 2023-11-15T06:25:09.955 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:09.949+0000 7f4d9500e700 1 --2- 172.21.15.62:0/3437817506 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d900703b0 0x7f4d90100d60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f4d8c00b5f0 tx=0x7f4d8c006ee0).ready entity=mon.0 client_cookie=6d38dc6c2b3cb37f server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:09.955 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:09.949+0000 7f4d7ffff700 1 -- 172.21.15.62:0/3437817506 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d8c00d600 con 0x7f4d900703b0 2023-11-15T06:25:09.956 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:09.949+0000 7f4d7ffff700 1 -- 172.21.15.62:0/3437817506 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4d8c00dbd0 con 0x7f4d900703b0 2023-11-15T06:25:09.956 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:09.949+0000 7f4d7ffff700 1 -- 172.21.15.62:0/3437817506 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d8c011c00 con 0x7f4d900703b0 2023-11-15T06:25:09.956 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:09.950+0000 7f4d97272700 1 -- 172.21.15.62:0/3437817506 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d900703b0 msgr2=0x7f4d90100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:09.956 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:09.950+0000 7f4d97272700 1 --2- 172.21.15.62:0/3437817506 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d900703b0 0x7f4d90100d60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f4d8c00b5f0 tx=0x7f4d8c006ee0).stop 2023-11-15T06:25:09.957 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:09.950+0000 7f4d97272700 1 -- 172.21.15.62:0/3437817506 shutdown_connections 2023-11-15T06:25:09.957 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:09.950+0000 7f4d97272700 1 --2- 172.21.15.62:0/3437817506 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d900703b0 0x7f4d90100d60 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:09.957 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:09.950+0000 7f4d97272700 1 -- 172.21.15.62:0/3437817506 >> 172.21.15.62:0/3437817506 conn(0x7f4d900f63c0 msgr2=0x7f4d900f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:09.958 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:09.950+0000 7f4d97272700 1 -- 172.21.15.62:0/3437817506 shutdown_connections 2023-11-15T06:25:09.958 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:09.950+0000 7f4d97272700 1 -- 172.21.15.62:0/3437817506 wait complete. 2023-11-15T06:25:09.958 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:09.950+0000 7f4d97272700 1 Processor -- start 2023-11-15T06:25:09.959 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:09.950+0000 7f4d97272700 1 -- start start 2023-11-15T06:25:09.959 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:09.951+0000 7f4d97272700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d900703b0 0x7f4d9010ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:09.959 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:09.951+0000 7f4d97272700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9010d1e0 con 0x7f4d900703b0 2023-11-15T06:25:09.960 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:09.951+0000 7f4d9500e700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d900703b0 0x7f4d9010ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:09.960 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:09.951+0000 7f4d9500e700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d900703b0 0x7f4d9010ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33844/0 (socket says 172.21.15.62:33844) 2023-11-15T06:25:09.961 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:09.951+0000 7f4d9500e700 1 -- 172.21.15.62:0/2628547901 learned_addr learned my addr 172.21.15.62:0/2628547901 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:09.961 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:09.951+0000 7f4d9500e700 1 -- 172.21.15.62:0/2628547901 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d8c00b200 con 0x7f4d900703b0 2023-11-15T06:25:09.962 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:09.951+0000 7f4d9500e700 1 --2- 172.21.15.62:0/2628547901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d900703b0 0x7f4d9010ccf0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f4d8c00e040 tx=0x7f4d8c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:09.963 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:09.951+0000 7f4d7dffb700 1 -- 172.21.15.62:0/2628547901 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d8c00d9f0 con 0x7f4d900703b0 2023-11-15T06:25:09.963 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:09.951+0000 7f4d7dffb700 1 -- 172.21.15.62:0/2628547901 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4d8c01a9f0 con 0x7f4d900703b0 2023-11-15T06:25:09.964 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:09.951+0000 7f4d97272700 1 -- 172.21.15.62:0/2628547901 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d9010d3e0 con 0x7f4d900703b0 2023-11-15T06:25:09.964 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:09.951+0000 7f4d7dffb700 1 -- 172.21.15.62:0/2628547901 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d8c01b8c0 con 0x7f4d900703b0 2023-11-15T06:25:09.964 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:09.952+0000 7f4d97272700 1 -- 172.21.15.62:0/2628547901 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d9010d7f0 con 0x7f4d900703b0 2023-11-15T06:25:09.965 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:09.952+0000 7f4d7dffb700 1 -- 172.21.15.62:0/2628547901 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 2) v1 ==== 35959+0+0 (secure 0 0 0) 0x7f4d8c00f070 con 0x7f4d900703b0 2023-11-15T06:25:09.965 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:09.952+0000 7f4d7dffb700 1 -- 172.21.15.62:0/2628547901 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f4d8c041050 con 0x7f4d900703b0 2023-11-15T06:25:09.965 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:09.953+0000 7f4d97272700 1 -- 172.21.15.62:0/2628547901 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d90045ad0 con 0x7f4d900703b0 2023-11-15T06:25:09.966 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:09.955+0000 7f4d7dffb700 1 -- 172.21.15.62:0/2628547901 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f4d8c016070 con 0x7f4d900703b0 2023-11-15T06:25:10.035 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:10.024+0000 7f4d97272700 1 -- 172.21.15.62:0/2628547901 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f4d90101db0 con 0x7f4d900703b0 2023-11-15T06:25:10.035 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:10.024+0000 7f4d7dffb700 1 -- 172.21.15.62:0/2628547901 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f4d90101db0 con 0x7f4d900703b0 2023-11-15T06:25:10.036 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:10.036 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: { 2023-11-15T06:25:10.037 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "fsid": "a3c2da76-837f-11ee-8dbe-212e2dc638e7", 2023-11-15T06:25:10.037 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "health": { 2023-11-15T06:25:10.037 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-15T06:25:10.038 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "checks": {}, 2023-11-15T06:25:10.038 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "mutes": [] 2023-11-15T06:25:10.038 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2023-11-15T06:25:10.039 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-15T06:25:10.039 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "quorum": [ 2023-11-15T06:25:10.040 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 0 2023-11-15T06:25:10.040 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: ], 2023-11-15T06:25:10.040 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-15T06:25:10.041 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "smithi062" 2023-11-15T06:25:10.041 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: ], 2023-11-15T06:25:10.041 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "quorum_age": 7, 2023-11-15T06:25:10.041 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "monmap": { 2023-11-15T06:25:10.041 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T06:25:10.042 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-15T06:25:10.042 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-15T06:25:10.042 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2023-11-15T06:25:10.042 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "osdmap": { 2023-11-15T06:25:10.043 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T06:25:10.043 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-15T06:25:10.043 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-15T06:25:10.044 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-15T06:25:10.044 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-15T06:25:10.044 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-15T06:25:10.045 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-15T06:25:10.045 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2023-11-15T06:25:10.047 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "pgmap": { 2023-11-15T06:25:10.047 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-15T06:25:10.047 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-15T06:25:10.048 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-15T06:25:10.048 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-15T06:25:10.048 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-15T06:25:10.048 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-15T06:25:10.049 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-15T06:25:10.049 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-15T06:25:10.049 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2023-11-15T06:25:10.049 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "fsmap": { 2023-11-15T06:25:10.050 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T06:25:10.050 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-15T06:25:10.050 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-15T06:25:10.050 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2023-11-15T06:25:10.050 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-15T06:25:10.051 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "available": false, 2023-11-15T06:25:10.051 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-15T06:25:10.051 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "modules": [ 2023-11-15T06:25:10.051 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "iostat", 2023-11-15T06:25:10.051 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "restful" 2023-11-15T06:25:10.052 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: ], 2023-11-15T06:25:10.052 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "services": {} 2023-11-15T06:25:10.052 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2023-11-15T06:25:10.052 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "servicemap": { 2023-11-15T06:25:10.052 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T06:25:10.053 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "modified": "2023-11-15T06:24:59.271791+0000", 2023-11-15T06:25:10.053 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "services": {} 2023-11-15T06:25:10.053 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2023-11-15T06:25:10.053 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-15T06:25:10.053 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: } 2023-11-15T06:25:10.054 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:10.025+0000 7f4d97272700 1 -- 172.21.15.62:0/2628547901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d900703b0 msgr2=0x7f4d9010ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:10.054 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:10.025+0000 7f4d97272700 1 --2- 172.21.15.62:0/2628547901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d900703b0 0x7f4d9010ccf0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f4d8c00e040 tx=0x7f4d8c006ee0).stop 2023-11-15T06:25:10.054 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:10.025+0000 7f4d97272700 1 -- 172.21.15.62:0/2628547901 shutdown_connections 2023-11-15T06:25:10.054 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:10.025+0000 7f4d97272700 1 --2- 172.21.15.62:0/2628547901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d900703b0 0x7f4d9010ccf0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:10.054 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:10.025+0000 7f4d97272700 1 -- 172.21.15.62:0/2628547901 >> 172.21.15.62:0/2628547901 conn(0x7f4d900f63c0 msgr2=0x7f4d900f7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:10.054 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:10.025+0000 7f4d97272700 1 -- 172.21.15.62:0/2628547901 shutdown_connections 2023-11-15T06:25:10.055 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:10.025+0000 7f4d97272700 1 -- 172.21.15.62:0/2628547901 wait complete. 2023-11-15T06:25:10.321 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:10 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:10.024611+0000 mon.smithi062 (mon.0) 22 : audit [DBG] from='client.? 172.21.15.62:0/2628547901' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-15T06:25:10.322 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:10 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:25:10.098622+0000 mon.smithi062 (mon.0) 23 : cluster [DBG] mgrmap e3: smithi062.rwurtk(active, since 1.00515s) 2023-11-15T06:25:10.378 INFO:teuthology.orchestra.run.smithi062.stderr:mgr not available, waiting (3/15)... 2023-11-15T06:25:12.380 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi062 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptvq9ksio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpf12kfgcg:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-11-15T06:25:12.417 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:12 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:25:11.151527+0000 mon.smithi062 (mon.0) 24 : cluster [DBG] mgrmap e4: smithi062.rwurtk(active, since 2s) 2023-11-15T06:25:12.776 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:12.773+0000 7f6eb2e3b700 1 Processor -- start 2023-11-15T06:25:12.776 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:12.773+0000 7f6eb2e3b700 1 -- start start 2023-11-15T06:25:12.776 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:12.773+0000 7f6eb2e3b700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6eac0703b0 0x7f6eac100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:12.776 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:12.773+0000 7f6eb2e3b700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eac0fc750 con 0x7f6eac0703b0 2023-11-15T06:25:12.777 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:12.773+0000 7f6eb0bd7700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6eac0703b0 0x7f6eac100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:12.777 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:12.773+0000 7f6eb0bd7700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6eac0703b0 0x7f6eac100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33860/0 (socket says 172.21.15.62:33860) 2023-11-15T06:25:12.777 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:12.773+0000 7f6eb0bd7700 1 -- 172.21.15.62:0/2257170400 learned_addr learned my addr 172.21.15.62:0/2257170400 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:12.777 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:12.774+0000 7f6eb0bd7700 1 -- 172.21.15.62:0/2257170400 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eac101250 con 0x7f6eac0703b0 2023-11-15T06:25:12.777 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:12.774+0000 7f6eb0bd7700 1 --2- 172.21.15.62:0/2257170400 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6eac0703b0 0x7f6eac100d60 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f6ea000b5f0 tx=0x7f6ea0006ee0).ready entity=mon.0 client_cookie=f5fad217f406d163 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:12.778 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:12.774+0000 7f6eab7fe700 1 -- 172.21.15.62:0/2257170400 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ea000d600 con 0x7f6eac0703b0 2023-11-15T06:25:12.778 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:12.774+0000 7f6eab7fe700 1 -- 172.21.15.62:0/2257170400 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f6ea000dbd0 con 0x7f6eac0703b0 2023-11-15T06:25:12.778 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:12.774+0000 7f6eab7fe700 1 -- 172.21.15.62:0/2257170400 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ea0011c00 con 0x7f6eac0703b0 2023-11-15T06:25:12.778 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:12.774+0000 7f6eb2e3b700 1 -- 172.21.15.62:0/2257170400 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6eac0703b0 msgr2=0x7f6eac100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:12.778 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:12.774+0000 7f6eb2e3b700 1 --2- 172.21.15.62:0/2257170400 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6eac0703b0 0x7f6eac100d60 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f6ea000b5f0 tx=0x7f6ea0006ee0).stop 2023-11-15T06:25:12.781 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:12.775+0000 7f6eb2e3b700 1 -- 172.21.15.62:0/2257170400 shutdown_connections 2023-11-15T06:25:12.781 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:12.775+0000 7f6eb2e3b700 1 --2- 172.21.15.62:0/2257170400 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6eac0703b0 0x7f6eac100d60 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:12.781 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:12.775+0000 7f6eb2e3b700 1 -- 172.21.15.62:0/2257170400 >> 172.21.15.62:0/2257170400 conn(0x7f6eac0f63c0 msgr2=0x7f6eac0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:12.781 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:12.775+0000 7f6eb2e3b700 1 -- 172.21.15.62:0/2257170400 shutdown_connections 2023-11-15T06:25:12.782 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:12.775+0000 7f6eb2e3b700 1 -- 172.21.15.62:0/2257170400 wait complete. 2023-11-15T06:25:12.782 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:12.776+0000 7f6eb2e3b700 1 Processor -- start 2023-11-15T06:25:12.782 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:12.776+0000 7f6eb2e3b700 1 -- start start 2023-11-15T06:25:12.782 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:12.776+0000 7f6eb2e3b700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6eac0703b0 0x7f6eac10cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:12.783 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:12.776+0000 7f6eb2e3b700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eac10d100 con 0x7f6eac0703b0 2023-11-15T06:25:12.783 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:12.776+0000 7f6eb0bd7700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6eac0703b0 0x7f6eac10cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:12.783 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:12.776+0000 7f6eb0bd7700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6eac0703b0 0x7f6eac10cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33862/0 (socket says 172.21.15.62:33862) 2023-11-15T06:25:12.783 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:12.776+0000 7f6eb0bd7700 1 -- 172.21.15.62:0/159506628 learned_addr learned my addr 172.21.15.62:0/159506628 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:12.783 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:12.776+0000 7f6eb0bd7700 1 -- 172.21.15.62:0/159506628 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea000b200 con 0x7f6eac0703b0 2023-11-15T06:25:12.784 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:12.777+0000 7f6eb0bd7700 1 --2- 172.21.15.62:0/159506628 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6eac0703b0 0x7f6eac10cc10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f6ea000b190 tx=0x7f6ea0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:12.784 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:12.777+0000 7f6ea97fa700 1 -- 172.21.15.62:0/159506628 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ea001b410 con 0x7f6eac0703b0 2023-11-15T06:25:12.784 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:12.777+0000 7f6ea97fa700 1 -- 172.21.15.62:0/159506628 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f6ea001b9e0 con 0x7f6eac0703b0 2023-11-15T06:25:12.784 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:12.777+0000 7f6eb2e3b700 1 -- 172.21.15.62:0/159506628 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6eac10d300 con 0x7f6eac0703b0 2023-11-15T06:25:12.785 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:12.777+0000 7f6ea97fa700 1 -- 172.21.15.62:0/159506628 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ea001a880 con 0x7f6eac0703b0 2023-11-15T06:25:12.785 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:12.777+0000 7f6eb2e3b700 1 -- 172.21.15.62:0/159506628 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6eac10d710 con 0x7f6eac0703b0 2023-11-15T06:25:12.785 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:12.778+0000 7f6ea97fa700 1 -- 172.21.15.62:0/159506628 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7f6ea000f070 con 0x7f6eac0703b0 2023-11-15T06:25:12.785 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:12.778+0000 7f6ea97fa700 1 --2- 172.21.15.62:0/159506628 >> [v2:172.21.15.62:6800/778351850,v1:172.21.15.62:6801/778351850] conn(0x7f6e9002fd00 0x7f6e900321c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:12.785 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:12.778+0000 7f6ea97fa700 1 -- 172.21.15.62:0/159506628 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f6ea00411b0 con 0x7f6eac0703b0 2023-11-15T06:25:12.786 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:12.778+0000 7f6eabfff700 1 --2- 172.21.15.62:0/159506628 >> [v2:172.21.15.62:6800/778351850,v1:172.21.15.62:6801/778351850] conn(0x7f6e9002fd00 0x7f6e900321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:12.786 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:12.778+0000 7f6eabfff700 1 --2- 172.21.15.62:0/159506628 >> [v2:172.21.15.62:6800/778351850,v1:172.21.15.62:6801/778351850] conn(0x7f6e9002fd00 0x7f6e900321c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f6e980096a0 tx=0x7f6e98006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:12.786 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:12.778+0000 7f6eb2e3b700 1 -- 172.21.15.62:0/159506628 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6eac059100 con 0x7f6eac0703b0 2023-11-15T06:25:12.786 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:12.783+0000 7f6ea97fa700 1 -- 172.21.15.62:0/159506628 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6ea0016070 con 0x7f6eac0703b0 2023-11-15T06:25:13.005 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:12.998+0000 7f6eb2e3b700 1 -- 172.21.15.62:0/159506628 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f6eac045ad0 con 0x7f6eac0703b0 2023-11-15T06:25:13.005 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:12.998+0000 7f6ea97fa700 1 -- 172.21.15.62:0/159506628 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f6ea0025460 con 0x7f6eac0703b0 2023-11-15T06:25:13.005 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.006 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: { 2023-11-15T06:25:13.006 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "fsid": "a3c2da76-837f-11ee-8dbe-212e2dc638e7", 2023-11-15T06:25:13.006 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "health": { 2023-11-15T06:25:13.006 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-15T06:25:13.007 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "checks": {}, 2023-11-15T06:25:13.007 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "mutes": [] 2023-11-15T06:25:13.007 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2023-11-15T06:25:13.007 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-15T06:25:13.007 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "quorum": [ 2023-11-15T06:25:13.007 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 0 2023-11-15T06:25:13.008 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: ], 2023-11-15T06:25:13.008 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-15T06:25:13.008 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "smithi062" 2023-11-15T06:25:13.008 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: ], 2023-11-15T06:25:13.008 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "quorum_age": 10, 2023-11-15T06:25:13.009 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "monmap": { 2023-11-15T06:25:13.009 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T06:25:13.009 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-15T06:25:13.009 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-15T06:25:13.009 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2023-11-15T06:25:13.010 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "osdmap": { 2023-11-15T06:25:13.010 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T06:25:13.010 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-15T06:25:13.010 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-15T06:25:13.010 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-15T06:25:13.011 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-15T06:25:13.011 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-15T06:25:13.011 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-15T06:25:13.011 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2023-11-15T06:25:13.011 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "pgmap": { 2023-11-15T06:25:13.011 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-15T06:25:13.012 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-15T06:25:13.012 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-15T06:25:13.012 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-15T06:25:13.012 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-15T06:25:13.012 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-15T06:25:13.013 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-15T06:25:13.013 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-15T06:25:13.013 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2023-11-15T06:25:13.013 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "fsmap": { 2023-11-15T06:25:13.013 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T06:25:13.014 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-15T06:25:13.014 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-15T06:25:13.014 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2023-11-15T06:25:13.014 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-15T06:25:13.014 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "available": true, 2023-11-15T06:25:13.014 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-15T06:25:13.015 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "modules": [ 2023-11-15T06:25:13.015 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "iostat", 2023-11-15T06:25:13.015 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "restful" 2023-11-15T06:25:13.015 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: ], 2023-11-15T06:25:13.015 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "services": {} 2023-11-15T06:25:13.016 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2023-11-15T06:25:13.016 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "servicemap": { 2023-11-15T06:25:13.016 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T06:25:13.016 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "modified": "2023-11-15T06:24:59.271791+0000", 2023-11-15T06:25:13.016 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "services": {} 2023-11-15T06:25:13.017 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2023-11-15T06:25:13.017 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-15T06:25:13.017 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: } 2023-11-15T06:25:13.017 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.000+0000 7f6eb2e3b700 1 -- 172.21.15.62:0/159506628 >> [v2:172.21.15.62:6800/778351850,v1:172.21.15.62:6801/778351850] conn(0x7f6e9002fd00 msgr2=0x7f6e900321c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:13.017 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.000+0000 7f6eb2e3b700 1 --2- 172.21.15.62:0/159506628 >> [v2:172.21.15.62:6800/778351850,v1:172.21.15.62:6801/778351850] conn(0x7f6e9002fd00 0x7f6e900321c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f6e980096a0 tx=0x7f6e98006b40).stop 2023-11-15T06:25:13.018 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.000+0000 7f6eb2e3b700 1 -- 172.21.15.62:0/159506628 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6eac0703b0 msgr2=0x7f6eac10cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:13.018 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.000+0000 7f6eb2e3b700 1 --2- 172.21.15.62:0/159506628 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6eac0703b0 0x7f6eac10cc10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f6ea000b190 tx=0x7f6ea0006ee0).stop 2023-11-15T06:25:13.018 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.000+0000 7f6eb2e3b700 1 -- 172.21.15.62:0/159506628 shutdown_connections 2023-11-15T06:25:13.018 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.000+0000 7f6eb2e3b700 1 --2- 172.21.15.62:0/159506628 >> [v2:172.21.15.62:6800/778351850,v1:172.21.15.62:6801/778351850] conn(0x7f6e9002fd00 0x7f6e900321c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:13.018 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.000+0000 7f6eb2e3b700 1 --2- 172.21.15.62:0/159506628 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6eac0703b0 0x7f6eac10cc10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:13.019 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.000+0000 7f6eb2e3b700 1 -- 172.21.15.62:0/159506628 >> 172.21.15.62:0/159506628 conn(0x7f6eac0f63c0 msgr2=0x7f6eac0f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:13.019 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.000+0000 7f6eb2e3b700 1 -- 172.21.15.62:0/159506628 shutdown_connections 2023-11-15T06:25:13.019 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.000+0000 7f6eb2e3b700 1 -- 172.21.15.62:0/159506628 wait complete. 2023-11-15T06:25:13.360 INFO:teuthology.orchestra.run.smithi062.stderr:mgr is available 2023-11-15T06:25:13.361 INFO:teuthology.orchestra.run.smithi062.stderr:Enabling cephadm module... 2023-11-15T06:25:13.361 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi062 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptvq9ksio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpf12kfgcg:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2023-11-15T06:25:13.699 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.697+0000 7f630c8af700 1 Processor -- start 2023-11-15T06:25:13.699 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.697+0000 7f630c8af700 1 -- start start 2023-11-15T06:25:13.699 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.697+0000 7f630c8af700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6308070430 0x7f63080fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:13.700 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.697+0000 7f630c8af700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63080fedd0 con 0x7f6308070430 2023-11-15T06:25:13.702 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.698+0000 7f630659c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6308070430 0x7f63080fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:13.702 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.698+0000 7f630659c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6308070430 0x7f63080fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33878/0 (socket says 172.21.15.62:33878) 2023-11-15T06:25:13.702 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.698+0000 7f630659c700 1 -- 172.21.15.62:0/1168431086 learned_addr learned my addr 172.21.15.62:0/1168431086 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:13.703 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.698+0000 7f630659c700 1 -- 172.21.15.62:0/1168431086 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63080fef10 con 0x7f6308070430 2023-11-15T06:25:13.703 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.698+0000 7f630659c700 1 --2- 172.21.15.62:0/1168431086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6308070430 0x7f63080fe8e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f62fc00b5f0 tx=0x7f62fc006ee0).ready entity=mon.0 client_cookie=168d9f72fd884907 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:13.703 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.698+0000 7f630559a700 1 -- 172.21.15.62:0/1168431086 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62fc00d600 con 0x7f6308070430 2023-11-15T06:25:13.703 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.698+0000 7f630559a700 1 -- 172.21.15.62:0/1168431086 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f62fc00dbd0 con 0x7f6308070430 2023-11-15T06:25:13.703 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.698+0000 7f630559a700 1 -- 172.21.15.62:0/1168431086 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62fc011c00 con 0x7f6308070430 2023-11-15T06:25:13.704 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.699+0000 7f630c8af700 1 -- 172.21.15.62:0/1168431086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6308070430 msgr2=0x7f63080fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:13.704 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.699+0000 7f630c8af700 1 --2- 172.21.15.62:0/1168431086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6308070430 0x7f63080fe8e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f62fc00b5f0 tx=0x7f62fc006ee0).stop 2023-11-15T06:25:13.704 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.699+0000 7f630c8af700 1 -- 172.21.15.62:0/1168431086 shutdown_connections 2023-11-15T06:25:13.704 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.699+0000 7f630c8af700 1 --2- 172.21.15.62:0/1168431086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6308070430 0x7f63080fe8e0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:13.704 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.699+0000 7f630c8af700 1 -- 172.21.15.62:0/1168431086 >> 172.21.15.62:0/1168431086 conn(0x7f63080f63c0 msgr2=0x7f63080f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:13.705 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.699+0000 7f630c8af700 1 -- 172.21.15.62:0/1168431086 shutdown_connections 2023-11-15T06:25:13.705 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.699+0000 7f630c8af700 1 -- 172.21.15.62:0/1168431086 wait complete. 2023-11-15T06:25:13.705 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.699+0000 7f630c8af700 1 Processor -- start 2023-11-15T06:25:13.705 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.699+0000 7f630c8af700 1 -- start start 2023-11-15T06:25:13.705 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.700+0000 7f630c8af700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6308070430 0x7f63080fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:13.706 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.700+0000 7f630c8af700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63080fa7e0 con 0x7f6308070430 2023-11-15T06:25:13.706 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.700+0000 7f630659c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6308070430 0x7f63080fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:13.706 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.700+0000 7f630659c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6308070430 0x7f63080fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33884/0 (socket says 172.21.15.62:33884) 2023-11-15T06:25:13.706 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.700+0000 7f630659c700 1 -- 172.21.15.62:0/4013102835 learned_addr learned my addr 172.21.15.62:0/4013102835 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:13.706 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.700+0000 7f630659c700 1 -- 172.21.15.62:0/4013102835 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62fc00b200 con 0x7f6308070430 2023-11-15T06:25:13.707 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.700+0000 7f630659c700 1 --2- 172.21.15.62:0/4013102835 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6308070430 0x7f63080fc360 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f62fc002f40 tx=0x7f62fc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:13.707 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.700+0000 7f62f6ffd700 1 -- 172.21.15.62:0/4013102835 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62fc011690 con 0x7f6308070430 2023-11-15T06:25:13.707 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.700+0000 7f62f6ffd700 1 -- 172.21.15.62:0/4013102835 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f62fc01b880 con 0x7f6308070430 2023-11-15T06:25:13.707 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.700+0000 7f62f6ffd700 1 -- 172.21.15.62:0/4013102835 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62fc01a6b0 con 0x7f6308070430 2023-11-15T06:25:13.707 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.700+0000 7f630c8af700 1 -- 172.21.15.62:0/4013102835 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63080fa980 con 0x7f6308070430 2023-11-15T06:25:13.708 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.701+0000 7f630c8af700 1 -- 172.21.15.62:0/4013102835 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63080fad90 con 0x7f6308070430 2023-11-15T06:25:13.709 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.701+0000 7f630c8af700 1 -- 172.21.15.62:0/4013102835 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6308045ad0 con 0x7f6308070430 2023-11-15T06:25:13.709 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.702+0000 7f62f6ffd700 1 -- 172.21.15.62:0/4013102835 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7f62fc00f070 con 0x7f6308070430 2023-11-15T06:25:13.709 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.702+0000 7f62f6ffd700 1 --2- 172.21.15.62:0/4013102835 >> [v2:172.21.15.62:6800/778351850,v1:172.21.15.62:6801/778351850] conn(0x7f62ec02fd50 0x7f62ec032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:13.709 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.702+0000 7f62f6ffd700 1 -- 172.21.15.62:0/4013102835 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f62fc040fe0 con 0x7f6308070430 2023-11-15T06:25:13.709 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.702+0000 7f6305d9b700 1 --2- 172.21.15.62:0/4013102835 >> [v2:172.21.15.62:6800/778351850,v1:172.21.15.62:6801/778351850] conn(0x7f62ec02fd50 0x7f62ec032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:13.710 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.702+0000 7f6305d9b700 1 --2- 172.21.15.62:0/4013102835 >> [v2:172.21.15.62:6800/778351850,v1:172.21.15.62:6801/778351850] conn(0x7f62ec02fd50 0x7f62ec032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f62f80096a0 tx=0x7f62f8006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:13.710 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.706+0000 7f62f6ffd700 1 -- 172.21.15.62:0/4013102835 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f62fc06d2b0 con 0x7f6308070430 2023-11-15T06:25:13.904 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:13.902+0000 7f630c8af700 1 -- 172.21.15.62:0/4013102835 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f63080ffb90 con 0x7f6308070430 2023-11-15T06:25:13.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:13 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:12.998834+0000 mon.smithi062 (mon.0) 25 : audit [DBG] from='client.? 172.21.15.62:0/159506628' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-15T06:25:14.581 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:14.580+0000 7f62f6ffd700 1 -- 172.21.15.62:0/4013102835 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f62fc019d70 con 0x7f6308070430 2023-11-15T06:25:14.582 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:14.581+0000 7f62f6ffd700 1 -- 172.21.15.62:0/4013102835 <== mon.0 v2:172.21.15.62:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f62fc044850 con 0x7f6308070430 2023-11-15T06:25:14.586 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:14.584+0000 7f630c8af700 1 -- 172.21.15.62:0/4013102835 >> [v2:172.21.15.62:6800/778351850,v1:172.21.15.62:6801/778351850] conn(0x7f62ec02fd50 msgr2=0x7f62ec032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:14.586 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:14.584+0000 7f630c8af700 1 --2- 172.21.15.62:0/4013102835 >> [v2:172.21.15.62:6800/778351850,v1:172.21.15.62:6801/778351850] conn(0x7f62ec02fd50 0x7f62ec032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f62f80096a0 tx=0x7f62f8006b40).stop 2023-11-15T06:25:14.586 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:14.584+0000 7f630c8af700 1 -- 172.21.15.62:0/4013102835 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6308070430 msgr2=0x7f63080fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:14.587 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:14.584+0000 7f630c8af700 1 --2- 172.21.15.62:0/4013102835 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6308070430 0x7f63080fc360 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f62fc002f40 tx=0x7f62fc006ee0).stop 2023-11-15T06:25:14.588 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:14.585+0000 7f630c8af700 1 -- 172.21.15.62:0/4013102835 shutdown_connections 2023-11-15T06:25:14.588 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:14.585+0000 7f630c8af700 1 --2- 172.21.15.62:0/4013102835 >> [v2:172.21.15.62:6800/778351850,v1:172.21.15.62:6801/778351850] conn(0x7f62ec02fd50 0x7f62ec032210 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:14.588 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:14.585+0000 7f630c8af700 1 --2- 172.21.15.62:0/4013102835 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6308070430 0x7f63080fc360 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:14.588 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:14.585+0000 7f630c8af700 1 -- 172.21.15.62:0/4013102835 >> 172.21.15.62:0/4013102835 conn(0x7f63080f63c0 msgr2=0x7f63080f7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:14.588 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:14.586+0000 7f630c8af700 1 -- 172.21.15.62:0/4013102835 shutdown_connections 2023-11-15T06:25:14.589 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:14.586+0000 7f630c8af700 1 -- 172.21.15.62:0/4013102835 wait complete. 2023-11-15T06:25:14.904 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:14 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:13.903222+0000 mon.smithi062 (mon.0) 26 : audit [INF] from='client.? 172.21.15.62:0/4013102835' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-11-15T06:25:14.952 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi062 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptvq9ksio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpf12kfgcg:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2023-11-15T06:25:15.289 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.288+0000 7fd8bad15700 1 Processor -- start 2023-11-15T06:25:15.290 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.288+0000 7fd8bad15700 1 -- start start 2023-11-15T06:25:15.290 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.288+0000 7fd8bad15700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8b40703d0 0x7fd8b40fe790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:15.293 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.288+0000 7fd8bad15700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b40fec80 con 0x7fd8b40703d0 2023-11-15T06:25:15.293 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.288+0000 7fd8b9d13700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8b40703d0 0x7fd8b40fe790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:15.293 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.288+0000 7fd8b9d13700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8b40703d0 0x7fd8b40fe790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33900/0 (socket says 172.21.15.62:33900) 2023-11-15T06:25:15.293 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.288+0000 7fd8b9d13700 1 -- 172.21.15.62:0/2078028452 learned_addr learned my addr 172.21.15.62:0/2078028452 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:15.294 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.288+0000 7fd8b9d13700 1 -- 172.21.15.62:0/2078028452 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b40fedc0 con 0x7fd8b40703d0 2023-11-15T06:25:15.294 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.289+0000 7fd8b9d13700 1 --2- 172.21.15.62:0/2078028452 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8b40703d0 0x7fd8b40fe790 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fd8b000b5f0 tx=0x7fd8b0006ee0).ready entity=mon.0 client_cookie=e219fcc2a7de51c8 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:15.294 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.289+0000 7fd8b8d11700 1 -- 172.21.15.62:0/2078028452 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8b000d600 con 0x7fd8b40703d0 2023-11-15T06:25:15.294 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.289+0000 7fd8b8d11700 1 -- 172.21.15.62:0/2078028452 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd8b000dbd0 con 0x7fd8b40703d0 2023-11-15T06:25:15.295 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.289+0000 7fd8b8d11700 1 -- 172.21.15.62:0/2078028452 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8b0011c00 con 0x7fd8b40703d0 2023-11-15T06:25:15.295 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.290+0000 7fd8bad15700 1 -- 172.21.15.62:0/2078028452 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8b40703d0 msgr2=0x7fd8b40fe790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:15.295 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.290+0000 7fd8bad15700 1 --2- 172.21.15.62:0/2078028452 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8b40703d0 0x7fd8b40fe790 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fd8b000b5f0 tx=0x7fd8b0006ee0).stop 2023-11-15T06:25:15.295 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.290+0000 7fd8bad15700 1 -- 172.21.15.62:0/2078028452 shutdown_connections 2023-11-15T06:25:15.295 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.290+0000 7fd8bad15700 1 --2- 172.21.15.62:0/2078028452 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8b40703d0 0x7fd8b40fe790 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:15.296 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.290+0000 7fd8bad15700 1 -- 172.21.15.62:0/2078028452 >> 172.21.15.62:0/2078028452 conn(0x7fd8b40f6210 msgr2=0x7fd8b40f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:15.296 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.290+0000 7fd8bad15700 1 -- 172.21.15.62:0/2078028452 shutdown_connections 2023-11-15T06:25:15.296 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.290+0000 7fd8bad15700 1 -- 172.21.15.62:0/2078028452 wait complete. 2023-11-15T06:25:15.296 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.291+0000 7fd8bad15700 1 Processor -- start 2023-11-15T06:25:15.296 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.291+0000 7fd8bad15700 1 -- start start 2023-11-15T06:25:15.296 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.291+0000 7fd8bad15700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8b40703d0 0x7fd8b40fc210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:15.297 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.291+0000 7fd8bad15700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b40fa690 con 0x7fd8b40703d0 2023-11-15T06:25:15.297 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.291+0000 7fd8b9d13700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8b40703d0 0x7fd8b40fc210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:15.297 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.291+0000 7fd8b9d13700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8b40703d0 0x7fd8b40fc210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33910/0 (socket says 172.21.15.62:33910) 2023-11-15T06:25:15.297 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.291+0000 7fd8b9d13700 1 -- 172.21.15.62:0/3531360621 learned_addr learned my addr 172.21.15.62:0/3531360621 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:15.299 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.291+0000 7fd8b9d13700 1 -- 172.21.15.62:0/3531360621 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b000b200 con 0x7fd8b40703d0 2023-11-15T06:25:15.299 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.291+0000 7fd8b9d13700 1 --2- 172.21.15.62:0/3531360621 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8b40703d0 0x7fd8b40fc210 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fd8b000b190 tx=0x7fd8b0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:15.299 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.292+0000 7fd8a27fc700 1 -- 172.21.15.62:0/3531360621 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8b001b410 con 0x7fd8b40703d0 2023-11-15T06:25:15.299 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.292+0000 7fd8a27fc700 1 -- 172.21.15.62:0/3531360621 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd8b001b9e0 con 0x7fd8b40703d0 2023-11-15T06:25:15.300 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.292+0000 7fd8bad15700 1 -- 172.21.15.62:0/3531360621 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8b40fa830 con 0x7fd8b40703d0 2023-11-15T06:25:15.300 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.292+0000 7fd8a27fc700 1 -- 172.21.15.62:0/3531360621 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8b001a880 con 0x7fd8b40703d0 2023-11-15T06:25:15.300 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.292+0000 7fd8bad15700 1 -- 172.21.15.62:0/3531360621 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8b40fac40 con 0x7fd8b40703d0 2023-11-15T06:25:15.300 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.293+0000 7fd8bad15700 1 -- 172.21.15.62:0/3531360621 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8b4045ad0 con 0x7fd8b40703d0 2023-11-15T06:25:15.300 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.293+0000 7fd8a27fc700 1 -- 172.21.15.62:0/3531360621 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fd8b000f070 con 0x7fd8b40703d0 2023-11-15T06:25:15.301 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.293+0000 7fd8a27fc700 1 --2- 172.21.15.62:0/3531360621 >> [v2:172.21.15.62:6800/778351850,v1:172.21.15.62:6801/778351850] conn(0x7fd89c02fd50 0x7fd89c032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:15.301 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.293+0000 7fd8a27fc700 1 -- 172.21.15.62:0/3531360621 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fd8b0041200 con 0x7fd8b40703d0 2023-11-15T06:25:15.301 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.293+0000 7fd8b9512700 1 -- 172.21.15.62:0/3531360621 >> [v2:172.21.15.62:6800/778351850,v1:172.21.15.62:6801/778351850] conn(0x7fd89c02fd50 msgr2=0x7fd89c032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/778351850 2023-11-15T06:25:15.301 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.293+0000 7fd8b9512700 1 --2- 172.21.15.62:0/3531360621 >> [v2:172.21.15.62:6800/778351850,v1:172.21.15.62:6801/778351850] conn(0x7fd89c02fd50 0x7fd89c032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-15T06:25:15.301 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.298+0000 7fd8a27fc700 1 -- 172.21.15.62:0/3531360621 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd8b00224f0 con 0x7fd8b40703d0 2023-11-15T06:25:15.495 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.494+0000 7fd8b9512700 1 -- 172.21.15.62:0/3531360621 >> [v2:172.21.15.62:6800/778351850,v1:172.21.15.62:6801/778351850] conn(0x7fd89c02fd50 msgr2=0x7fd89c032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/778351850 2023-11-15T06:25:15.496 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.494+0000 7fd8b9512700 1 --2- 172.21.15.62:0/3531360621 >> [v2:172.21.15.62:6800/778351850,v1:172.21.15.62:6801/778351850] conn(0x7fd89c02fd50 0x7fd89c032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-15T06:25:15.499 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.498+0000 7fd8bad15700 1 -- 172.21.15.62:0/3531360621 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fd8b4110850 con 0x7fd8b40703d0 2023-11-15T06:25:15.500 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.499+0000 7fd8a27fc700 1 -- 172.21.15.62:0/3531360621 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7fd8b0016070 con 0x7fd8b40703d0 2023-11-15T06:25:15.501 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: { 2023-11-15T06:25:15.501 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "epoch": 5, 2023-11-15T06:25:15.501 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "available": true, 2023-11-15T06:25:15.501 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "active_name": "smithi062.rwurtk", 2023-11-15T06:25:15.502 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_standby": 0 2023-11-15T06:25:15.502 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: } 2023-11-15T06:25:15.503 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.500+0000 7fd8bad15700 1 -- 172.21.15.62:0/3531360621 >> [v2:172.21.15.62:6800/778351850,v1:172.21.15.62:6801/778351850] conn(0x7fd89c02fd50 msgr2=0x7fd89c032210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-15T06:25:15.503 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.500+0000 7fd8bad15700 1 --2- 172.21.15.62:0/3531360621 >> [v2:172.21.15.62:6800/778351850,v1:172.21.15.62:6801/778351850] conn(0x7fd89c02fd50 0x7fd89c032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T06:25:15.504 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.500+0000 7fd8bad15700 1 -- 172.21.15.62:0/3531360621 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8b40703d0 msgr2=0x7fd8b40fc210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:15.504 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.500+0000 7fd8bad15700 1 --2- 172.21.15.62:0/3531360621 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8b40703d0 0x7fd8b40fc210 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fd8b000b190 tx=0x7fd8b0006ee0).stop 2023-11-15T06:25:15.504 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.501+0000 7fd8bad15700 1 -- 172.21.15.62:0/3531360621 shutdown_connections 2023-11-15T06:25:15.504 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.501+0000 7fd8bad15700 1 --2- 172.21.15.62:0/3531360621 >> [v2:172.21.15.62:6800/778351850,v1:172.21.15.62:6801/778351850] conn(0x7fd89c02fd50 0x7fd89c032210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T06:25:15.505 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.501+0000 7fd8bad15700 1 --2- 172.21.15.62:0/3531360621 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8b40703d0 0x7fd8b40fc210 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:15.505 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.501+0000 7fd8bad15700 1 -- 172.21.15.62:0/3531360621 >> 172.21.15.62:0/3531360621 conn(0x7fd8b40f6210 msgr2=0x7fd8b40f7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:15.505 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.501+0000 7fd8bad15700 1 -- 172.21.15.62:0/3531360621 shutdown_connections 2023-11-15T06:25:15.505 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:15.501+0000 7fd8bad15700 1 -- 172.21.15.62:0/3531360621 wait complete. 2023-11-15T06:25:15.917 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:15 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:14.582075+0000 mon.smithi062 (mon.0) 27 : audit [INF] from='client.? 172.21.15.62:0/4013102835' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-11-15T06:25:15.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:15 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:25:14.582145+0000 mon.smithi062 (mon.0) 28 : cluster [DBG] mgrmap e5: smithi062.rwurtk(active, since 5s) 2023-11-15T06:25:15.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:15 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:15.499545+0000 mon.smithi062 (mon.0) 29 : audit [DBG] from='client.? 172.21.15.62:0/3531360621' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-11-15T06:25:16.991 INFO:teuthology.orchestra.run.smithi062.stderr:Waiting for the mgr to restart... 2023-11-15T06:25:16.991 INFO:teuthology.orchestra.run.smithi062.stderr:Waiting for mgr epoch 5... 2023-11-15T06:25:16.991 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi062 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptvq9ksio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpf12kfgcg:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2023-11-15T06:25:17.951 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:17.948+0000 7ffa81da1700 1 Processor -- start 2023-11-15T06:25:17.952 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:17.949+0000 7ffa81da1700 1 -- start start 2023-11-15T06:25:17.952 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:17.949+0000 7ffa81da1700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa7c0703b0 0x7ffa7c100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:17.955 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:17.949+0000 7ffa81da1700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7c0fc5a0 con 0x7ffa7c0703b0 2023-11-15T06:25:17.955 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:17.949+0000 7ffa80d9f700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa7c0703b0 0x7ffa7c100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:17.955 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:17.949+0000 7ffa80d9f700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa7c0703b0 0x7ffa7c100bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33922/0 (socket says 172.21.15.62:33922) 2023-11-15T06:25:17.956 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:17.949+0000 7ffa80d9f700 1 -- 172.21.15.62:0/1623046812 learned_addr learned my addr 172.21.15.62:0/1623046812 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:17.956 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:17.949+0000 7ffa80d9f700 1 -- 172.21.15.62:0/1623046812 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa7c1010a0 con 0x7ffa7c0703b0 2023-11-15T06:25:17.956 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:17.949+0000 7ffa80d9f700 1 --2- 172.21.15.62:0/1623046812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa7c0703b0 0x7ffa7c100bb0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7ffa7000b5f0 tx=0x7ffa70006ee0).ready entity=mon.0 client_cookie=3c349d0cd393cdd0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:17.956 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:17.950+0000 7ffa7b7fe700 1 -- 172.21.15.62:0/1623046812 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa7000d600 con 0x7ffa7c0703b0 2023-11-15T06:25:17.956 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:17.950+0000 7ffa7b7fe700 1 -- 172.21.15.62:0/1623046812 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ffa7000dbd0 con 0x7ffa7c0703b0 2023-11-15T06:25:17.957 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:17.950+0000 7ffa7b7fe700 1 -- 172.21.15.62:0/1623046812 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa70011c00 con 0x7ffa7c0703b0 2023-11-15T06:25:17.957 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:17.950+0000 7ffa81da1700 1 -- 172.21.15.62:0/1623046812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa7c0703b0 msgr2=0x7ffa7c100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:17.957 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:17.950+0000 7ffa81da1700 1 --2- 172.21.15.62:0/1623046812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa7c0703b0 0x7ffa7c100bb0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7ffa7000b5f0 tx=0x7ffa70006ee0).stop 2023-11-15T06:25:17.957 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:17.950+0000 7ffa81da1700 1 -- 172.21.15.62:0/1623046812 shutdown_connections 2023-11-15T06:25:17.957 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:17.950+0000 7ffa81da1700 1 --2- 172.21.15.62:0/1623046812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa7c0703b0 0x7ffa7c100bb0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:17.958 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:17.950+0000 7ffa81da1700 1 -- 172.21.15.62:0/1623046812 >> 172.21.15.62:0/1623046812 conn(0x7ffa7c0f6210 msgr2=0x7ffa7c0f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:17.958 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:17.950+0000 7ffa81da1700 1 -- 172.21.15.62:0/1623046812 shutdown_connections 2023-11-15T06:25:17.958 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:17.950+0000 7ffa81da1700 1 -- 172.21.15.62:0/1623046812 wait complete. 2023-11-15T06:25:17.958 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:17.951+0000 7ffa81da1700 1 Processor -- start 2023-11-15T06:25:17.958 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:17.951+0000 7ffa81da1700 1 -- start start 2023-11-15T06:25:17.959 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:17.951+0000 7ffa81da1700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa7c0703b0 0x7ffa7c10ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:17.959 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:17.951+0000 7ffa81da1700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7c10cf20 con 0x7ffa7c0703b0 2023-11-15T06:25:17.959 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:17.951+0000 7ffa80d9f700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa7c0703b0 0x7ffa7c10ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:17.959 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:17.951+0000 7ffa80d9f700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa7c0703b0 0x7ffa7c10ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33930/0 (socket says 172.21.15.62:33930) 2023-11-15T06:25:17.959 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:17.951+0000 7ffa80d9f700 1 -- 172.21.15.62:0/3785644954 learned_addr learned my addr 172.21.15.62:0/3785644954 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:17.959 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:17.952+0000 7ffa80d9f700 1 -- 172.21.15.62:0/3785644954 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa7000b200 con 0x7ffa7c0703b0 2023-11-15T06:25:17.960 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:17.952+0000 7ffa80d9f700 1 --2- 172.21.15.62:0/3785644954 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa7c0703b0 0x7ffa7c10ca30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7ffa7000b190 tx=0x7ffa70006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:17.960 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:17.952+0000 7ffa797fa700 1 -- 172.21.15.62:0/3785644954 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa7001b410 con 0x7ffa7c0703b0 2023-11-15T06:25:17.960 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:17.952+0000 7ffa797fa700 1 -- 172.21.15.62:0/3785644954 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ffa7001b9e0 con 0x7ffa7c0703b0 2023-11-15T06:25:17.960 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:17.952+0000 7ffa81da1700 1 -- 172.21.15.62:0/3785644954 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa7c10d120 con 0x7ffa7c0703b0 2023-11-15T06:25:17.960 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:17.952+0000 7ffa81da1700 1 -- 172.21.15.62:0/3785644954 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa7c10d530 con 0x7ffa7c0703b0 2023-11-15T06:25:17.961 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:17.953+0000 7ffa797fa700 1 -- 172.21.15.62:0/3785644954 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa7001a880 con 0x7ffa7c0703b0 2023-11-15T06:25:17.961 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:17.953+0000 7ffa797fa700 1 -- 172.21.15.62:0/3785644954 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7ffa7000f070 con 0x7ffa7c0703b0 2023-11-15T06:25:17.962 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:17.953+0000 7ffa797fa700 1 --2- 172.21.15.62:0/3785644954 >> [v2:172.21.15.62:6800/778351850,v1:172.21.15.62:6801/778351850] conn(0x7ffa6002fdb0 0x7ffa60032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:17.962 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:17.953+0000 7ffa797fa700 1 -- 172.21.15.62:0/3785644954 --> [v2:172.21.15.62:6800/778351850,v1:172.21.15.62:6801/778351850] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa60032930 con 0x7ffa6002fdb0 2023-11-15T06:25:17.962 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:17.953+0000 7ffa797fa700 1 -- 172.21.15.62:0/3785644954 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7ffa70042020 con 0x7ffa7c0703b0 2023-11-15T06:25:17.962 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:17.953+0000 7ffa7bfff700 1 -- 172.21.15.62:0/3785644954 >> [v2:172.21.15.62:6800/778351850,v1:172.21.15.62:6801/778351850] conn(0x7ffa6002fdb0 msgr2=0x7ffa60032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/778351850 2023-11-15T06:25:17.963 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:17.953+0000 7ffa7bfff700 1 --2- 172.21.15.62:0/3785644954 >> [v2:172.21.15.62:6800/778351850,v1:172.21.15.62:6801/778351850] conn(0x7ffa6002fdb0 0x7ffa60032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-15T06:25:18.156 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:18.154+0000 7ffa7bfff700 1 -- 172.21.15.62:0/3785644954 >> [v2:172.21.15.62:6800/778351850,v1:172.21.15.62:6801/778351850] conn(0x7ffa6002fdb0 msgr2=0x7ffa60032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/778351850 2023-11-15T06:25:18.156 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:18.154+0000 7ffa7bfff700 1 --2- 172.21.15.62:0/3785644954 >> [v2:172.21.15.62:6800/778351850,v1:172.21.15.62:6801/778351850] conn(0x7ffa6002fdb0 0x7ffa60032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-15T06:25:18.558 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:18.555+0000 7ffa7bfff700 1 -- 172.21.15.62:0/3785644954 >> [v2:172.21.15.62:6800/778351850,v1:172.21.15.62:6801/778351850] conn(0x7ffa6002fdb0 msgr2=0x7ffa60032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/778351850 2023-11-15T06:25:18.558 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:18.555+0000 7ffa7bfff700 1 --2- 172.21.15.62:0/3785644954 >> [v2:172.21.15.62:6800/778351850,v1:172.21.15.62:6801/778351850] conn(0x7ffa6002fdb0 0x7ffa60032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-11-15T06:25:19.358 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:19.356+0000 7ffa7bfff700 1 -- 172.21.15.62:0/3785644954 >> [v2:172.21.15.62:6800/778351850,v1:172.21.15.62:6801/778351850] conn(0x7ffa6002fdb0 msgr2=0x7ffa60032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/778351850 2023-11-15T06:25:19.359 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:19.356+0000 7ffa7bfff700 1 --2- 172.21.15.62:0/3785644954 >> [v2:172.21.15.62:6800/778351850,v1:172.21.15.62:6801/778351850] conn(0x7ffa6002fdb0 0x7ffa60032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-11-15T06:25:19.852 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:19.849+0000 7ffa797fa700 1 -- 172.21.15.62:0/3785644954 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mgrmap(e 6) v1 ==== 35970+0+0 (secure 0 0 0) 0x7ffa700257e0 con 0x7ffa7c0703b0 2023-11-15T06:25:19.853 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:19.849+0000 7ffa797fa700 1 -- 172.21.15.62:0/3785644954 >> [v2:172.21.15.62:6800/778351850,v1:172.21.15.62:6801/778351850] conn(0x7ffa6002fdb0 msgr2=0x7ffa60032270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-15T06:25:19.853 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:19.849+0000 7ffa797fa700 1 --2- 172.21.15.62:0/3785644954 >> [v2:172.21.15.62:6800/778351850,v1:172.21.15.62:6801/778351850] conn(0x7ffa6002fdb0 0x7ffa60032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T06:25:20.853 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:20.851+0000 7ffa797fa700 1 -- 172.21.15.62:0/3785644954 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7ffa70019d70 con 0x7ffa7c0703b0 2023-11-15T06:25:20.854 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:20.851+0000 7ffa797fa700 1 --2- 172.21.15.62:0/3785644954 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7ffa60033880 0x7ffa60035c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:20.854 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:20.851+0000 7ffa797fa700 1 -- 172.21.15.62:0/3785644954 --> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa60032930 con 0x7ffa60033880 2023-11-15T06:25:20.854 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:20.851+0000 7ffa7bfff700 1 --2- 172.21.15.62:0/3785644954 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7ffa60033880 0x7ffa60035c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:20.855 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:20.852+0000 7ffa7bfff700 1 --2- 172.21.15.62:0/3785644954 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7ffa60033880 0x7ffa60035c70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7ffa680036d0 tx=0x7ffa68006d20).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:20.855 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:20.853+0000 7ffa797fa700 1 -- 172.21.15.62:0/3785644954 <== mgr.14116 v2:172.21.15.62:6800/3280254567 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7ffa60032930 con 0x7ffa60033880 2023-11-15T06:25:20.861 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:20.859+0000 7ffa81da1700 1 -- 172.21.15.62:0/3785644954 --> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7ffa7c059100 con 0x7ffa60033880 2023-11-15T06:25:20.863 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:20.859+0000 7ffa797fa700 1 -- 172.21.15.62:0/3785644954 <== mgr.14116 v2:172.21.15.62:6800/3280254567 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7ffa7c059100 con 0x7ffa60033880 2023-11-15T06:25:20.863 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: { 2023-11-15T06:25:20.864 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2023-11-15T06:25:20.864 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "initialized": true 2023-11-15T06:25:20.864 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: } 2023-11-15T06:25:20.864 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:20.860+0000 7ffa81da1700 1 -- 172.21.15.62:0/3785644954 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7ffa60033880 msgr2=0x7ffa60035c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:20.864 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:20.860+0000 7ffa81da1700 1 --2- 172.21.15.62:0/3785644954 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7ffa60033880 0x7ffa60035c70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7ffa680036d0 tx=0x7ffa68006d20).stop 2023-11-15T06:25:20.865 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:20.860+0000 7ffa81da1700 1 -- 172.21.15.62:0/3785644954 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa7c0703b0 msgr2=0x7ffa7c10ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:20.865 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:20.860+0000 7ffa81da1700 1 --2- 172.21.15.62:0/3785644954 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa7c0703b0 0x7ffa7c10ca30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7ffa7000b190 tx=0x7ffa70006ee0).stop 2023-11-15T06:25:20.865 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:20.860+0000 7ffa81da1700 1 -- 172.21.15.62:0/3785644954 shutdown_connections 2023-11-15T06:25:20.865 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:20.860+0000 7ffa81da1700 1 --2- 172.21.15.62:0/3785644954 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7ffa60033880 0x7ffa60035c70 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:20.865 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:20.860+0000 7ffa81da1700 1 --2- 172.21.15.62:0/3785644954 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa7c0703b0 0x7ffa7c10ca30 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:20.866 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:20.860+0000 7ffa81da1700 1 --2- 172.21.15.62:0/3785644954 >> [v2:172.21.15.62:6800/778351850,v1:172.21.15.62:6801/778351850] conn(0x7ffa6002fdb0 0x7ffa60032270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T06:25:20.866 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:20.860+0000 7ffa81da1700 1 -- 172.21.15.62:0/3785644954 >> 172.21.15.62:0/3785644954 conn(0x7ffa7c0f6210 msgr2=0x7ffa7c0f7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:20.866 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:20.860+0000 7ffa81da1700 1 -- 172.21.15.62:0/3785644954 shutdown_connections 2023-11-15T06:25:20.866 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:20.860+0000 7ffa81da1700 1 -- 172.21.15.62:0/3785644954 wait complete. 2023-11-15T06:25:21.145 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:20 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:25:19.798133+0000 mon.smithi062 (mon.0) 30 : cluster [INF] Active manager daemon smithi062.rwurtk restarted 2023-11-15T06:25:21.146 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:20 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:25:19.798643+0000 mon.smithi062 (mon.0) 31 : cluster [INF] Activating manager daemon smithi062.rwurtk 2023-11-15T06:25:21.146 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:20 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:25:19.800241+0000 mon.smithi062 (mon.0) 32 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2023-11-15T06:25:21.146 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:20 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:25:19.851696+0000 mon.smithi062 (mon.0) 33 : cluster [DBG] mgrmap e6: smithi062.rwurtk(active, starting, since 0.0531595s) 2023-11-15T06:25:21.146 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:20 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:19.853008+0000 mon.smithi062 (mon.0) 34 : audit [DBG] from='mgr.14116 172.21.15.62:0/1148526200' entity='mgr.smithi062.rwurtk' cmd=[{"prefix": "mon metadata", "id": "smithi062"}]: dispatch 2023-11-15T06:25:21.147 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:20 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:19.853610+0000 mon.smithi062 (mon.0) 35 : audit [DBG] from='mgr.14116 172.21.15.62:0/1148526200' entity='mgr.smithi062.rwurtk' cmd=[{"prefix": "mgr metadata", "who": "smithi062.rwurtk", "id": "smithi062.rwurtk"}]: dispatch 2023-11-15T06:25:21.147 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:20 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:19.854286+0000 mon.smithi062 (mon.0) 36 : audit [DBG] from='mgr.14116 172.21.15.62:0/1148526200' entity='mgr.smithi062.rwurtk' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-15T06:25:21.147 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:20 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:19.854464+0000 mon.smithi062 (mon.0) 37 : audit [DBG] from='mgr.14116 172.21.15.62:0/1148526200' entity='mgr.smithi062.rwurtk' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-15T06:25:21.147 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:20 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:19.854598+0000 mon.smithi062 (mon.0) 38 : audit [DBG] from='mgr.14116 172.21.15.62:0/1148526200' entity='mgr.smithi062.rwurtk' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-15T06:25:21.147 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:20 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:25:19.867701+0000 mon.smithi062 (mon.0) 39 : cluster [INF] Manager daemon smithi062.rwurtk is now available 2023-11-15T06:25:21.148 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:20 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:19.881174+0000 mon.smithi062 (mon.0) 40 : audit [INF] from='mgr.14116 172.21.15.62:0/1148526200' entity='mgr.smithi062.rwurtk' 2023-11-15T06:25:21.148 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:20 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:19.884025+0000 mon.smithi062 (mon.0) 41 : audit [INF] from='mgr.14116 172.21.15.62:0/1148526200' entity='mgr.smithi062.rwurtk' 2023-11-15T06:25:21.148 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:20 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:19.886478+0000 mon.smithi062 (mon.0) 42 : audit [INF] from='mgr.14116 172.21.15.62:0/1148526200' entity='mgr.smithi062.rwurtk' 2023-11-15T06:25:21.148 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:20 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:19.980915+0000 mon.smithi062 (mon.0) 43 : audit [INF] from='mgr.14116 172.21.15.62:0/1148526200' entity='mgr.smithi062.rwurtk' 2023-11-15T06:25:21.148 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:20 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:19.981607+0000 mon.smithi062 (mon.0) 44 : audit [DBG] from='mgr.14116 172.21.15.62:0/1148526200' entity='mgr.smithi062.rwurtk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T06:25:21.149 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:20 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:19.982797+0000 mon.smithi062 (mon.0) 45 : audit [DBG] from='mgr.14116 172.21.15.62:0/1148526200' entity='mgr.smithi062.rwurtk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T06:25:21.149 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:20 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:19.984211+0000 mon.smithi062 (mon.0) 46 : audit [DBG] from='mgr.14116 172.21.15.62:0/1148526200' entity='mgr.smithi062.rwurtk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T06:25:21.149 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:20 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:19.995011+0000 mon.smithi062 (mon.0) 47 : audit [INF] from='mgr.14116 172.21.15.62:0/1148526200' entity='mgr.smithi062.rwurtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi062.rwurtk/mirror_snapshot_schedule"}]: dispatch 2023-11-15T06:25:21.149 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:20 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:19.996264+0000 mon.smithi062 (mon.0) 48 : audit [INF] from='mgr.14116 172.21.15.62:0/1148526200' entity='mgr.smithi062.rwurtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi062.rwurtk/trash_purge_schedule"}]: dispatch 2023-11-15T06:25:21.201 INFO:teuthology.orchestra.run.smithi062.stderr:mgr epoch 5 is available 2023-11-15T06:25:21.202 INFO:teuthology.orchestra.run.smithi062.stderr:Setting orchestrator backend to cephadm... 2023-11-15T06:25:21.202 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi062 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptvq9ksio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpf12kfgcg:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2023-11-15T06:25:21.563 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.559+0000 7f1aa36a2700 1 Processor -- start 2023-11-15T06:25:21.564 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.559+0000 7f1aa36a2700 1 -- start start 2023-11-15T06:25:21.564 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.560+0000 7f1aa36a2700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a9c0fab80 0x7f1a9c0fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:21.564 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.560+0000 7f1aa36a2700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a9c0fb490 con 0x7f1a9c0fab80 2023-11-15T06:25:21.564 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.560+0000 7f1aa143e700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a9c0fab80 0x7f1a9c0fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:21.565 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.560+0000 7f1aa143e700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a9c0fab80 0x7f1a9c0fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56468/0 (socket says 172.21.15.62:56468) 2023-11-15T06:25:21.565 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.560+0000 7f1aa143e700 1 -- 172.21.15.62:0/366823923 learned_addr learned my addr 172.21.15.62:0/366823923 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:21.565 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.560+0000 7f1aa143e700 1 -- 172.21.15.62:0/366823923 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a9c0fb5d0 con 0x7f1a9c0fab80 2023-11-15T06:25:21.565 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.560+0000 7f1aa143e700 1 --2- 172.21.15.62:0/366823923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a9c0fab80 0x7f1a9c0fafa0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f1a8c00b5f0 tx=0x7f1a8c006ee0).ready entity=mon.0 client_cookie=89773265bb40da63 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:21.565 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.560+0000 7f1a8bfff700 1 -- 172.21.15.62:0/366823923 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a8c00d600 con 0x7f1a9c0fab80 2023-11-15T06:25:21.566 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.560+0000 7f1a8bfff700 1 -- 172.21.15.62:0/366823923 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1a8c00dbd0 con 0x7f1a9c0fab80 2023-11-15T06:25:21.566 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.560+0000 7f1a8bfff700 1 -- 172.21.15.62:0/366823923 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a8c011c00 con 0x7f1a9c0fab80 2023-11-15T06:25:21.566 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.561+0000 7f1aa36a2700 1 -- 172.21.15.62:0/366823923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a9c0fab80 msgr2=0x7f1a9c0fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:21.566 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.561+0000 7f1aa36a2700 1 --2- 172.21.15.62:0/366823923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a9c0fab80 0x7f1a9c0fafa0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f1a8c00b5f0 tx=0x7f1a8c006ee0).stop 2023-11-15T06:25:21.566 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.561+0000 7f1aa36a2700 1 -- 172.21.15.62:0/366823923 shutdown_connections 2023-11-15T06:25:21.567 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.561+0000 7f1aa36a2700 1 --2- 172.21.15.62:0/366823923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a9c0fab80 0x7f1a9c0fafa0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:21.567 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.561+0000 7f1aa36a2700 1 -- 172.21.15.62:0/366823923 >> 172.21.15.62:0/366823923 conn(0x7f1a9c0f63c0 msgr2=0x7f1a9c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:21.567 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.561+0000 7f1aa36a2700 1 -- 172.21.15.62:0/366823923 shutdown_connections 2023-11-15T06:25:21.568 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.561+0000 7f1aa36a2700 1 -- 172.21.15.62:0/366823923 wait complete. 2023-11-15T06:25:21.568 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.561+0000 7f1aa36a2700 1 Processor -- start 2023-11-15T06:25:21.570 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.562+0000 7f1aa36a2700 1 -- start start 2023-11-15T06:25:21.570 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.562+0000 7f1aa36a2700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a9c0fab80 0x7f1a9c100e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:21.570 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.562+0000 7f1aa36a2700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a9c0ff290 con 0x7f1a9c0fab80 2023-11-15T06:25:21.571 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.562+0000 7f1aa143e700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a9c0fab80 0x7f1a9c100e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:21.571 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.562+0000 7f1aa143e700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a9c0fab80 0x7f1a9c100e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56474/0 (socket says 172.21.15.62:56474) 2023-11-15T06:25:21.571 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.562+0000 7f1aa143e700 1 -- 172.21.15.62:0/2037503613 learned_addr learned my addr 172.21.15.62:0/2037503613 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:21.571 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.562+0000 7f1aa143e700 1 -- 172.21.15.62:0/2037503613 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a8c00b200 con 0x7f1a9c0fab80 2023-11-15T06:25:21.571 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.562+0000 7f1aa143e700 1 --2- 172.21.15.62:0/2037503613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a9c0fab80 0x7f1a9c100e10 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f1a8c00b1d0 tx=0x7f1a8c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:21.572 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.562+0000 7f1a89ffb700 1 -- 172.21.15.62:0/2037503613 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a8c01b410 con 0x7f1a9c0fab80 2023-11-15T06:25:21.572 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.562+0000 7f1a89ffb700 1 -- 172.21.15.62:0/2037503613 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1a8c01b9e0 con 0x7f1a9c0fab80 2023-11-15T06:25:21.572 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.562+0000 7f1aa36a2700 1 -- 172.21.15.62:0/2037503613 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a9c0ff430 con 0x7f1a9c0fab80 2023-11-15T06:25:21.572 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.562+0000 7f1a89ffb700 1 -- 172.21.15.62:0/2037503613 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a8c01a840 con 0x7f1a9c0fab80 2023-11-15T06:25:21.572 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.563+0000 7f1aa36a2700 1 -- 172.21.15.62:0/2037503613 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a9c0ff840 con 0x7f1a9c0fab80 2023-11-15T06:25:21.573 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.563+0000 7f1a89ffb700 1 -- 172.21.15.62:0/2037503613 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f1a8c00f070 con 0x7f1a9c0fab80 2023-11-15T06:25:21.573 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.564+0000 7f1aa36a2700 1 -- 172.21.15.62:0/2037503613 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a9c045ad0 con 0x7f1a9c0fab80 2023-11-15T06:25:21.573 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.564+0000 7f1a89ffb700 1 --2- 172.21.15.62:0/2037503613 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f1a8402fcf0 0x7f1a840321b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:21.573 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.564+0000 7f1a89ffb700 1 -- 172.21.15.62:0/2037503613 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f1a8c041130 con 0x7f1a9c0fab80 2023-11-15T06:25:21.573 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.564+0000 7f1aa0c3d700 1 --2- 172.21.15.62:0/2037503613 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f1a8402fcf0 0x7f1a840321b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:21.574 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.564+0000 7f1aa0c3d700 1 --2- 172.21.15.62:0/2037503613 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f1a8402fcf0 0x7f1a840321b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f1a980096a0 tx=0x7f1a98006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:21.574 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.568+0000 7f1a89ffb700 1 -- 172.21.15.62:0/2037503613 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1a8c01a9a0 con 0x7f1a9c0fab80 2023-11-15T06:25:21.759 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.756+0000 7f1aa36a2700 1 -- 172.21.15.62:0/2037503613 --> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f1a9c0fca10 con 0x7f1a8402fcf0 2023-11-15T06:25:21.764 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.761+0000 7f1a89ffb700 1 -- 172.21.15.62:0/2037503613 <== mgr.14116 v2:172.21.15.62:6800/3280254567 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1a9c0fca10 con 0x7f1a8402fcf0 2023-11-15T06:25:21.767 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.763+0000 7f1aa36a2700 1 -- 172.21.15.62:0/2037503613 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f1a8402fcf0 msgr2=0x7f1a840321b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:21.767 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.763+0000 7f1aa36a2700 1 --2- 172.21.15.62:0/2037503613 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f1a8402fcf0 0x7f1a840321b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f1a980096a0 tx=0x7f1a98006b40).stop 2023-11-15T06:25:21.767 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.763+0000 7f1aa36a2700 1 -- 172.21.15.62:0/2037503613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a9c0fab80 msgr2=0x7f1a9c100e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:21.767 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.763+0000 7f1aa36a2700 1 --2- 172.21.15.62:0/2037503613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a9c0fab80 0x7f1a9c100e10 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f1a8c00b1d0 tx=0x7f1a8c006ee0).stop 2023-11-15T06:25:21.767 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.763+0000 7f1aa36a2700 1 -- 172.21.15.62:0/2037503613 shutdown_connections 2023-11-15T06:25:21.768 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.763+0000 7f1aa36a2700 1 --2- 172.21.15.62:0/2037503613 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f1a8402fcf0 0x7f1a840321b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:21.768 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.763+0000 7f1aa36a2700 1 --2- 172.21.15.62:0/2037503613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a9c0fab80 0x7f1a9c100e10 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:21.768 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.763+0000 7f1aa36a2700 1 -- 172.21.15.62:0/2037503613 >> 172.21.15.62:0/2037503613 conn(0x7f1a9c0f63c0 msgr2=0x7f1a9c070050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:21.768 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.764+0000 7f1aa36a2700 1 -- 172.21.15.62:0/2037503613 shutdown_connections 2023-11-15T06:25:21.768 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:21.764+0000 7f1aa36a2700 1 -- 172.21.15.62:0/2037503613 wait complete. 2023-11-15T06:25:22.052 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:21 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:20.853948+0000 mgr.smithi062.rwurtk (mgr.14116) 1 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-11-15T06:25:22.053 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:21 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:25:20.854750+0000 mon.smithi062 (mon.0) 49 : cluster [DBG] mgrmap e7: smithi062.rwurtk(active, since 1.05621s) 2023-11-15T06:25:22.053 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:21 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:20.861108+0000 mgr.smithi062.rwurtk (mgr.14116) 2 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-11-15T06:25:22.053 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:21 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:21.762954+0000 mon.smithi062 (mon.0) 50 : audit [INF] from='mgr.14116 172.21.15.62:0/1148526200' entity='mgr.smithi062.rwurtk' 2023-11-15T06:25:22.054 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:21 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:21.766411+0000 mon.smithi062 (mon.0) 51 : audit [DBG] from='mgr.14116 172.21.15.62:0/1148526200' entity='mgr.smithi062.rwurtk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T06:25:22.125 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi062 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptvq9ksio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpf12kfgcg:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2023-11-15T06:25:22.472 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.469+0000 7f4248471700 1 Processor -- start 2023-11-15T06:25:22.473 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.469+0000 7f4248471700 1 -- start start 2023-11-15T06:25:22.473 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.470+0000 7f4248471700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42400703b0 0x7f4240100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:22.473 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.470+0000 7f4248471700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42400fc750 con 0x7f42400703b0 2023-11-15T06:25:22.473 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.470+0000 7f424620d700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42400703b0 0x7f4240100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:22.474 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.470+0000 7f424620d700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42400703b0 0x7f4240100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56482/0 (socket says 172.21.15.62:56482) 2023-11-15T06:25:22.474 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.470+0000 7f424620d700 1 -- 172.21.15.62:0/3959042909 learned_addr learned my addr 172.21.15.62:0/3959042909 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:22.474 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.470+0000 7f424620d700 1 -- 172.21.15.62:0/3959042909 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4240101250 con 0x7f42400703b0 2023-11-15T06:25:22.478 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.470+0000 7f424620d700 1 --2- 172.21.15.62:0/3959042909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42400703b0 0x7f4240100d60 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f423000b5f0 tx=0x7f4230006ee0).ready entity=mon.0 client_cookie=a950d37f4d99da93 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:22.478 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.470+0000 7f424520b700 1 -- 172.21.15.62:0/3959042909 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f423000d600 con 0x7f42400703b0 2023-11-15T06:25:22.478 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.470+0000 7f424520b700 1 -- 172.21.15.62:0/3959042909 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f423000dbd0 con 0x7f42400703b0 2023-11-15T06:25:22.479 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.470+0000 7f424520b700 1 -- 172.21.15.62:0/3959042909 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4230011c00 con 0x7f42400703b0 2023-11-15T06:25:22.479 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.471+0000 7f4248471700 1 -- 172.21.15.62:0/3959042909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42400703b0 msgr2=0x7f4240100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:22.479 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.471+0000 7f4248471700 1 --2- 172.21.15.62:0/3959042909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42400703b0 0x7f4240100d60 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f423000b5f0 tx=0x7f4230006ee0).stop 2023-11-15T06:25:22.479 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.471+0000 7f4248471700 1 -- 172.21.15.62:0/3959042909 shutdown_connections 2023-11-15T06:25:22.479 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.471+0000 7f4248471700 1 --2- 172.21.15.62:0/3959042909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42400703b0 0x7f4240100d60 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:22.480 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.471+0000 7f4248471700 1 -- 172.21.15.62:0/3959042909 >> 172.21.15.62:0/3959042909 conn(0x7f42400f63c0 msgr2=0x7f42400f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:22.480 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.471+0000 7f4248471700 1 -- 172.21.15.62:0/3959042909 shutdown_connections 2023-11-15T06:25:22.480 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.471+0000 7f4248471700 1 -- 172.21.15.62:0/3959042909 wait complete. 2023-11-15T06:25:22.480 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.472+0000 7f4248471700 1 Processor -- start 2023-11-15T06:25:22.481 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.472+0000 7f4248471700 1 -- start start 2023-11-15T06:25:22.481 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.472+0000 7f4248471700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42400703b0 0x7f424010cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:22.482 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.472+0000 7f4248471700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f424010d100 con 0x7f42400703b0 2023-11-15T06:25:22.482 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.472+0000 7f424620d700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42400703b0 0x7f424010cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:22.482 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.472+0000 7f424620d700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42400703b0 0x7f424010cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56488/0 (socket says 172.21.15.62:56488) 2023-11-15T06:25:22.482 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.472+0000 7f424620d700 1 -- 172.21.15.62:0/3279740268 learned_addr learned my addr 172.21.15.62:0/3279740268 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:22.482 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.473+0000 7f424620d700 1 -- 172.21.15.62:0/3279740268 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f423000b200 con 0x7f42400703b0 2023-11-15T06:25:22.483 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.473+0000 7f424620d700 1 --2- 172.21.15.62:0/3279740268 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42400703b0 0x7f424010cc10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f423000b190 tx=0x7f4230006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:22.483 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.473+0000 7f422effd700 1 -- 172.21.15.62:0/3279740268 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f423001b410 con 0x7f42400703b0 2023-11-15T06:25:22.483 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.473+0000 7f422effd700 1 -- 172.21.15.62:0/3279740268 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f423001b9e0 con 0x7f42400703b0 2023-11-15T06:25:22.483 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.473+0000 7f4248471700 1 -- 172.21.15.62:0/3279740268 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f424010d300 con 0x7f42400703b0 2023-11-15T06:25:22.483 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.473+0000 7f422effd700 1 -- 172.21.15.62:0/3279740268 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f423001a880 con 0x7f42400703b0 2023-11-15T06:25:22.483 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.473+0000 7f4248471700 1 -- 172.21.15.62:0/3279740268 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f424010d710 con 0x7f42400703b0 2023-11-15T06:25:22.484 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.474+0000 7f422effd700 1 -- 172.21.15.62:0/3279740268 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f423000f070 con 0x7f42400703b0 2023-11-15T06:25:22.484 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.474+0000 7f422effd700 1 --2- 172.21.15.62:0/3279740268 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f422802fcf0 0x7f42280321b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:22.484 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.474+0000 7f422effd700 1 -- 172.21.15.62:0/3279740268 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f42300410b0 con 0x7f42400703b0 2023-11-15T06:25:22.484 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.474+0000 7f4248471700 1 -- 172.21.15.62:0/3279740268 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4240045ad0 con 0x7f42400703b0 2023-11-15T06:25:22.484 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.474+0000 7f4245a0c700 1 --2- 172.21.15.62:0/3279740268 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f422802fcf0 0x7f42280321b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:22.485 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.474+0000 7f4245a0c700 1 --2- 172.21.15.62:0/3279740268 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f422802fcf0 0x7f42280321b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f423c0096a0 tx=0x7f423c006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:22.485 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.478+0000 7f422effd700 1 -- 172.21.15.62:0/3279740268 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4230016070 con 0x7f42400703b0 2023-11-15T06:25:22.642 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.640+0000 7f4248471700 1 -- 172.21.15.62:0/3279740268 --> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f4240106b40 con 0x7f422802fcf0 2023-11-15T06:25:22.643 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.640+0000 7f422effd700 1 -- 172.21.15.62:0/3279740268 <== mgr.14116 v2:172.21.15.62:6800/3280254567 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f4240106b40 con 0x7f422802fcf0 2023-11-15T06:25:22.643 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: value unchanged 2023-11-15T06:25:22.645 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.642+0000 7f4248471700 1 -- 172.21.15.62:0/3279740268 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f422802fcf0 msgr2=0x7f42280321b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:22.645 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.642+0000 7f4248471700 1 --2- 172.21.15.62:0/3279740268 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f422802fcf0 0x7f42280321b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f423c0096a0 tx=0x7f423c006b40).stop 2023-11-15T06:25:22.645 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.642+0000 7f4248471700 1 -- 172.21.15.62:0/3279740268 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42400703b0 msgr2=0x7f424010cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:22.645 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.642+0000 7f4248471700 1 --2- 172.21.15.62:0/3279740268 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42400703b0 0x7f424010cc10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f423000b190 tx=0x7f4230006ee0).stop 2023-11-15T06:25:22.646 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.642+0000 7f4248471700 1 -- 172.21.15.62:0/3279740268 shutdown_connections 2023-11-15T06:25:22.646 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.642+0000 7f4248471700 1 --2- 172.21.15.62:0/3279740268 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f422802fcf0 0x7f42280321b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:22.646 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.642+0000 7f4248471700 1 --2- 172.21.15.62:0/3279740268 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42400703b0 0x7f424010cc10 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:22.646 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.642+0000 7f4248471700 1 -- 172.21.15.62:0/3279740268 >> 172.21.15.62:0/3279740268 conn(0x7f42400f63c0 msgr2=0x7f42400f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:22.647 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.642+0000 7f4248471700 1 -- 172.21.15.62:0/3279740268 shutdown_connections 2023-11-15T06:25:22.647 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:22.642+0000 7f4248471700 1 -- 172.21.15.62:0/3279740268 wait complete. 2023-11-15T06:25:23.016 INFO:teuthology.orchestra.run.smithi062.stderr:Generating ssh key... 2023-11-15T06:25:23.017 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi062 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptvq9ksio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpf12kfgcg:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2023-11-15T06:25:23.866 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:23 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:21.759141+0000 mgr.smithi062.rwurtk (mgr.14116) 3 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T06:25:23.866 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:23 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:25:22.763943+0000 mon.smithi062 (mon.0) 52 : cluster [DBG] mgrmap e8: smithi062.rwurtk(active, since 2s) 2023-11-15T06:25:24.478 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.475+0000 7f3964f19700 1 Processor -- start 2023-11-15T06:25:24.479 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.475+0000 7f3964f19700 1 -- start start 2023-11-15T06:25:24.479 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.476+0000 7f3964f19700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3960070430 0x7f39600fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:24.479 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.476+0000 7f3964f19700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39600fedd0 con 0x7f3960070430 2023-11-15T06:25:24.479 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.476+0000 7f395e59c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3960070430 0x7f39600fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:24.479 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.476+0000 7f395e59c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3960070430 0x7f39600fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56498/0 (socket says 172.21.15.62:56498) 2023-11-15T06:25:24.480 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.476+0000 7f395e59c700 1 -- 172.21.15.62:0/395595889 learned_addr learned my addr 172.21.15.62:0/395595889 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:24.482 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.476+0000 7f395e59c700 1 -- 172.21.15.62:0/395595889 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39600fef10 con 0x7f3960070430 2023-11-15T06:25:24.483 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.476+0000 7f395e59c700 1 --2- 172.21.15.62:0/395595889 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3960070430 0x7f39600fe8e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f395000b5f0 tx=0x7f3950006ee0).ready entity=mon.0 client_cookie=31f106adc617aed3 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:24.483 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.476+0000 7f395d59a700 1 -- 172.21.15.62:0/395595889 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f395000d600 con 0x7f3960070430 2023-11-15T06:25:24.483 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.476+0000 7f395d59a700 1 -- 172.21.15.62:0/395595889 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f395000dbd0 con 0x7f3960070430 2023-11-15T06:25:24.483 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.476+0000 7f395d59a700 1 -- 172.21.15.62:0/395595889 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3950011c00 con 0x7f3960070430 2023-11-15T06:25:24.483 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.477+0000 7f3964f19700 1 -- 172.21.15.62:0/395595889 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3960070430 msgr2=0x7f39600fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:24.484 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.477+0000 7f3964f19700 1 --2- 172.21.15.62:0/395595889 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3960070430 0x7f39600fe8e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f395000b5f0 tx=0x7f3950006ee0).stop 2023-11-15T06:25:24.484 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.477+0000 7f3964f19700 1 -- 172.21.15.62:0/395595889 shutdown_connections 2023-11-15T06:25:24.484 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.477+0000 7f3964f19700 1 --2- 172.21.15.62:0/395595889 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3960070430 0x7f39600fe8e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:24.484 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.477+0000 7f3964f19700 1 -- 172.21.15.62:0/395595889 >> 172.21.15.62:0/395595889 conn(0x7f39600f63c0 msgr2=0x7f39600f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:24.484 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.477+0000 7f3964f19700 1 -- 172.21.15.62:0/395595889 shutdown_connections 2023-11-15T06:25:24.485 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.477+0000 7f3964f19700 1 -- 172.21.15.62:0/395595889 wait complete. 2023-11-15T06:25:24.485 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.478+0000 7f3964f19700 1 Processor -- start 2023-11-15T06:25:24.485 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.478+0000 7f3964f19700 1 -- start start 2023-11-15T06:25:24.485 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.478+0000 7f3964f19700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3960070430 0x7f396010aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:24.485 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.478+0000 7f3964f19700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396010aef0 con 0x7f3960070430 2023-11-15T06:25:24.485 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.479+0000 7f395e59c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3960070430 0x7f396010aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:24.486 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.479+0000 7f395e59c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3960070430 0x7f396010aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56508/0 (socket says 172.21.15.62:56508) 2023-11-15T06:25:24.486 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.479+0000 7f395e59c700 1 -- 172.21.15.62:0/3555513422 learned_addr learned my addr 172.21.15.62:0/3555513422 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:24.486 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.479+0000 7f395e59c700 1 -- 172.21.15.62:0/3555513422 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f395000b200 con 0x7f3960070430 2023-11-15T06:25:24.486 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.479+0000 7f395e59c700 1 --2- 172.21.15.62:0/3555513422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3960070430 0x7f396010aa00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f395000b190 tx=0x7f3950006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:24.486 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.479+0000 7f394effd700 1 -- 172.21.15.62:0/3555513422 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f395001b410 con 0x7f3960070430 2023-11-15T06:25:24.487 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.479+0000 7f394effd700 1 -- 172.21.15.62:0/3555513422 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f395001b9e0 con 0x7f3960070430 2023-11-15T06:25:24.487 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.479+0000 7f394effd700 1 -- 172.21.15.62:0/3555513422 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f395001a880 con 0x7f3960070430 2023-11-15T06:25:24.487 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.479+0000 7f3964f19700 1 -- 172.21.15.62:0/3555513422 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f396010b0f0 con 0x7f3960070430 2023-11-15T06:25:24.487 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.479+0000 7f3964f19700 1 -- 172.21.15.62:0/3555513422 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f396010b500 con 0x7f3960070430 2023-11-15T06:25:24.487 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.480+0000 7f394effd700 1 -- 172.21.15.62:0/3555513422 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f395000f070 con 0x7f3960070430 2023-11-15T06:25:24.488 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.480+0000 7f394effd700 1 --2- 172.21.15.62:0/3555513422 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f394402fdb0 0x7f3944032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:24.488 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.480+0000 7f394effd700 1 -- 172.21.15.62:0/3555513422 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f39500411a0 con 0x7f3960070430 2023-11-15T06:25:24.488 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.480+0000 7f3964f19700 1 -- 172.21.15.62:0/3555513422 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3960045ad0 con 0x7f3960070430 2023-11-15T06:25:24.488 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.480+0000 7f395dd9b700 1 --2- 172.21.15.62:0/3555513422 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f394402fdb0 0x7f3944032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:24.489 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.481+0000 7f395dd9b700 1 --2- 172.21.15.62:0/3555513422 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f394402fdb0 0x7f3944032270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f39540096a0 tx=0x7f3954006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:24.489 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.485+0000 7f394effd700 1 -- 172.21.15.62:0/3555513422 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3950016070 con 0x7f3960070430 2023-11-15T06:25:24.647 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.645+0000 7f3964f19700 1 -- 172.21.15.62:0/3555513422 --> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f3960104960 con 0x7f394402fdb0 2023-11-15T06:25:24.864 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:24 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:22.641890+0000 mgr.smithi062.rwurtk (mgr.14116) 4 : audit [DBG] from='client.14128 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T06:25:24.870 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.868+0000 7f394effd700 1 -- 172.21.15.62:0/3555513422 <== mgr.14116 v2:172.21.15.62:6800/3280254567 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3960104960 con 0x7f394402fdb0 2023-11-15T06:25:24.878 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.870+0000 7f3964f19700 1 -- 172.21.15.62:0/3555513422 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f394402fdb0 msgr2=0x7f3944032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:24.878 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.870+0000 7f3964f19700 1 --2- 172.21.15.62:0/3555513422 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f394402fdb0 0x7f3944032270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f39540096a0 tx=0x7f3954006b40).stop 2023-11-15T06:25:24.878 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.870+0000 7f3964f19700 1 -- 172.21.15.62:0/3555513422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3960070430 msgr2=0x7f396010aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:24.879 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.870+0000 7f3964f19700 1 --2- 172.21.15.62:0/3555513422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3960070430 0x7f396010aa00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f395000b190 tx=0x7f3950006ee0).stop 2023-11-15T06:25:24.879 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.870+0000 7f3964f19700 1 -- 172.21.15.62:0/3555513422 shutdown_connections 2023-11-15T06:25:24.879 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.870+0000 7f3964f19700 1 --2- 172.21.15.62:0/3555513422 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f394402fdb0 0x7f3944032270 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:24.880 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.870+0000 7f3964f19700 1 --2- 172.21.15.62:0/3555513422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3960070430 0x7f396010aa00 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:24.880 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.870+0000 7f3964f19700 1 -- 172.21.15.62:0/3555513422 >> 172.21.15.62:0/3555513422 conn(0x7f39600f63c0 msgr2=0x7f39600f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:24.881 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.870+0000 7f3964f19700 1 -- 172.21.15.62:0/3555513422 shutdown_connections 2023-11-15T06:25:24.881 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:24.870+0000 7f3964f19700 1 -- 172.21.15.62:0/3555513422 wait complete. 2023-11-15T06:25:26.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:25 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cephadm 2023-11-15T06:25:24.647586+0000 mgr.smithi062.rwurtk (mgr.14116) 6 : cephadm [INF] Generating ssh key... 2023-11-15T06:25:26.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:25 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:24.866736+0000 mon.smithi062 (mon.0) 53 : audit [INF] from='mgr.14116 172.21.15.62:0/1148526200' entity='mgr.smithi062.rwurtk' 2023-11-15T06:25:26.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:25 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:24.869158+0000 mon.smithi062 (mon.0) 54 : audit [INF] from='mgr.14116 172.21.15.62:0/1148526200' entity='mgr.smithi062.rwurtk' 2023-11-15T06:25:26.875 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi062 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptvq9ksio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpf12kfgcg:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2023-11-15T06:25:28.404 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.291+0000 7fc996fcd700 1 Processor -- start 2023-11-15T06:25:28.404 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.291+0000 7fc996fcd700 1 -- start start 2023-11-15T06:25:28.405 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.292+0000 7fc996fcd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9900fbca0 0x7fc9900fc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:28.405 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.292+0000 7fc996fcd700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9900fc5b0 con 0x7fc9900fbca0 2023-11-15T06:25:28.405 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.292+0000 7fc994d69700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9900fbca0 0x7fc9900fc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:28.405 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.292+0000 7fc994d69700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9900fbca0 0x7fc9900fc0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56520/0 (socket says 172.21.15.62:56520) 2023-11-15T06:25:28.406 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.292+0000 7fc994d69700 1 -- 172.21.15.62:0/3525495827 learned_addr learned my addr 172.21.15.62:0/3525495827 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:28.406 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.292+0000 7fc994d69700 1 -- 172.21.15.62:0/3525495827 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9900fc6f0 con 0x7fc9900fbca0 2023-11-15T06:25:28.406 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.292+0000 7fc994d69700 1 --2- 172.21.15.62:0/3525495827 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9900fbca0 0x7fc9900fc0c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fc98400b5f0 tx=0x7fc984006ee0).ready entity=mon.0 client_cookie=f41bac20b9187112 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:28.406 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.292+0000 7fc98f7fe700 1 -- 172.21.15.62:0/3525495827 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc98400d600 con 0x7fc9900fbca0 2023-11-15T06:25:28.407 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.292+0000 7fc98f7fe700 1 -- 172.21.15.62:0/3525495827 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc98400dbd0 con 0x7fc9900fbca0 2023-11-15T06:25:28.407 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.292+0000 7fc98f7fe700 1 -- 172.21.15.62:0/3525495827 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc984011c00 con 0x7fc9900fbca0 2023-11-15T06:25:28.407 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.293+0000 7fc996fcd700 1 -- 172.21.15.62:0/3525495827 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9900fbca0 msgr2=0x7fc9900fc0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:28.407 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.293+0000 7fc996fcd700 1 --2- 172.21.15.62:0/3525495827 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9900fbca0 0x7fc9900fc0c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fc98400b5f0 tx=0x7fc984006ee0).stop 2023-11-15T06:25:28.407 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.293+0000 7fc996fcd700 1 -- 172.21.15.62:0/3525495827 shutdown_connections 2023-11-15T06:25:28.408 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.293+0000 7fc996fcd700 1 --2- 172.21.15.62:0/3525495827 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9900fbca0 0x7fc9900fc0c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:28.408 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.293+0000 7fc996fcd700 1 -- 172.21.15.62:0/3525495827 >> 172.21.15.62:0/3525495827 conn(0x7fc9900f74c0 msgr2=0x7fc9900f98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:28.408 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.293+0000 7fc996fcd700 1 -- 172.21.15.62:0/3525495827 shutdown_connections 2023-11-15T06:25:28.408 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.293+0000 7fc996fcd700 1 -- 172.21.15.62:0/3525495827 wait complete. 2023-11-15T06:25:28.408 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.294+0000 7fc996fcd700 1 Processor -- start 2023-11-15T06:25:28.409 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.294+0000 7fc996fcd700 1 -- start start 2023-11-15T06:25:28.409 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.294+0000 7fc996fcd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc99010aae0 0x7fc99010af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:28.409 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.294+0000 7fc996fcd700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc99010b3f0 con 0x7fc99010aae0 2023-11-15T06:25:28.410 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.294+0000 7fc994d69700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc99010aae0 0x7fc99010af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:28.410 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.294+0000 7fc994d69700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc99010aae0 0x7fc99010af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56522/0 (socket says 172.21.15.62:56522) 2023-11-15T06:25:28.410 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.294+0000 7fc994d69700 1 -- 172.21.15.62:0/1418668841 learned_addr learned my addr 172.21.15.62:0/1418668841 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:28.410 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.294+0000 7fc994d69700 1 -- 172.21.15.62:0/1418668841 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc98400b200 con 0x7fc99010aae0 2023-11-15T06:25:28.411 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.294+0000 7fc994d69700 1 --2- 172.21.15.62:0/1418668841 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc99010aae0 0x7fc99010af00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fc984011820 tx=0x7fc984006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:28.411 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.295+0000 7fc98d7fa700 1 -- 172.21.15.62:0/1418668841 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc98401b410 con 0x7fc99010aae0 2023-11-15T06:25:28.411 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.295+0000 7fc98d7fa700 1 -- 172.21.15.62:0/1418668841 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc98401b9e0 con 0x7fc99010aae0 2023-11-15T06:25:28.411 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.295+0000 7fc996fcd700 1 -- 172.21.15.62:0/1418668841 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc99010b5f0 con 0x7fc99010aae0 2023-11-15T06:25:28.412 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.295+0000 7fc98d7fa700 1 -- 172.21.15.62:0/1418668841 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc98401a920 con 0x7fc99010aae0 2023-11-15T06:25:28.412 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.295+0000 7fc996fcd700 1 -- 172.21.15.62:0/1418668841 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc99010c0c0 con 0x7fc99010aae0 2023-11-15T06:25:28.412 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.295+0000 7fc98d7fa700 1 -- 172.21.15.62:0/1418668841 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fc98400f070 con 0x7fc99010aae0 2023-11-15T06:25:28.412 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.296+0000 7fc98d7fa700 1 --2- 172.21.15.62:0/1418668841 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7fc97402fdb0 0x7fc974032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:28.412 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.296+0000 7fc98d7fa700 1 -- 172.21.15.62:0/1418668841 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fc984040fa0 con 0x7fc99010aae0 2023-11-15T06:25:28.413 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.296+0000 7fc996fcd700 1 -- 172.21.15.62:0/1418668841 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc990059100 con 0x7fc99010aae0 2023-11-15T06:25:28.413 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.296+0000 7fc98ffff700 1 --2- 172.21.15.62:0/1418668841 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7fc97402fdb0 0x7fc974032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:28.413 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.296+0000 7fc98ffff700 1 --2- 172.21.15.62:0/1418668841 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7fc97402fdb0 0x7fc974032270 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fc97c0096a0 tx=0x7fc97c006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:28.413 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.300+0000 7fc98d7fa700 1 -- 172.21.15.62:0/1418668841 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc984016070 con 0x7fc99010aae0 2023-11-15T06:25:28.462 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.460+0000 7fc996fcd700 1 -- 172.21.15.62:0/1418668841 --> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fc990104540 con 0x7fc97402fdb0 2023-11-15T06:25:28.463 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.460+0000 7fc98d7fa700 1 -- 172.21.15.62:0/1418668841 <== mgr.14116 v2:172.21.15.62:6800/3280254567 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fc990104540 con 0x7fc97402fdb0 2023-11-15T06:25:28.463 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5+aqffi9x3Pq9kjlnQYPUXA/1lo4stG9BltXRno7qEJ/+OWtYt5Wmb9jCJ33mRCfVPPj1QKZDTHse1niDh1wX7rkYZocmeUMS5P7xr5YA/DP5sqqP87WsOjWBMiRDK5yUWlD2T7bVUO8QhLGHJFYfhrARhnqUxnhFwF2jkKao9Z1pB+PG8bTmv+S8nio8EfDEwOzLtuSjo8Q9ZLekPhc4UpWuUIqB0xLQxwSIVst/0EE2sYME+DfXCZMiYSEZuvf1Fs0cEOXdfR6JpR10YR3HIhvQ2h7P8csIq842USQCOsWhIXRh8/L8S2wiprO0vb9tnZ96tiX7sj1YrKLoMlgnLc7BbHlBOwceF999ljHiHPGG1Xhd8X9Gp6kMoXIpcBzcN3kkeW/EeLsDp+v3YJlU4JcObznE7Xf7g+9XgIAbTKse5lP6k9rr8dhjByeW9zLxTVxtcKWX6uhKDGPjEdbbL1QDVPhNhlPWhGZhS8+WxZwp7sPgN1yYNJRP9+cwOeU= ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7 2023-11-15T06:25:28.465 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.462+0000 7fc996fcd700 1 -- 172.21.15.62:0/1418668841 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7fc97402fdb0 msgr2=0x7fc974032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:28.465 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.462+0000 7fc996fcd700 1 --2- 172.21.15.62:0/1418668841 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7fc97402fdb0 0x7fc974032270 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fc97c0096a0 tx=0x7fc97c006b40).stop 2023-11-15T06:25:28.465 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.462+0000 7fc996fcd700 1 -- 172.21.15.62:0/1418668841 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc99010aae0 msgr2=0x7fc99010af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:28.466 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.462+0000 7fc996fcd700 1 --2- 172.21.15.62:0/1418668841 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc99010aae0 0x7fc99010af00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fc984011820 tx=0x7fc984006ee0).stop 2023-11-15T06:25:28.466 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.462+0000 7fc996fcd700 1 -- 172.21.15.62:0/1418668841 shutdown_connections 2023-11-15T06:25:28.466 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.462+0000 7fc996fcd700 1 --2- 172.21.15.62:0/1418668841 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7fc97402fdb0 0x7fc974032270 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:28.466 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.462+0000 7fc996fcd700 1 --2- 172.21.15.62:0/1418668841 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc99010aae0 0x7fc99010af00 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:28.467 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.462+0000 7fc996fcd700 1 -- 172.21.15.62:0/1418668841 >> 172.21.15.62:0/1418668841 conn(0x7fc9900f74c0 msgr2=0x7fc9900f9040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:28.467 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.462+0000 7fc996fcd700 1 -- 172.21.15.62:0/1418668841 shutdown_connections 2023-11-15T06:25:28.467 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:28.462+0000 7fc996fcd700 1 -- 172.21.15.62:0/1418668841 wait complete. 2023-11-15T06:25:28.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:28 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:28.462188+0000 mgr.smithi062.rwurtk (mgr.14116) 7 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T06:25:29.987 INFO:teuthology.orchestra.run.smithi062.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-11-15T06:25:29.988 INFO:teuthology.orchestra.run.smithi062.stderr:Adding key to root@localhost authorized_keys... 2023-11-15T06:25:29.989 INFO:teuthology.orchestra.run.smithi062.stderr:Adding host smithi062... 2023-11-15T06:25:29.989 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi062 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptvq9ksio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpf12kfgcg:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi062 172.21.15.62 2023-11-15T06:25:31.060 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.055+0000 7fe29a7ba700 1 Processor -- start 2023-11-15T06:25:31.060 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.055+0000 7fe29a7ba700 1 -- start start 2023-11-15T06:25:31.061 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.056+0000 7fe29a7ba700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2940fab80 0x7fe2940fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:31.061 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.056+0000 7fe29a7ba700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2940fb490 con 0x7fe2940fab80 2023-11-15T06:25:31.061 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.056+0000 7fe293fff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2940fab80 0x7fe2940fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:31.061 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.056+0000 7fe293fff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2940fab80 0x7fe2940fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42776/0 (socket says 172.21.15.62:42776) 2023-11-15T06:25:31.062 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.056+0000 7fe293fff700 1 -- 172.21.15.62:0/123004057 learned_addr learned my addr 172.21.15.62:0/123004057 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:31.062 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.056+0000 7fe293fff700 1 -- 172.21.15.62:0/123004057 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2940fb5d0 con 0x7fe2940fab80 2023-11-15T06:25:31.062 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.056+0000 7fe293fff700 1 --2- 172.21.15.62:0/123004057 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2940fab80 0x7fe2940fafa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fe28400b5f0 tx=0x7fe284006ee0).ready entity=mon.0 client_cookie=a6e4de54dbf03a4e server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:31.062 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.057+0000 7fe292ffd700 1 -- 172.21.15.62:0/123004057 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe28400d600 con 0x7fe2940fab80 2023-11-15T06:25:31.062 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.057+0000 7fe292ffd700 1 -- 172.21.15.62:0/123004057 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe28400dbd0 con 0x7fe2940fab80 2023-11-15T06:25:31.063 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.057+0000 7fe292ffd700 1 -- 172.21.15.62:0/123004057 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe284011c00 con 0x7fe2940fab80 2023-11-15T06:25:31.063 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.057+0000 7fe29a7ba700 1 -- 172.21.15.62:0/123004057 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2940fab80 msgr2=0x7fe2940fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:31.063 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.057+0000 7fe29a7ba700 1 --2- 172.21.15.62:0/123004057 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2940fab80 0x7fe2940fafa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fe28400b5f0 tx=0x7fe284006ee0).stop 2023-11-15T06:25:31.063 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.057+0000 7fe29a7ba700 1 -- 172.21.15.62:0/123004057 shutdown_connections 2023-11-15T06:25:31.063 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.057+0000 7fe29a7ba700 1 --2- 172.21.15.62:0/123004057 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2940fab80 0x7fe2940fafa0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:31.064 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.057+0000 7fe29a7ba700 1 -- 172.21.15.62:0/123004057 >> 172.21.15.62:0/123004057 conn(0x7fe2940f63c0 msgr2=0x7fe2940f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:31.066 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.057+0000 7fe29a7ba700 1 -- 172.21.15.62:0/123004057 shutdown_connections 2023-11-15T06:25:31.066 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.057+0000 7fe29a7ba700 1 -- 172.21.15.62:0/123004057 wait complete. 2023-11-15T06:25:31.066 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.058+0000 7fe29a7ba700 1 Processor -- start 2023-11-15T06:25:31.071 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.058+0000 7fe29a7ba700 1 -- start start 2023-11-15T06:25:31.071 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.058+0000 7fe29a7ba700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2940fab80 0x7fe2941087d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:31.071 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.058+0000 7fe29a7ba700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe294108cc0 con 0x7fe2940fab80 2023-11-15T06:25:31.071 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.058+0000 7fe293fff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2940fab80 0x7fe2941087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:31.072 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.058+0000 7fe293fff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2940fab80 0x7fe2941087d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42782/0 (socket says 172.21.15.62:42782) 2023-11-15T06:25:31.072 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.058+0000 7fe293fff700 1 -- 172.21.15.62:0/1309042871 learned_addr learned my addr 172.21.15.62:0/1309042871 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:31.072 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.059+0000 7fe293fff700 1 -- 172.21.15.62:0/1309042871 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe28400b200 con 0x7fe2940fab80 2023-11-15T06:25:31.072 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.059+0000 7fe293fff700 1 --2- 172.21.15.62:0/1309042871 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2940fab80 0x7fe2941087d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fe28400b1d0 tx=0x7fe284006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:31.072 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.059+0000 7fe290ff9700 1 -- 172.21.15.62:0/1309042871 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe28401b410 con 0x7fe2940fab80 2023-11-15T06:25:31.072 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.059+0000 7fe290ff9700 1 -- 172.21.15.62:0/1309042871 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe28401b9e0 con 0x7fe2940fab80 2023-11-15T06:25:31.073 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.059+0000 7fe29a7ba700 1 -- 172.21.15.62:0/1309042871 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe294108ec0 con 0x7fe2940fab80 2023-11-15T06:25:31.073 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.059+0000 7fe290ff9700 1 -- 172.21.15.62:0/1309042871 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe28401a840 con 0x7fe2940fab80 2023-11-15T06:25:31.073 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.059+0000 7fe29a7ba700 1 -- 172.21.15.62:0/1309042871 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2941092d0 con 0x7fe2940fab80 2023-11-15T06:25:31.073 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.060+0000 7fe290ff9700 1 -- 172.21.15.62:0/1309042871 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fe28400f070 con 0x7fe2940fab80 2023-11-15T06:25:31.073 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.060+0000 7fe290ff9700 1 --2- 172.21.15.62:0/1309042871 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7fe274031fa0 0x7fe274034460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:31.074 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.060+0000 7fe290ff9700 1 -- 172.21.15.62:0/1309042871 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fe284042150 con 0x7fe2940fab80 2023-11-15T06:25:31.074 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.060+0000 7fe2937fe700 1 --2- 172.21.15.62:0/1309042871 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7fe274031fa0 0x7fe274034460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:31.074 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.060+0000 7fe29a7ba700 1 -- 172.21.15.62:0/1309042871 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe294059100 con 0x7fe2940fab80 2023-11-15T06:25:31.074 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.061+0000 7fe2937fe700 1 --2- 172.21.15.62:0/1309042871 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7fe274031fa0 0x7fe274034460 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fe27c0096a0 tx=0x7fe27c006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:31.075 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.065+0000 7fe290ff9700 1 -- 172.21.15.62:0/1309042871 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe284016070 con 0x7fe2940fab80 2023-11-15T06:25:31.249 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.246+0000 7fe29a7ba700 1 -- 172.21.15.62:0/1309042871 --> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi062", "addr": "172.21.15.62", "target": ["mon-mgr", ""]}) v1 -- 0x7fe2941026d0 con 0x7fe274031fa0 2023-11-15T06:25:31.862 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.859+0000 7fe290ff9700 1 -- 172.21.15.62:0/1309042871 <== mgr.14116 v2:172.21.15.62:6800/3280254567 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7fe2941026d0 con 0x7fe274031fa0 2023-11-15T06:25:31.863 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: Added host 'smithi062' 2023-11-15T06:25:31.864 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.861+0000 7fe29a7ba700 1 -- 172.21.15.62:0/1309042871 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7fe274031fa0 msgr2=0x7fe274034460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:31.864 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.861+0000 7fe29a7ba700 1 --2- 172.21.15.62:0/1309042871 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7fe274031fa0 0x7fe274034460 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fe27c0096a0 tx=0x7fe27c006b40).stop 2023-11-15T06:25:31.864 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.861+0000 7fe29a7ba700 1 -- 172.21.15.62:0/1309042871 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2940fab80 msgr2=0x7fe2941087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:31.865 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.861+0000 7fe29a7ba700 1 --2- 172.21.15.62:0/1309042871 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2940fab80 0x7fe2941087d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fe28400b1d0 tx=0x7fe284006ee0).stop 2023-11-15T06:25:31.865 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.861+0000 7fe29a7ba700 1 -- 172.21.15.62:0/1309042871 shutdown_connections 2023-11-15T06:25:31.865 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.861+0000 7fe29a7ba700 1 --2- 172.21.15.62:0/1309042871 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7fe274031fa0 0x7fe274034460 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:31.865 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.861+0000 7fe29a7ba700 1 --2- 172.21.15.62:0/1309042871 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe2940fab80 0x7fe2941087d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:31.866 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.861+0000 7fe29a7ba700 1 -- 172.21.15.62:0/1309042871 >> 172.21.15.62:0/1309042871 conn(0x7fe2940f63c0 msgr2=0x7fe2940ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:31.866 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.861+0000 7fe29a7ba700 1 -- 172.21.15.62:0/1309042871 shutdown_connections 2023-11-15T06:25:31.866 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:31.861+0000 7fe29a7ba700 1 -- 172.21.15.62:0/1309042871 wait complete. 2023-11-15T06:25:32.057 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:31 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:31.248736+0000 mgr.smithi062.rwurtk (mgr.14116) 8 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi062", "addr": "172.21.15.62", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T06:25:32.182 INFO:teuthology.orchestra.run.smithi062.stderr:Deploying mon service with default placement... 2023-11-15T06:25:32.183 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi062 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptvq9ksio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpf12kfgcg:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2023-11-15T06:25:32.516 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.512+0000 7f29ee98a700 1 Processor -- start 2023-11-15T06:25:32.516 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.512+0000 7f29ee98a700 1 -- start start 2023-11-15T06:25:32.516 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.513+0000 7f29ee98a700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29e80fab80 0x7f29e80fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:32.517 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.513+0000 7f29ee98a700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e80fb490 con 0x7f29e80fab80 2023-11-15T06:25:32.517 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.513+0000 7f29e7fff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29e80fab80 0x7f29e80fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:32.517 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.513+0000 7f29e7fff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29e80fab80 0x7f29e80fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42794/0 (socket says 172.21.15.62:42794) 2023-11-15T06:25:32.517 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.513+0000 7f29e7fff700 1 -- 172.21.15.62:0/1180043719 learned_addr learned my addr 172.21.15.62:0/1180043719 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:32.518 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.513+0000 7f29e7fff700 1 -- 172.21.15.62:0/1180043719 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e80fb5d0 con 0x7f29e80fab80 2023-11-15T06:25:32.519 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.513+0000 7f29e7fff700 1 --2- 172.21.15.62:0/1180043719 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29e80fab80 0x7f29e80fafa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f29d800b5f0 tx=0x7f29d8006ee0).ready entity=mon.0 client_cookie=90d700ea63a0a366 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:32.519 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.513+0000 7f29e6ffd700 1 -- 172.21.15.62:0/1180043719 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29d800d600 con 0x7f29e80fab80 2023-11-15T06:25:32.519 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.513+0000 7f29e6ffd700 1 -- 172.21.15.62:0/1180043719 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f29d800dbd0 con 0x7f29e80fab80 2023-11-15T06:25:32.519 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.513+0000 7f29e6ffd700 1 -- 172.21.15.62:0/1180043719 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29d8011c00 con 0x7f29e80fab80 2023-11-15T06:25:32.520 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.514+0000 7f29ee98a700 1 -- 172.21.15.62:0/1180043719 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29e80fab80 msgr2=0x7f29e80fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:32.520 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.514+0000 7f29ee98a700 1 --2- 172.21.15.62:0/1180043719 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29e80fab80 0x7f29e80fafa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f29d800b5f0 tx=0x7f29d8006ee0).stop 2023-11-15T06:25:32.520 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.514+0000 7f29ee98a700 1 -- 172.21.15.62:0/1180043719 shutdown_connections 2023-11-15T06:25:32.520 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.514+0000 7f29ee98a700 1 --2- 172.21.15.62:0/1180043719 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29e80fab80 0x7f29e80fafa0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:32.520 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.514+0000 7f29ee98a700 1 -- 172.21.15.62:0/1180043719 >> 172.21.15.62:0/1180043719 conn(0x7f29e80f63c0 msgr2=0x7f29e80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:32.520 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.514+0000 7f29ee98a700 1 -- 172.21.15.62:0/1180043719 shutdown_connections 2023-11-15T06:25:32.521 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.514+0000 7f29ee98a700 1 -- 172.21.15.62:0/1180043719 wait complete. 2023-11-15T06:25:32.521 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.514+0000 7f29ee98a700 1 Processor -- start 2023-11-15T06:25:32.521 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.515+0000 7f29ee98a700 1 -- start start 2023-11-15T06:25:32.521 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.515+0000 7f29ee98a700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29e80fab80 0x7f29e810ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:32.521 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.515+0000 7f29ee98a700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e810d1b0 con 0x7f29e80fab80 2023-11-15T06:25:32.522 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.515+0000 7f29e7fff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29e80fab80 0x7f29e810ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:32.522 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.515+0000 7f29e7fff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29e80fab80 0x7f29e810ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42800/0 (socket says 172.21.15.62:42800) 2023-11-15T06:25:32.522 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.515+0000 7f29e7fff700 1 -- 172.21.15.62:0/4269218136 learned_addr learned my addr 172.21.15.62:0/4269218136 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:32.522 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.515+0000 7f29e7fff700 1 -- 172.21.15.62:0/4269218136 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d800b200 con 0x7f29e80fab80 2023-11-15T06:25:32.522 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.515+0000 7f29e7fff700 1 --2- 172.21.15.62:0/4269218136 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29e80fab80 0x7f29e810ccc0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f29d800b1d0 tx=0x7f29d8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:32.523 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.516+0000 7f29e4ff9700 1 -- 172.21.15.62:0/4269218136 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29d801b410 con 0x7f29e80fab80 2023-11-15T06:25:32.523 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.516+0000 7f29e4ff9700 1 -- 172.21.15.62:0/4269218136 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f29d801b9e0 con 0x7f29e80fab80 2023-11-15T06:25:32.523 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.516+0000 7f29ee98a700 1 -- 172.21.15.62:0/4269218136 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29e810d3b0 con 0x7f29e80fab80 2023-11-15T06:25:32.523 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.516+0000 7f29e4ff9700 1 -- 172.21.15.62:0/4269218136 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29d801a840 con 0x7f29e80fab80 2023-11-15T06:25:32.523 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.516+0000 7f29ee98a700 1 -- 172.21.15.62:0/4269218136 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29e810d7c0 con 0x7f29e80fab80 2023-11-15T06:25:32.524 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.517+0000 7f29e4ff9700 1 -- 172.21.15.62:0/4269218136 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f29d800f070 con 0x7f29e80fab80 2023-11-15T06:25:32.525 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.517+0000 7f29e4ff9700 1 --2- 172.21.15.62:0/4269218136 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f29c802fdb0 0x7f29c8032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:32.525 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.517+0000 7f29ee98a700 1 -- 172.21.15.62:0/4269218136 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29e8045ad0 con 0x7f29e80fab80 2023-11-15T06:25:32.525 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.517+0000 7f29e4ff9700 1 -- 172.21.15.62:0/4269218136 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f29d8041200 con 0x7f29e80fab80 2023-11-15T06:25:32.525 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.517+0000 7f29e77fe700 1 --2- 172.21.15.62:0/4269218136 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f29c802fdb0 0x7f29c8032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:32.525 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.517+0000 7f29e77fe700 1 --2- 172.21.15.62:0/4269218136 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f29c802fdb0 0x7f29c8032270 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f29d00096a0 tx=0x7f29d0006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:32.526 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.521+0000 7f29e4ff9700 1 -- 172.21.15.62:0/4269218136 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f29d8025460 con 0x7f29e80fab80 2023-11-15T06:25:32.710 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.707+0000 7f29ee98a700 1 -- 172.21.15.62:0/4269218136 --> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f29e810e020 con 0x7f29c802fdb0 2023-11-15T06:25:32.714 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.712+0000 7f29e4ff9700 1 -- 172.21.15.62:0/4269218136 <== mgr.14116 v2:172.21.15.62:6800/3280254567 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f29e810e020 con 0x7f29c802fdb0 2023-11-15T06:25:32.714 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: Scheduled mon update... 2023-11-15T06:25:32.717 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.714+0000 7f29ee98a700 1 -- 172.21.15.62:0/4269218136 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f29c802fdb0 msgr2=0x7f29c8032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:32.717 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.714+0000 7f29ee98a700 1 --2- 172.21.15.62:0/4269218136 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f29c802fdb0 0x7f29c8032270 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f29d00096a0 tx=0x7f29d0006b40).stop 2023-11-15T06:25:32.717 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.714+0000 7f29ee98a700 1 -- 172.21.15.62:0/4269218136 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29e80fab80 msgr2=0x7f29e810ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:32.718 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.714+0000 7f29ee98a700 1 --2- 172.21.15.62:0/4269218136 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29e80fab80 0x7f29e810ccc0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f29d800b1d0 tx=0x7f29d8006ee0).stop 2023-11-15T06:25:32.718 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.714+0000 7f29ee98a700 1 -- 172.21.15.62:0/4269218136 shutdown_connections 2023-11-15T06:25:32.718 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.714+0000 7f29ee98a700 1 --2- 172.21.15.62:0/4269218136 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f29c802fdb0 0x7f29c8032270 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:32.718 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.714+0000 7f29ee98a700 1 --2- 172.21.15.62:0/4269218136 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29e80fab80 0x7f29e810ccc0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:32.719 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.714+0000 7f29ee98a700 1 -- 172.21.15.62:0/4269218136 >> 172.21.15.62:0/4269218136 conn(0x7f29e80f63c0 msgr2=0x7f29e8070090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:32.719 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.714+0000 7f29ee98a700 1 -- 172.21.15.62:0/4269218136 shutdown_connections 2023-11-15T06:25:32.719 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:32.714+0000 7f29ee98a700 1 -- 172.21.15.62:0/4269218136 wait complete. 2023-11-15T06:25:33.016 INFO:teuthology.orchestra.run.smithi062.stderr:Deploying mgr service with default placement... 2023-11-15T06:25:33.016 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi062 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptvq9ksio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpf12kfgcg:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2023-11-15T06:25:33.018 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:32 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:31.856297+0000 mon.smithi062 (mon.0) 55 : audit [INF] from='mgr.14116 172.21.15.62:0/1148526200' entity='mgr.smithi062.rwurtk' 2023-11-15T06:25:33.018 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:32 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cephadm 2023-11-15T06:25:31.859954+0000 mgr.smithi062.rwurtk (mgr.14116) 9 : cephadm [INF] Added host smithi062 2023-11-15T06:25:33.018 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:32 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:31.860291+0000 mon.smithi062 (mon.0) 56 : audit [DBG] from='mgr.14116 172.21.15.62:0/1148526200' entity='mgr.smithi062.rwurtk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T06:25:33.019 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:32 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:32.713471+0000 mon.smithi062 (mon.0) 57 : audit [INF] from='mgr.14116 172.21.15.62:0/1148526200' entity='mgr.smithi062.rwurtk' 2023-11-15T06:25:33.355 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.346+0000 7fb396590700 1 Processor -- start 2023-11-15T06:25:33.356 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.347+0000 7fb396590700 1 -- start start 2023-11-15T06:25:33.356 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.347+0000 7fb396590700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3900fab90 0x7fb3900fafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:33.356 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.347+0000 7fb396590700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3900fb4a0 con 0x7fb3900fab90 2023-11-15T06:25:33.357 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.347+0000 7fb38ffff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3900fab90 0x7fb3900fafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:33.357 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.347+0000 7fb38ffff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3900fab90 0x7fb3900fafb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42806/0 (socket says 172.21.15.62:42806) 2023-11-15T06:25:33.358 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.347+0000 7fb38ffff700 1 -- 172.21.15.62:0/2773166130 learned_addr learned my addr 172.21.15.62:0/2773166130 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:33.358 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.347+0000 7fb38ffff700 1 -- 172.21.15.62:0/2773166130 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3900fb5e0 con 0x7fb3900fab90 2023-11-15T06:25:33.358 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.348+0000 7fb38ffff700 1 --2- 172.21.15.62:0/2773166130 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3900fab90 0x7fb3900fafb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fb37800b5f0 tx=0x7fb378006ee0).ready entity=mon.0 client_cookie=aa225c50432e5ad4 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:33.359 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.348+0000 7fb38effd700 1 -- 172.21.15.62:0/2773166130 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb37800d600 con 0x7fb3900fab90 2023-11-15T06:25:33.359 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.348+0000 7fb38effd700 1 -- 172.21.15.62:0/2773166130 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb37800dbd0 con 0x7fb3900fab90 2023-11-15T06:25:33.359 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.348+0000 7fb38effd700 1 -- 172.21.15.62:0/2773166130 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb378011c00 con 0x7fb3900fab90 2023-11-15T06:25:33.360 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.348+0000 7fb396590700 1 -- 172.21.15.62:0/2773166130 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3900fab90 msgr2=0x7fb3900fafb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:33.360 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.348+0000 7fb396590700 1 --2- 172.21.15.62:0/2773166130 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3900fab90 0x7fb3900fafb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fb37800b5f0 tx=0x7fb378006ee0).stop 2023-11-15T06:25:33.361 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.348+0000 7fb396590700 1 -- 172.21.15.62:0/2773166130 shutdown_connections 2023-11-15T06:25:33.361 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.348+0000 7fb396590700 1 --2- 172.21.15.62:0/2773166130 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3900fab90 0x7fb3900fafb0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:33.361 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.348+0000 7fb396590700 1 -- 172.21.15.62:0/2773166130 >> 172.21.15.62:0/2773166130 conn(0x7fb3900f63b0 msgr2=0x7fb3900f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:33.362 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.348+0000 7fb396590700 1 -- 172.21.15.62:0/2773166130 shutdown_connections 2023-11-15T06:25:33.362 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.348+0000 7fb396590700 1 -- 172.21.15.62:0/2773166130 wait complete. 2023-11-15T06:25:33.363 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.349+0000 7fb396590700 1 Processor -- start 2023-11-15T06:25:33.363 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.349+0000 7fb396590700 1 -- start start 2023-11-15T06:25:33.363 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.349+0000 7fb396590700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3900fab90 0x7fb39010a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:33.364 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.349+0000 7fb396590700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb39010aee0 con 0x7fb3900fab90 2023-11-15T06:25:33.364 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.349+0000 7fb38ffff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3900fab90 0x7fb39010a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:33.364 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.349+0000 7fb38ffff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3900fab90 0x7fb39010a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42814/0 (socket says 172.21.15.62:42814) 2023-11-15T06:25:33.365 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.349+0000 7fb38ffff700 1 -- 172.21.15.62:0/3967142875 learned_addr learned my addr 172.21.15.62:0/3967142875 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:33.365 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.350+0000 7fb38ffff700 1 -- 172.21.15.62:0/3967142875 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb37800b200 con 0x7fb3900fab90 2023-11-15T06:25:33.365 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.350+0000 7fb38ffff700 1 --2- 172.21.15.62:0/3967142875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3900fab90 0x7fb39010a9f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fb37800b1d0 tx=0x7fb378006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:33.366 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.350+0000 7fb38cff9700 1 -- 172.21.15.62:0/3967142875 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb37801b410 con 0x7fb3900fab90 2023-11-15T06:25:33.366 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.350+0000 7fb396590700 1 -- 172.21.15.62:0/3967142875 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb39010b0e0 con 0x7fb3900fab90 2023-11-15T06:25:33.366 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.350+0000 7fb38cff9700 1 -- 172.21.15.62:0/3967142875 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb37801b9e0 con 0x7fb3900fab90 2023-11-15T06:25:33.367 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.350+0000 7fb38cff9700 1 -- 172.21.15.62:0/3967142875 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb37801a840 con 0x7fb3900fab90 2023-11-15T06:25:33.367 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.350+0000 7fb396590700 1 -- 172.21.15.62:0/3967142875 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb39010b4f0 con 0x7fb3900fab90 2023-11-15T06:25:33.367 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.351+0000 7fb38cff9700 1 -- 172.21.15.62:0/3967142875 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fb37800f070 con 0x7fb3900fab90 2023-11-15T06:25:33.368 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.351+0000 7fb38cff9700 1 --2- 172.21.15.62:0/3967142875 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7fb37002fd50 0x7fb370032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:33.368 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.351+0000 7fb38cff9700 1 -- 172.21.15.62:0/3967142875 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fb378041290 con 0x7fb3900fab90 2023-11-15T06:25:33.368 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.351+0000 7fb38f7fe700 1 --2- 172.21.15.62:0/3967142875 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7fb37002fd50 0x7fb370032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:33.369 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.351+0000 7fb396590700 1 -- 172.21.15.62:0/3967142875 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb390059100 con 0x7fb3900fab90 2023-11-15T06:25:33.369 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.351+0000 7fb38f7fe700 1 --2- 172.21.15.62:0/3967142875 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7fb37002fd50 0x7fb370032210 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fb3800096a0 tx=0x7fb380006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:33.370 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.356+0000 7fb38cff9700 1 -- 172.21.15.62:0/3967142875 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb378025460 con 0x7fb3900fab90 2023-11-15T06:25:33.544 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.542+0000 7fb396590700 1 -- 172.21.15.62:0/3967142875 --> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fb390104a40 con 0x7fb37002fd50 2023-11-15T06:25:33.548 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.546+0000 7fb38cff9700 1 -- 172.21.15.62:0/3967142875 <== mgr.14116 v2:172.21.15.62:6800/3280254567 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fb390104a40 con 0x7fb37002fd50 2023-11-15T06:25:33.549 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: Scheduled mgr update... 2023-11-15T06:25:33.551 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.547+0000 7fb396590700 1 -- 172.21.15.62:0/3967142875 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7fb37002fd50 msgr2=0x7fb370032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:33.552 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.547+0000 7fb396590700 1 --2- 172.21.15.62:0/3967142875 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7fb37002fd50 0x7fb370032210 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fb3800096a0 tx=0x7fb380006b40).stop 2023-11-15T06:25:33.553 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.547+0000 7fb396590700 1 -- 172.21.15.62:0/3967142875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3900fab90 msgr2=0x7fb39010a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:33.553 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.547+0000 7fb396590700 1 --2- 172.21.15.62:0/3967142875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3900fab90 0x7fb39010a9f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fb37800b1d0 tx=0x7fb378006ee0).stop 2023-11-15T06:25:33.553 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.547+0000 7fb396590700 1 -- 172.21.15.62:0/3967142875 shutdown_connections 2023-11-15T06:25:33.554 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.547+0000 7fb396590700 1 --2- 172.21.15.62:0/3967142875 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7fb37002fd50 0x7fb370032210 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:33.554 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.547+0000 7fb396590700 1 --2- 172.21.15.62:0/3967142875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3900fab90 0x7fb39010a9f0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:33.554 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.547+0000 7fb396590700 1 -- 172.21.15.62:0/3967142875 >> 172.21.15.62:0/3967142875 conn(0x7fb3900f63b0 msgr2=0x7fb3900f7e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:33.555 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.547+0000 7fb396590700 1 -- 172.21.15.62:0/3967142875 shutdown_connections 2023-11-15T06:25:33.555 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:33.547+0000 7fb396590700 1 -- 172.21.15.62:0/3967142875 wait complete. 2023-11-15T06:25:33.924 INFO:teuthology.orchestra.run.smithi062.stderr:Deploying crash service with default placement... 2023-11-15T06:25:33.925 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi062 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptvq9ksio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpf12kfgcg:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2023-11-15T06:25:34.305 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.302+0000 7f23e2845700 1 Processor -- start 2023-11-15T06:25:34.305 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.302+0000 7f23e2845700 1 -- start start 2023-11-15T06:25:34.305 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.302+0000 7f23e2845700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23dc0703d0 0x7f23dc100b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:34.306 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.302+0000 7f23e2845700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23dc0fc580 con 0x7f23dc0703d0 2023-11-15T06:25:34.306 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.302+0000 7f23e1843700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23dc0703d0 0x7f23dc100b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:34.306 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.302+0000 7f23e1843700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23dc0703d0 0x7f23dc100b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42822/0 (socket says 172.21.15.62:42822) 2023-11-15T06:25:34.306 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.302+0000 7f23e1843700 1 -- 172.21.15.62:0/3898967374 learned_addr learned my addr 172.21.15.62:0/3898967374 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:34.309 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.302+0000 7f23e1843700 1 -- 172.21.15.62:0/3898967374 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23dc101080 con 0x7f23dc0703d0 2023-11-15T06:25:34.310 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.303+0000 7f23e1843700 1 --2- 172.21.15.62:0/3898967374 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23dc0703d0 0x7f23dc100b90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f23d800b5f0 tx=0x7f23d8006ee0).ready entity=mon.0 client_cookie=6f9b825856a8b1cf server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:34.310 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.303+0000 7f23e0841700 1 -- 172.21.15.62:0/3898967374 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23d800d600 con 0x7f23dc0703d0 2023-11-15T06:25:34.310 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.303+0000 7f23e0841700 1 -- 172.21.15.62:0/3898967374 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f23d800dbd0 con 0x7f23dc0703d0 2023-11-15T06:25:34.310 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.303+0000 7f23e0841700 1 -- 172.21.15.62:0/3898967374 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23d8011c00 con 0x7f23dc0703d0 2023-11-15T06:25:34.310 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.304+0000 7f23e2845700 1 -- 172.21.15.62:0/3898967374 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23dc0703d0 msgr2=0x7f23dc100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:34.311 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.304+0000 7f23e2845700 1 --2- 172.21.15.62:0/3898967374 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23dc0703d0 0x7f23dc100b90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f23d800b5f0 tx=0x7f23d8006ee0).stop 2023-11-15T06:25:34.311 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.304+0000 7f23e2845700 1 -- 172.21.15.62:0/3898967374 shutdown_connections 2023-11-15T06:25:34.311 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.304+0000 7f23e2845700 1 --2- 172.21.15.62:0/3898967374 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23dc0703d0 0x7f23dc100b90 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:34.311 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.304+0000 7f23e2845700 1 -- 172.21.15.62:0/3898967374 >> 172.21.15.62:0/3898967374 conn(0x7f23dc0f6210 msgr2=0x7f23dc0f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:34.311 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.304+0000 7f23e2845700 1 -- 172.21.15.62:0/3898967374 shutdown_connections 2023-11-15T06:25:34.311 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.304+0000 7f23e2845700 1 -- 172.21.15.62:0/3898967374 wait complete. 2023-11-15T06:25:34.312 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.304+0000 7f23e2845700 1 Processor -- start 2023-11-15T06:25:34.312 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.304+0000 7f23e2845700 1 -- start start 2023-11-15T06:25:34.312 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.304+0000 7f23e2845700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23dc0703d0 0x7f23dc10ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:34.312 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.304+0000 7f23e2845700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23dc10cf60 con 0x7f23dc0703d0 2023-11-15T06:25:34.312 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.305+0000 7f23e1843700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23dc0703d0 0x7f23dc10ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:34.313 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.305+0000 7f23e1843700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23dc0703d0 0x7f23dc10ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42838/0 (socket says 172.21.15.62:42838) 2023-11-15T06:25:34.313 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.305+0000 7f23e1843700 1 -- 172.21.15.62:0/3323304118 learned_addr learned my addr 172.21.15.62:0/3323304118 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:34.313 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.305+0000 7f23e1843700 1 -- 172.21.15.62:0/3323304118 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d800b200 con 0x7f23dc0703d0 2023-11-15T06:25:34.313 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.305+0000 7f23e1843700 1 --2- 172.21.15.62:0/3323304118 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23dc0703d0 0x7f23dc10ca70 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f23d800b190 tx=0x7f23d8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:34.313 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.305+0000 7f23ca7fc700 1 -- 172.21.15.62:0/3323304118 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23d801b410 con 0x7f23dc0703d0 2023-11-15T06:25:34.314 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.305+0000 7f23ca7fc700 1 -- 172.21.15.62:0/3323304118 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f23d801b9e0 con 0x7f23dc0703d0 2023-11-15T06:25:34.314 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.305+0000 7f23e2845700 1 -- 172.21.15.62:0/3323304118 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23dc10d160 con 0x7f23dc0703d0 2023-11-15T06:25:34.314 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.305+0000 7f23ca7fc700 1 -- 172.21.15.62:0/3323304118 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23d801a880 con 0x7f23dc0703d0 2023-11-15T06:25:34.314 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.305+0000 7f23e2845700 1 -- 172.21.15.62:0/3323304118 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23dc10d570 con 0x7f23dc0703d0 2023-11-15T06:25:34.314 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.306+0000 7f23ca7fc700 1 -- 172.21.15.62:0/3323304118 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f23d800f070 con 0x7f23dc0703d0 2023-11-15T06:25:34.314 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.306+0000 7f23ca7fc700 1 --2- 172.21.15.62:0/3323304118 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f23c402fd50 0x7f23c4032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:34.315 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.306+0000 7f23ca7fc700 1 -- 172.21.15.62:0/3323304118 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f23d8041340 con 0x7f23dc0703d0 2023-11-15T06:25:34.315 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.306+0000 7f23e2845700 1 -- 172.21.15.62:0/3323304118 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23dc045ad0 con 0x7f23dc0703d0 2023-11-15T06:25:34.315 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.306+0000 7f23e1042700 1 --2- 172.21.15.62:0/3323304118 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f23c402fd50 0x7f23c4032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:34.315 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.307+0000 7f23e1042700 1 --2- 172.21.15.62:0/3323304118 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f23c402fd50 0x7f23c4032210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f23cc0096a0 tx=0x7f23cc006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:34.316 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.311+0000 7f23ca7fc700 1 -- 172.21.15.62:0/3323304118 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f23d8010360 con 0x7f23dc0703d0 2023-11-15T06:25:34.500 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.498+0000 7f23e2845700 1 -- 172.21.15.62:0/3323304118 --> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f23dc106950 con 0x7f23c402fd50 2023-11-15T06:25:34.504 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.501+0000 7f23ca7fc700 1 -- 172.21.15.62:0/3323304118 <== mgr.14116 v2:172.21.15.62:6800/3280254567 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f23dc106950 con 0x7f23c402fd50 2023-11-15T06:25:34.504 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: Scheduled crash update... 2023-11-15T06:25:34.506 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.503+0000 7f23e2845700 1 -- 172.21.15.62:0/3323304118 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f23c402fd50 msgr2=0x7f23c4032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:34.506 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.503+0000 7f23e2845700 1 --2- 172.21.15.62:0/3323304118 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f23c402fd50 0x7f23c4032210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f23cc0096a0 tx=0x7f23cc006b40).stop 2023-11-15T06:25:34.506 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.503+0000 7f23e2845700 1 -- 172.21.15.62:0/3323304118 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23dc0703d0 msgr2=0x7f23dc10ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:34.507 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.503+0000 7f23e2845700 1 --2- 172.21.15.62:0/3323304118 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23dc0703d0 0x7f23dc10ca70 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f23d800b190 tx=0x7f23d8006ee0).stop 2023-11-15T06:25:34.507 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.503+0000 7f23e2845700 1 -- 172.21.15.62:0/3323304118 shutdown_connections 2023-11-15T06:25:34.507 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.503+0000 7f23e2845700 1 --2- 172.21.15.62:0/3323304118 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f23c402fd50 0x7f23c4032210 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:34.507 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.503+0000 7f23e2845700 1 --2- 172.21.15.62:0/3323304118 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23dc0703d0 0x7f23dc10ca70 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:34.507 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.503+0000 7f23e2845700 1 -- 172.21.15.62:0/3323304118 >> 172.21.15.62:0/3323304118 conn(0x7f23dc0f6210 msgr2=0x7f23dc0f7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:34.508 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.503+0000 7f23e2845700 1 -- 172.21.15.62:0/3323304118 shutdown_connections 2023-11-15T06:25:34.508 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:34.503+0000 7f23e2845700 1 -- 172.21.15.62:0/3323304118 wait complete. 2023-11-15T06:25:34.793 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:34 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:32.709746+0000 mgr.smithi062.rwurtk (mgr.14116) 10 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T06:25:34.793 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:34 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cephadm 2023-11-15T06:25:32.710471+0000 mgr.smithi062.rwurtk (mgr.14116) 11 : cephadm [INF] Saving service mon spec with placement count:5 2023-11-15T06:25:34.793 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:34 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:33.544414+0000 mgr.smithi062.rwurtk (mgr.14116) 12 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T06:25:34.793 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:34 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cephadm 2023-11-15T06:25:33.544905+0000 mgr.smithi062.rwurtk (mgr.14116) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2023-11-15T06:25:34.794 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:34 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:33.547116+0000 mon.smithi062 (mon.0) 58 : audit [INF] from='mgr.14116 172.21.15.62:0/1148526200' entity='mgr.smithi062.rwurtk' 2023-11-15T06:25:34.794 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:34 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:34.502819+0000 mon.smithi062 (mon.0) 59 : audit [INF] from='mgr.14116 172.21.15.62:0/1148526200' entity='mgr.smithi062.rwurtk' 2023-11-15T06:25:34.841 INFO:teuthology.orchestra.run.smithi062.stderr:Enabling mgr prometheus module... 2023-11-15T06:25:34.841 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi062 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptvq9ksio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpf12kfgcg:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2023-11-15T06:25:35.202 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.193+0000 7f87978b7700 1 Processor -- start 2023-11-15T06:25:35.203 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.193+0000 7f87978b7700 1 -- start start 2023-11-15T06:25:35.203 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.193+0000 7f87978b7700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87900704a0 0x7f87900fe8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:35.204 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.193+0000 7f87978b7700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87900fede0 con 0x7f87900704a0 2023-11-15T06:25:35.204 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.193+0000 7f8795653700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87900704a0 0x7f87900fe8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:35.204 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.193+0000 7f8795653700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87900704a0 0x7f87900fe8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42846/0 (socket says 172.21.15.62:42846) 2023-11-15T06:25:35.205 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.193+0000 7f8795653700 1 -- 172.21.15.62:0/3267987138 learned_addr learned my addr 172.21.15.62:0/3267987138 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:35.205 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.194+0000 7f8795653700 1 -- 172.21.15.62:0/3267987138 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87900fef20 con 0x7f87900704a0 2023-11-15T06:25:35.205 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.194+0000 7f8795653700 1 --2- 172.21.15.62:0/3267987138 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87900704a0 0x7f87900fe8f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f878c00b5f0 tx=0x7f878c006ee0).ready entity=mon.0 client_cookie=35cdd7bc62e68c7a server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:35.206 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.194+0000 7f877ffff700 1 -- 172.21.15.62:0/3267987138 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f878c00d600 con 0x7f87900704a0 2023-11-15T06:25:35.206 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.194+0000 7f877ffff700 1 -- 172.21.15.62:0/3267987138 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f878c00dbd0 con 0x7f87900704a0 2023-11-15T06:25:35.206 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.194+0000 7f877ffff700 1 -- 172.21.15.62:0/3267987138 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f878c011c00 con 0x7f87900704a0 2023-11-15T06:25:35.207 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.195+0000 7f87978b7700 1 -- 172.21.15.62:0/3267987138 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87900704a0 msgr2=0x7f87900fe8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:35.207 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.195+0000 7f87978b7700 1 --2- 172.21.15.62:0/3267987138 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87900704a0 0x7f87900fe8f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f878c00b5f0 tx=0x7f878c006ee0).stop 2023-11-15T06:25:35.208 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.195+0000 7f87978b7700 1 -- 172.21.15.62:0/3267987138 shutdown_connections 2023-11-15T06:25:35.208 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.195+0000 7f87978b7700 1 --2- 172.21.15.62:0/3267987138 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87900704a0 0x7f87900fe8f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:35.208 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.195+0000 7f87978b7700 1 -- 172.21.15.62:0/3267987138 >> 172.21.15.62:0/3267987138 conn(0x7f87900f63b0 msgr2=0x7f87900f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:35.209 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.195+0000 7f87978b7700 1 -- 172.21.15.62:0/3267987138 shutdown_connections 2023-11-15T06:25:35.209 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.195+0000 7f87978b7700 1 -- 172.21.15.62:0/3267987138 wait complete. 2023-11-15T06:25:35.209 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.195+0000 7f87978b7700 1 Processor -- start 2023-11-15T06:25:35.210 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.195+0000 7f87978b7700 1 -- start start 2023-11-15T06:25:35.210 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.195+0000 7f87978b7700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87900704a0 0x7f879010aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:35.211 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.195+0000 7f87978b7700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f879010af50 con 0x7f87900704a0 2023-11-15T06:25:35.211 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.196+0000 7f8795653700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87900704a0 0x7f879010aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:35.211 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.196+0000 7f8795653700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87900704a0 0x7f879010aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42860/0 (socket says 172.21.15.62:42860) 2023-11-15T06:25:35.212 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.196+0000 7f8795653700 1 -- 172.21.15.62:0/2299737133 learned_addr learned my addr 172.21.15.62:0/2299737133 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:35.212 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.196+0000 7f8795653700 1 -- 172.21.15.62:0/2299737133 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f878c00b200 con 0x7f87900704a0 2023-11-15T06:25:35.212 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.196+0000 7f8795653700 1 --2- 172.21.15.62:0/2299737133 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87900704a0 0x7f879010aa60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f878c00e040 tx=0x7f878c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:35.213 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.196+0000 7f877dffb700 1 -- 172.21.15.62:0/2299737133 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f878c00d9f0 con 0x7f87900704a0 2023-11-15T06:25:35.213 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.196+0000 7f877dffb700 1 -- 172.21.15.62:0/2299737133 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f878c01a9f0 con 0x7f87900704a0 2023-11-15T06:25:35.213 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.196+0000 7f87978b7700 1 -- 172.21.15.62:0/2299737133 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f879010b150 con 0x7f87900704a0 2023-11-15T06:25:35.214 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.196+0000 7f877dffb700 1 -- 172.21.15.62:0/2299737133 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f878c01b8c0 con 0x7f87900704a0 2023-11-15T06:25:35.214 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.196+0000 7f87978b7700 1 -- 172.21.15.62:0/2299737133 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f879010b560 con 0x7f87900704a0 2023-11-15T06:25:35.215 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.197+0000 7f877dffb700 1 -- 172.21.15.62:0/2299737133 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f878c00f070 con 0x7f87900704a0 2023-11-15T06:25:35.215 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.197+0000 7f87978b7700 1 -- 172.21.15.62:0/2299737133 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8790045ad0 con 0x7f87900704a0 2023-11-15T06:25:35.215 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.197+0000 7f877dffb700 1 --2- 172.21.15.62:0/2299737133 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f877802fd50 0x7f8778032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:35.216 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.197+0000 7f877dffb700 1 -- 172.21.15.62:0/2299737133 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f878c0412a0 con 0x7f87900704a0 2023-11-15T06:25:35.216 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.197+0000 7f8794e52700 1 --2- 172.21.15.62:0/2299737133 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f877802fd50 0x7f8778032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:35.217 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.198+0000 7f8794e52700 1 --2- 172.21.15.62:0/2299737133 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f877802fd50 0x7f8778032210 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f87800096a0 tx=0x7f8780006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:35.218 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.202+0000 7f877dffb700 1 -- 172.21.15.62:0/2299737133 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f878c022d70 con 0x7f87900704a0 2023-11-15T06:25:35.401 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.398+0000 7f87978b7700 1 -- 172.21.15.62:0/2299737133 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7f87900ffba0 con 0x7f87900704a0 2023-11-15T06:25:35.551 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.549+0000 7f877dffb700 1 -- 172.21.15.62:0/2299737133 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f878c0257e0 con 0x7f87900704a0 2023-11-15T06:25:35.552 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.550+0000 7f877dffb700 1 -- 172.21.15.62:0/2299737133 <== mon.0 v2:172.21.15.62:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v9) v1 ==== 89+0+0 (secure 0 0 0) 0x7f878c010c40 con 0x7f87900704a0 2023-11-15T06:25:35.555 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.552+0000 7f87978b7700 1 -- 172.21.15.62:0/2299737133 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f877802fd50 msgr2=0x7f8778032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:35.555 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.552+0000 7f87978b7700 1 --2- 172.21.15.62:0/2299737133 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f877802fd50 0x7f8778032210 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f87800096a0 tx=0x7f8780006b40).stop 2023-11-15T06:25:35.555 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.552+0000 7f87978b7700 1 -- 172.21.15.62:0/2299737133 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87900704a0 msgr2=0x7f879010aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:35.556 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.552+0000 7f87978b7700 1 --2- 172.21.15.62:0/2299737133 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87900704a0 0x7f879010aa60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f878c00e040 tx=0x7f878c006ee0).stop 2023-11-15T06:25:35.556 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.552+0000 7f87978b7700 1 -- 172.21.15.62:0/2299737133 shutdown_connections 2023-11-15T06:25:35.556 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.552+0000 7f87978b7700 1 --2- 172.21.15.62:0/2299737133 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f877802fd50 0x7f8778032210 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:35.556 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.552+0000 7f87978b7700 1 --2- 172.21.15.62:0/2299737133 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87900704a0 0x7f879010aa60 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:35.556 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.552+0000 7f87978b7700 1 -- 172.21.15.62:0/2299737133 >> 172.21.15.62:0/2299737133 conn(0x7f87900f63b0 msgr2=0x7f87900f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:35.556 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.552+0000 7f87978b7700 1 -- 172.21.15.62:0/2299737133 shutdown_connections 2023-11-15T06:25:35.557 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:35.552+0000 7f87978b7700 1 -- 172.21.15.62:0/2299737133 wait complete. 2023-11-15T06:25:35.843 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:35 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:34.500055+0000 mgr.smithi062.rwurtk (mgr.14116) 14 : audit [DBG] from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T06:25:35.843 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:35 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cephadm 2023-11-15T06:25:34.500681+0000 mgr.smithi062.rwurtk (mgr.14116) 15 : cephadm [INF] Saving service crash spec with placement * 2023-11-15T06:25:35.843 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:35 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:35.400200+0000 mon.smithi062 (mon.0) 60 : audit [INF] from='client.? 172.21.15.62:0/2299737133' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2023-11-15T06:25:35.915 INFO:teuthology.orchestra.run.smithi062.stderr:Deploying prometheus service with default placement... 2023-11-15T06:25:35.915 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi062 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptvq9ksio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpf12kfgcg:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2023-11-15T06:25:36.289 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:36.284+0000 7f392ae94700 1 Processor -- start 2023-11-15T06:25:36.289 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:36.285+0000 7f392ae94700 1 -- start start 2023-11-15T06:25:36.289 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:36.285+0000 7f392ae94700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39240fa9d0 0x7f39240fadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:36.289 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:36.285+0000 7f392ae94700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39240fb2e0 con 0x7f39240fa9d0 2023-11-15T06:25:36.290 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:36.285+0000 7f3929e92700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39240fa9d0 0x7f39240fadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:36.290 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:36.285+0000 7f3929e92700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39240fa9d0 0x7f39240fadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42890/0 (socket says 172.21.15.62:42890) 2023-11-15T06:25:36.290 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:36.285+0000 7f3929e92700 1 -- 172.21.15.62:0/102624308 learned_addr learned my addr 172.21.15.62:0/102624308 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:36.290 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:36.285+0000 7f3929e92700 1 -- 172.21.15.62:0/102624308 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39240fb420 con 0x7f39240fa9d0 2023-11-15T06:25:36.291 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:36.286+0000 7f3929e92700 1 --2- 172.21.15.62:0/102624308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39240fa9d0 0x7f39240fadf0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f392000b870 tx=0x7f3920006f70).ready entity=mon.0 client_cookie=6c72441e51e503cd server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:36.291 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:36.286+0000 7f3928e90700 1 -- 172.21.15.62:0/102624308 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f392000da20 con 0x7f39240fa9d0 2023-11-15T06:25:36.291 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:36.286+0000 7f3928e90700 1 -- 172.21.15.62:0/102624308 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f3920010470 con 0x7f39240fa9d0 2023-11-15T06:25:36.291 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:36.286+0000 7f3928e90700 1 -- 172.21.15.62:0/102624308 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f392000f5b0 con 0x7f39240fa9d0 2023-11-15T06:25:36.291 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:36.286+0000 7f392ae94700 1 -- 172.21.15.62:0/102624308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39240fa9d0 msgr2=0x7f39240fadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:36.292 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:36.286+0000 7f392ae94700 1 --2- 172.21.15.62:0/102624308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39240fa9d0 0x7f39240fadf0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f392000b870 tx=0x7f3920006f70).stop 2023-11-15T06:25:36.292 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:36.286+0000 7f392ae94700 1 -- 172.21.15.62:0/102624308 shutdown_connections 2023-11-15T06:25:36.292 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:36.286+0000 7f392ae94700 1 --2- 172.21.15.62:0/102624308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39240fa9d0 0x7f39240fadf0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:36.292 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:36.286+0000 7f392ae94700 1 -- 172.21.15.62:0/102624308 >> 172.21.15.62:0/102624308 conn(0x7f39240f6210 msgr2=0x7f39240f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:36.295 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:36.287+0000 7f392ae94700 1 -- 172.21.15.62:0/102624308 shutdown_connections 2023-11-15T06:25:36.295 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:36.287+0000 7f392ae94700 1 -- 172.21.15.62:0/102624308 wait complete. 2023-11-15T06:25:36.295 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:36.288+0000 7f392ae94700 1 Processor -- start 2023-11-15T06:25:36.296 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:36.288+0000 7f392ae94700 1 -- start start 2023-11-15T06:25:36.296 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:36.288+0000 7f392ae94700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39240fa9d0 0x7f3924108570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:36.296 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:36.288+0000 7f392ae94700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3924108a60 con 0x7f39240fa9d0 2023-11-15T06:25:36.297 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:36.288+0000 7f3929e92700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39240fa9d0 0x7f3924108570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:36.297 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:36.288+0000 7f3929e92700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39240fa9d0 0x7f3924108570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42898/0 (socket says 172.21.15.62:42898) 2023-11-15T06:25:36.297 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:36.288+0000 7f3929e92700 1 -- 172.21.15.62:0/3817689909 learned_addr learned my addr 172.21.15.62:0/3817689909 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:36.297 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:36.288+0000 7f3929e92700 1 -- 172.21.15.62:0/3817689909 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f392000b480 con 0x7f39240fa9d0 2023-11-15T06:25:36.297 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:36.288+0000 7f3929e92700 1 --2- 172.21.15.62:0/3817689909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39240fa9d0 0x7f3924108570 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f3920006a40 tx=0x7f3920006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:36.298 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:36.289+0000 7f39127fc700 1 -- 172.21.15.62:0/3817689909 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f392000f3d0 con 0x7f39240fa9d0 2023-11-15T06:25:36.298 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:36.289+0000 7f39127fc700 1 -- 172.21.15.62:0/3817689909 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f392000fef0 con 0x7f39240fa9d0 2023-11-15T06:25:36.298 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:36.289+0000 7f392ae94700 1 -- 172.21.15.62:0/3817689909 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3924108c60 con 0x7f39240fa9d0 2023-11-15T06:25:36.298 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:36.289+0000 7f39127fc700 1 -- 172.21.15.62:0/3817689909 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3920019d40 con 0x7f39240fa9d0 2023-11-15T06:25:36.298 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:36.289+0000 7f392ae94700 1 -- 172.21.15.62:0/3817689909 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3924109070 con 0x7f39240fa9d0 2023-11-15T06:25:36.299 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:36.289+0000 7f39127fc700 1 -- 172.21.15.62:0/3817689909 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f392002b060 con 0x7f39240fa9d0 2023-11-15T06:25:36.299 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:36.290+0000 7f39127fc700 1 --2- 172.21.15.62:0/3817689909 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f390c02fe00 0x7f390c0322c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:36.299 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:36.290+0000 7f39127fc700 1 -- 172.21.15.62:0/3817689909 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f3920044f30 con 0x7f39240fa9d0 2023-11-15T06:25:36.299 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:36.290+0000 7f3929691700 1 -- 172.21.15.62:0/3817689909 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f390c02fe00 msgr2=0x7f390c0322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/3280254567 2023-11-15T06:25:36.299 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:36.290+0000 7f3929691700 1 --2- 172.21.15.62:0/3817689909 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f390c02fe00 0x7f390c0322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-15T06:25:36.300 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:36.291+0000 7f392ae94700 1 -- 172.21.15.62:0/3817689909 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3924059100 con 0x7f39240fa9d0 2023-11-15T06:25:36.300 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:36.295+0000 7f39127fc700 1 -- 172.21.15.62:0/3817689909 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f392002c340 con 0x7f39240fa9d0 2023-11-15T06:25:36.487 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:36.485+0000 7f392ae94700 1 -- 172.21.15.62:0/3817689909 --> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f39241024f0 con 0x7f390c02fe00 2023-11-15T06:25:36.494 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:36.491+0000 7f3929691700 1 -- 172.21.15.62:0/3817689909 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f390c02fe00 msgr2=0x7f390c0322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/3280254567 2023-11-15T06:25:36.494 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:36.491+0000 7f3929691700 1 --2- 172.21.15.62:0/3817689909 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f390c02fe00 0x7f390c0322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-15T06:25:36.894 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:36.892+0000 7f3929691700 1 -- 172.21.15.62:0/3817689909 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f390c02fe00 msgr2=0x7f390c0322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/3280254567 2023-11-15T06:25:36.895 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:36.892+0000 7f3929691700 1 --2- 172.21.15.62:0/3817689909 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f390c02fe00 0x7f390c0322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-11-15T06:25:36.917 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:36 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:35.551744+0000 mon.smithi062 (mon.0) 61 : audit [INF] from='client.? 172.21.15.62:0/2299737133' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2023-11-15T06:25:36.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:36 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:25:35.551792+0000 mon.smithi062 (mon.0) 62 : cluster [DBG] mgrmap e9: smithi062.rwurtk(active, since 15s) 2023-11-15T06:25:37.696 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:37.693+0000 7f3929691700 1 -- 172.21.15.62:0/3817689909 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f390c02fe00 msgr2=0x7f390c0322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/3280254567 2023-11-15T06:25:37.696 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:37.693+0000 7f3929691700 1 --2- 172.21.15.62:0/3817689909 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f390c02fe00 0x7f390c0322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-11-15T06:25:39.297 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:39.294+0000 7f3929691700 1 -- 172.21.15.62:0/3817689909 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f390c02fe00 msgr2=0x7f390c0322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/3280254567 2023-11-15T06:25:39.297 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:39.294+0000 7f3929691700 1 --2- 172.21.15.62:0/3817689909 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f390c02fe00 0x7f390c0322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-11-15T06:25:40.767 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:40.763+0000 7f39127fc700 1 -- 172.21.15.62:0/3817689909 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mgrmap(e 10) v1 ==== 35984+0+0 (secure 0 0 0) 0x7f3920016030 con 0x7f39240fa9d0 2023-11-15T06:25:40.767 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:40.764+0000 7f39127fc700 1 -- 172.21.15.62:0/3817689909 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f390c02fe00 msgr2=0x7f390c0322c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-15T06:25:40.767 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:40.764+0000 7f39127fc700 1 --2- 172.21.15.62:0/3817689909 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f390c02fe00 0x7f390c0322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T06:25:41.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:40 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:25:40.713287+0000 mon.smithi062 (mon.0) 63 : cluster [INF] Active manager daemon smithi062.rwurtk restarted 2023-11-15T06:25:41.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:40 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:25:40.713687+0000 mon.smithi062 (mon.0) 64 : cluster [INF] Activating manager daemon smithi062.rwurtk 2023-11-15T06:25:41.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:40 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:25:40.715267+0000 mon.smithi062 (mon.0) 65 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2023-11-15T06:25:41.775 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:41.771+0000 7f39127fc700 1 -- 172.21.15.62:0/3817689909 <== mon.0 v2:172.21.15.62:3300/0 8 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f39200194e0 con 0x7f39240fa9d0 2023-11-15T06:25:41.775 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:41.771+0000 7f39127fc700 1 --2- 172.21.15.62:0/3817689909 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7f390c033c20 0x7f390c036010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:41.775 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:41.771+0000 7f39127fc700 1 -- 172.21.15.62:0/3817689909 --> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f39241024f0 con 0x7f390c033c20 2023-11-15T06:25:41.775 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:41.771+0000 7f3929691700 1 --2- 172.21.15.62:0/3817689909 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7f390c033c20 0x7f390c036010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:41.775 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:41.772+0000 7f3929691700 1 --2- 172.21.15.62:0/3817689909 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7f390c033c20 0x7f390c036010 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f3914006fd0 tx=0x7f3914006d30).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:41.786 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:41.784+0000 7f39127fc700 1 -- 172.21.15.62:0/3817689909 <== mgr.14144 v2:172.21.15.62:6800/1948381890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f39241024f0 con 0x7f390c033c20 2023-11-15T06:25:41.787 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: Scheduled prometheus update... 2023-11-15T06:25:41.790 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:41.786+0000 7f392ae94700 1 -- 172.21.15.62:0/3817689909 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7f390c033c20 msgr2=0x7f390c036010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:41.790 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:41.786+0000 7f392ae94700 1 --2- 172.21.15.62:0/3817689909 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7f390c033c20 0x7f390c036010 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f3914006fd0 tx=0x7f3914006d30).stop 2023-11-15T06:25:41.790 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:41.786+0000 7f392ae94700 1 -- 172.21.15.62:0/3817689909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39240fa9d0 msgr2=0x7f3924108570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:41.790 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:41.786+0000 7f392ae94700 1 --2- 172.21.15.62:0/3817689909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39240fa9d0 0x7f3924108570 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f3920006a40 tx=0x7f3920006f70).stop 2023-11-15T06:25:41.791 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:41.786+0000 7f392ae94700 1 -- 172.21.15.62:0/3817689909 shutdown_connections 2023-11-15T06:25:41.791 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:41.786+0000 7f392ae94700 1 --2- 172.21.15.62:0/3817689909 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7f390c033c20 0x7f390c036010 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:41.791 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:41.786+0000 7f392ae94700 1 --2- 172.21.15.62:0/3817689909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39240fa9d0 0x7f3924108570 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:41.791 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:41.786+0000 7f392ae94700 1 --2- 172.21.15.62:0/3817689909 >> [v2:172.21.15.62:6800/3280254567,v1:172.21.15.62:6801/3280254567] conn(0x7f390c02fe00 0x7f390c0322c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T06:25:41.791 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:41.786+0000 7f392ae94700 1 -- 172.21.15.62:0/3817689909 >> 172.21.15.62:0/3817689909 conn(0x7f39240f6210 msgr2=0x7f39240ff760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:41.792 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:41.786+0000 7f392ae94700 1 -- 172.21.15.62:0/3817689909 shutdown_connections 2023-11-15T06:25:41.792 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:41.786+0000 7f392ae94700 1 -- 172.21.15.62:0/3817689909 wait complete. 2023-11-15T06:25:42.131 INFO:teuthology.orchestra.run.smithi062.stderr:Deploying grafana service with default placement... 2023-11-15T06:25:42.132 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi062 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptvq9ksio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpf12kfgcg:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2023-11-15T06:25:42.134 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:41 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:25:40.766094+0000 mon.smithi062 (mon.0) 66 : cluster [DBG] mgrmap e10: smithi062.rwurtk(active, starting, since 0.0524911s) 2023-11-15T06:25:42.134 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:41 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:40.768343+0000 mon.smithi062 (mon.0) 67 : audit [DBG] from='mgr.14144 172.21.15.62:0/1570055551' entity='mgr.smithi062.rwurtk' cmd=[{"prefix": "mon metadata", "id": "smithi062"}]: dispatch 2023-11-15T06:25:42.135 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:41 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:40.769062+0000 mon.smithi062 (mon.0) 68 : audit [DBG] from='mgr.14144 172.21.15.62:0/1570055551' entity='mgr.smithi062.rwurtk' cmd=[{"prefix": "mgr metadata", "who": "smithi062.rwurtk", "id": "smithi062.rwurtk"}]: dispatch 2023-11-15T06:25:42.135 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:41 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:40.769778+0000 mon.smithi062 (mon.0) 69 : audit [DBG] from='mgr.14144 172.21.15.62:0/1570055551' entity='mgr.smithi062.rwurtk' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-15T06:25:42.135 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:41 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:40.769914+0000 mon.smithi062 (mon.0) 70 : audit [DBG] from='mgr.14144 172.21.15.62:0/1570055551' entity='mgr.smithi062.rwurtk' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-15T06:25:42.135 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:41 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:40.770031+0000 mon.smithi062 (mon.0) 71 : audit [DBG] from='mgr.14144 172.21.15.62:0/1570055551' entity='mgr.smithi062.rwurtk' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-15T06:25:42.135 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:41 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:25:40.783878+0000 mon.smithi062 (mon.0) 72 : cluster [INF] Manager daemon smithi062.rwurtk is now available 2023-11-15T06:25:42.136 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:41 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:40.886940+0000 mon.smithi062 (mon.0) 73 : audit [DBG] from='mgr.14144 172.21.15.62:0/1570055551' entity='mgr.smithi062.rwurtk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T06:25:42.136 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:41 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:40.888323+0000 mon.smithi062 (mon.0) 74 : audit [DBG] from='mgr.14144 172.21.15.62:0/1570055551' entity='mgr.smithi062.rwurtk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T06:25:42.136 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:41 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:40.904010+0000 mon.smithi062 (mon.0) 75 : audit [INF] from='mgr.14144 172.21.15.62:0/1570055551' entity='mgr.smithi062.rwurtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi062.rwurtk/mirror_snapshot_schedule"}]: dispatch 2023-11-15T06:25:42.136 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:41 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:40.913002+0000 mon.smithi062 (mon.0) 76 : audit [INF] from='mgr.14144 172.21.15.62:0/1570055551' entity='mgr.smithi062.rwurtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi062.rwurtk/trash_purge_schedule"}]: dispatch 2023-11-15T06:25:42.501 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.497+0000 7f5c3edbc700 1 Processor -- start 2023-11-15T06:25:42.501 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.497+0000 7f5c3edbc700 1 -- start start 2023-11-15T06:25:42.501 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.497+0000 7f5c3edbc700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c380703f0 0x7f5c380fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:42.502 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.497+0000 7f5c3edbc700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c380fee10 con 0x7f5c380703f0 2023-11-15T06:25:42.502 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.498+0000 7f5c3cb58700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c380703f0 0x7f5c380fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:42.502 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.498+0000 7f5c3cb58700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c380703f0 0x7f5c380fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57340/0 (socket says 172.21.15.62:57340) 2023-11-15T06:25:42.502 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.498+0000 7f5c3cb58700 1 -- 172.21.15.62:0/3939428216 learned_addr learned my addr 172.21.15.62:0/3939428216 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:42.502 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.498+0000 7f5c3cb58700 1 -- 172.21.15.62:0/3939428216 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c380fef50 con 0x7f5c380703f0 2023-11-15T06:25:42.503 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.498+0000 7f5c3cb58700 1 --2- 172.21.15.62:0/3939428216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c380703f0 0x7f5c380fe920 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f5c2c00b5f0 tx=0x7f5c2c006ee0).ready entity=mon.0 client_cookie=ec59ed51d6735447 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:42.503 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.498+0000 7f5c377fe700 1 -- 172.21.15.62:0/3939428216 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c2c00d600 con 0x7f5c380703f0 2023-11-15T06:25:42.503 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.498+0000 7f5c377fe700 1 -- 172.21.15.62:0/3939428216 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f5c2c00dbd0 con 0x7f5c380703f0 2023-11-15T06:25:42.503 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.498+0000 7f5c377fe700 1 -- 172.21.15.62:0/3939428216 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c2c011c00 con 0x7f5c380703f0 2023-11-15T06:25:42.506 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.499+0000 7f5c3edbc700 1 -- 172.21.15.62:0/3939428216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c380703f0 msgr2=0x7f5c380fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:42.506 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.499+0000 7f5c3edbc700 1 --2- 172.21.15.62:0/3939428216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c380703f0 0x7f5c380fe920 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f5c2c00b5f0 tx=0x7f5c2c006ee0).stop 2023-11-15T06:25:42.506 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.499+0000 7f5c3edbc700 1 -- 172.21.15.62:0/3939428216 shutdown_connections 2023-11-15T06:25:42.506 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.499+0000 7f5c3edbc700 1 --2- 172.21.15.62:0/3939428216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c380703f0 0x7f5c380fe920 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:42.506 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.499+0000 7f5c3edbc700 1 -- 172.21.15.62:0/3939428216 >> 172.21.15.62:0/3939428216 conn(0x7f5c380f63c0 msgr2=0x7f5c380f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:42.507 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.499+0000 7f5c3edbc700 1 -- 172.21.15.62:0/3939428216 shutdown_connections 2023-11-15T06:25:42.507 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.499+0000 7f5c3edbc700 1 -- 172.21.15.62:0/3939428216 wait complete. 2023-11-15T06:25:42.507 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.500+0000 7f5c3edbc700 1 Processor -- start 2023-11-15T06:25:42.507 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.500+0000 7f5c3edbc700 1 -- start start 2023-11-15T06:25:42.507 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.500+0000 7f5c3edbc700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c380703f0 0x7f5c380fc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:42.508 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.500+0000 7f5c3edbc700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c380fa820 con 0x7f5c380703f0 2023-11-15T06:25:42.508 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.500+0000 7f5c3cb58700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c380703f0 0x7f5c380fc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:42.508 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.500+0000 7f5c3cb58700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c380703f0 0x7f5c380fc3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57348/0 (socket says 172.21.15.62:57348) 2023-11-15T06:25:42.508 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.500+0000 7f5c3cb58700 1 -- 172.21.15.62:0/317975736 learned_addr learned my addr 172.21.15.62:0/317975736 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:42.508 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.500+0000 7f5c3cb58700 1 -- 172.21.15.62:0/317975736 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c2c00b200 con 0x7f5c380703f0 2023-11-15T06:25:42.509 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.501+0000 7f5c3cb58700 1 --2- 172.21.15.62:0/317975736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c380703f0 0x7f5c380fc3a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f5c2c00b190 tx=0x7f5c2c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:42.509 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.501+0000 7f5c357fa700 1 -- 172.21.15.62:0/317975736 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c2c01b410 con 0x7f5c380703f0 2023-11-15T06:25:42.509 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.501+0000 7f5c3edbc700 1 -- 172.21.15.62:0/317975736 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c380fa9c0 con 0x7f5c380703f0 2023-11-15T06:25:42.509 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.501+0000 7f5c357fa700 1 -- 172.21.15.62:0/317975736 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f5c2c01b9e0 con 0x7f5c380703f0 2023-11-15T06:25:42.509 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.501+0000 7f5c357fa700 1 -- 172.21.15.62:0/317975736 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c2c01a880 con 0x7f5c380703f0 2023-11-15T06:25:42.510 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.501+0000 7f5c3edbc700 1 -- 172.21.15.62:0/317975736 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c380fadd0 con 0x7f5c380703f0 2023-11-15T06:25:42.510 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.502+0000 7f5c357fa700 1 -- 172.21.15.62:0/317975736 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f5c2c00f070 con 0x7f5c380703f0 2023-11-15T06:25:42.510 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.502+0000 7f5c357fa700 1 --2- 172.21.15.62:0/317975736 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7f5c1c02fcf0 0x7f5c1c0321b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:42.510 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.502+0000 7f5c357fa700 1 -- 172.21.15.62:0/317975736 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f5c2c041280 con 0x7f5c380703f0 2023-11-15T06:25:42.510 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.502+0000 7f5c3edbc700 1 -- 172.21.15.62:0/317975736 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c38045ad0 con 0x7f5c380703f0 2023-11-15T06:25:42.510 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.502+0000 7f5c37fff700 1 --2- 172.21.15.62:0/317975736 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7f5c1c02fcf0 0x7f5c1c0321b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:42.511 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.502+0000 7f5c37fff700 1 --2- 172.21.15.62:0/317975736 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7f5c1c02fcf0 0x7f5c1c0321b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f5c240096a0 tx=0x7f5c24006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:42.511 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.506+0000 7f5c357fa700 1 -- 172.21.15.62:0/317975736 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5c2c016070 con 0x7f5c380703f0 2023-11-15T06:25:42.696 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.694+0000 7f5c3edbc700 1 -- 172.21.15.62:0/317975736 --> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f5c380ffd50 con 0x7f5c1c02fcf0 2023-11-15T06:25:42.703 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.701+0000 7f5c357fa700 1 -- 172.21.15.62:0/317975736 <== mgr.14144 v2:172.21.15.62:6800/1948381890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f5c380ffd50 con 0x7f5c1c02fcf0 2023-11-15T06:25:42.704 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: Scheduled grafana update... 2023-11-15T06:25:42.706 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.703+0000 7f5c3edbc700 1 -- 172.21.15.62:0/317975736 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7f5c1c02fcf0 msgr2=0x7f5c1c0321b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:42.706 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.703+0000 7f5c3edbc700 1 --2- 172.21.15.62:0/317975736 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7f5c1c02fcf0 0x7f5c1c0321b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f5c240096a0 tx=0x7f5c24006b40).stop 2023-11-15T06:25:42.706 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.703+0000 7f5c3edbc700 1 -- 172.21.15.62:0/317975736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c380703f0 msgr2=0x7f5c380fc3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:42.707 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.703+0000 7f5c3edbc700 1 --2- 172.21.15.62:0/317975736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c380703f0 0x7f5c380fc3a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f5c2c00b190 tx=0x7f5c2c006ee0).stop 2023-11-15T06:25:42.707 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.703+0000 7f5c3edbc700 1 -- 172.21.15.62:0/317975736 shutdown_connections 2023-11-15T06:25:42.707 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.703+0000 7f5c3edbc700 1 --2- 172.21.15.62:0/317975736 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7f5c1c02fcf0 0x7f5c1c0321b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:42.707 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.703+0000 7f5c3edbc700 1 --2- 172.21.15.62:0/317975736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c380703f0 0x7f5c380fc3a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:42.707 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.703+0000 7f5c3edbc700 1 -- 172.21.15.62:0/317975736 >> 172.21.15.62:0/317975736 conn(0x7f5c380f63c0 msgr2=0x7f5c380f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:42.707 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.703+0000 7f5c3edbc700 1 -- 172.21.15.62:0/317975736 shutdown_connections 2023-11-15T06:25:42.708 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:42.703+0000 7f5c3edbc700 1 -- 172.21.15.62:0/317975736 wait complete. 2023-11-15T06:25:42.984 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:42 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:25:41.774707+0000 mon.smithi062 (mon.0) 77 : cluster [DBG] mgrmap e11: smithi062.rwurtk(active, since 1.0611s) 2023-11-15T06:25:42.984 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:42 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cephadm 2023-11-15T06:25:41.775798+0000 mgr.smithi062.rwurtk (mgr.14144) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2023-11-15T06:25:42.985 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:42 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:41.785486+0000 mon.smithi062 (mon.0) 78 : audit [INF] from='mgr.14144 172.21.15.62:0/1570055551' entity='mgr.smithi062.rwurtk' 2023-11-15T06:25:42.985 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:42 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:42.702644+0000 mon.smithi062 (mon.0) 79 : audit [INF] from='mgr.14144 172.21.15.62:0/1570055551' entity='mgr.smithi062.rwurtk' 2023-11-15T06:25:43.048 INFO:teuthology.orchestra.run.smithi062.stderr:Deploying node-exporter service with default placement... 2023-11-15T06:25:43.049 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi062 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptvq9ksio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpf12kfgcg:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2023-11-15T06:25:43.382 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.376+0000 7f1a922da700 1 Processor -- start 2023-11-15T06:25:43.382 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.376+0000 7f1a922da700 1 -- start start 2023-11-15T06:25:43.383 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.376+0000 7f1a922da700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a8c0fbaf0 0x7f1a8c0fbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:43.383 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.376+0000 7f1a922da700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8c0fc400 con 0x7f1a8c0fbaf0 2023-11-15T06:25:43.383 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.376+0000 7f1a912d8700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a8c0fbaf0 0x7f1a8c0fbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:43.383 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.376+0000 7f1a912d8700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a8c0fbaf0 0x7f1a8c0fbf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57364/0 (socket says 172.21.15.62:57364) 2023-11-15T06:25:43.384 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.377+0000 7f1a912d8700 1 -- 172.21.15.62:0/2069431915 learned_addr learned my addr 172.21.15.62:0/2069431915 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:43.384 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.377+0000 7f1a912d8700 1 -- 172.21.15.62:0/2069431915 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a8c0fc540 con 0x7f1a8c0fbaf0 2023-11-15T06:25:43.384 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.377+0000 7f1a912d8700 1 --2- 172.21.15.62:0/2069431915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a8c0fbaf0 0x7f1a8c0fbf10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f1a8800b5f0 tx=0x7f1a88006ee0).ready entity=mon.0 client_cookie=1b983389dc5ee1d9 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:43.384 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.377+0000 7f1a7bfff700 1 -- 172.21.15.62:0/2069431915 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a8800d600 con 0x7f1a8c0fbaf0 2023-11-15T06:25:43.384 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.377+0000 7f1a7bfff700 1 -- 172.21.15.62:0/2069431915 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1a8800dbd0 con 0x7f1a8c0fbaf0 2023-11-15T06:25:43.385 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.377+0000 7f1a7bfff700 1 -- 172.21.15.62:0/2069431915 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a88011c00 con 0x7f1a8c0fbaf0 2023-11-15T06:25:43.385 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.378+0000 7f1a922da700 1 -- 172.21.15.62:0/2069431915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a8c0fbaf0 msgr2=0x7f1a8c0fbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:43.385 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.378+0000 7f1a922da700 1 --2- 172.21.15.62:0/2069431915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a8c0fbaf0 0x7f1a8c0fbf10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f1a8800b5f0 tx=0x7f1a88006ee0).stop 2023-11-15T06:25:43.385 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.378+0000 7f1a922da700 1 -- 172.21.15.62:0/2069431915 shutdown_connections 2023-11-15T06:25:43.385 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.378+0000 7f1a922da700 1 --2- 172.21.15.62:0/2069431915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a8c0fbaf0 0x7f1a8c0fbf10 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:43.386 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.378+0000 7f1a922da700 1 -- 172.21.15.62:0/2069431915 >> 172.21.15.62:0/2069431915 conn(0x7f1a8c0f7350 msgr2=0x7f1a8c0f9770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:43.386 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.378+0000 7f1a922da700 1 -- 172.21.15.62:0/2069431915 shutdown_connections 2023-11-15T06:25:43.386 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.378+0000 7f1a922da700 1 -- 172.21.15.62:0/2069431915 wait complete. 2023-11-15T06:25:43.386 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.378+0000 7f1a922da700 1 Processor -- start 2023-11-15T06:25:43.386 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.379+0000 7f1a922da700 1 -- start start 2023-11-15T06:25:43.387 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.379+0000 7f1a922da700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a8c0fbaf0 0x7f1a8c10a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:43.387 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.379+0000 7f1a922da700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8c10ada0 con 0x7f1a8c0fbaf0 2023-11-15T06:25:43.387 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.379+0000 7f1a912d8700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a8c0fbaf0 0x7f1a8c10a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:43.387 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.379+0000 7f1a912d8700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a8c0fbaf0 0x7f1a8c10a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57368/0 (socket says 172.21.15.62:57368) 2023-11-15T06:25:43.387 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.379+0000 7f1a912d8700 1 -- 172.21.15.62:0/153309774 learned_addr learned my addr 172.21.15.62:0/153309774 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:43.388 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.379+0000 7f1a912d8700 1 -- 172.21.15.62:0/153309774 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a8800b200 con 0x7f1a8c0fbaf0 2023-11-15T06:25:43.388 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.379+0000 7f1a912d8700 1 --2- 172.21.15.62:0/153309774 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a8c0fbaf0 0x7f1a8c10a8b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f1a8800e040 tx=0x7f1a88006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:43.388 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.379+0000 7f1a79ffb700 1 -- 172.21.15.62:0/153309774 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a8801b410 con 0x7f1a8c0fbaf0 2023-11-15T06:25:43.388 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.380+0000 7f1a79ffb700 1 -- 172.21.15.62:0/153309774 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1a8801b9e0 con 0x7f1a8c0fbaf0 2023-11-15T06:25:43.388 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.380+0000 7f1a922da700 1 -- 172.21.15.62:0/153309774 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a8c10afa0 con 0x7f1a8c0fbaf0 2023-11-15T06:25:43.389 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.380+0000 7f1a79ffb700 1 -- 172.21.15.62:0/153309774 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a8801a770 con 0x7f1a8c0fbaf0 2023-11-15T06:25:43.389 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.380+0000 7f1a922da700 1 -- 172.21.15.62:0/153309774 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a8c10b3b0 con 0x7f1a8c0fbaf0 2023-11-15T06:25:43.390 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.381+0000 7f1a922da700 1 -- 172.21.15.62:0/153309774 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a8c045ad0 con 0x7f1a8c0fbaf0 2023-11-15T06:25:43.390 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.381+0000 7f1a79ffb700 1 -- 172.21.15.62:0/153309774 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f1a8800f070 con 0x7f1a8c0fbaf0 2023-11-15T06:25:43.390 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.381+0000 7f1a79ffb700 1 --2- 172.21.15.62:0/153309774 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7f1a7402fd50 0x7f1a74032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:43.390 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.381+0000 7f1a79ffb700 1 -- 172.21.15.62:0/153309774 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f1a88041740 con 0x7f1a8c0fbaf0 2023-11-15T06:25:43.391 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.382+0000 7f1a90ad7700 1 --2- 172.21.15.62:0/153309774 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7f1a7402fd50 0x7f1a74032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:43.391 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.382+0000 7f1a90ad7700 1 --2- 172.21.15.62:0/153309774 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7f1a7402fd50 0x7f1a74032210 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f1a7c0096a0 tx=0x7f1a7c006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:43.391 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.385+0000 7f1a79ffb700 1 -- 172.21.15.62:0/153309774 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1a8802ce60 con 0x7f1a8c0fbaf0 2023-11-15T06:25:43.573 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.571+0000 7f1a922da700 1 -- 172.21.15.62:0/153309774 --> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f1a8c10bcf0 con 0x7f1a7402fd50 2023-11-15T06:25:43.577 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.575+0000 7f1a79ffb700 1 -- 172.21.15.62:0/153309774 <== mgr.14144 v2:172.21.15.62:6800/1948381890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f1a8c10bcf0 con 0x7f1a7402fd50 2023-11-15T06:25:43.577 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2023-11-15T06:25:43.579 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.576+0000 7f1a922da700 1 -- 172.21.15.62:0/153309774 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7f1a7402fd50 msgr2=0x7f1a74032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:43.579 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.576+0000 7f1a922da700 1 --2- 172.21.15.62:0/153309774 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7f1a7402fd50 0x7f1a74032210 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f1a7c0096a0 tx=0x7f1a7c006b40).stop 2023-11-15T06:25:43.579 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.576+0000 7f1a922da700 1 -- 172.21.15.62:0/153309774 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a8c0fbaf0 msgr2=0x7f1a8c10a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:43.580 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.576+0000 7f1a922da700 1 --2- 172.21.15.62:0/153309774 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a8c0fbaf0 0x7f1a8c10a8b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f1a8800e040 tx=0x7f1a88006ee0).stop 2023-11-15T06:25:43.580 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.577+0000 7f1a922da700 1 -- 172.21.15.62:0/153309774 shutdown_connections 2023-11-15T06:25:43.581 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.577+0000 7f1a922da700 1 --2- 172.21.15.62:0/153309774 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7f1a7402fd50 0x7f1a74032210 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:43.581 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.577+0000 7f1a922da700 1 --2- 172.21.15.62:0/153309774 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a8c0fbaf0 0x7f1a8c10a8b0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:43.581 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.577+0000 7f1a922da700 1 -- 172.21.15.62:0/153309774 >> 172.21.15.62:0/153309774 conn(0x7f1a8c0f7350 msgr2=0x7f1a8c0f8e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:43.581 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.577+0000 7f1a922da700 1 -- 172.21.15.62:0/153309774 shutdown_connections 2023-11-15T06:25:43.582 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:43.577+0000 7f1a922da700 1 -- 172.21.15.62:0/153309774 wait complete. 2023-11-15T06:25:43.939 INFO:teuthology.orchestra.run.smithi062.stderr:Deploying alertmanager service with default placement... 2023-11-15T06:25:43.939 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi062 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptvq9ksio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpf12kfgcg:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2023-11-15T06:25:44.286 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.278+0000 7fd678348700 1 Processor -- start 2023-11-15T06:25:44.287 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.278+0000 7fd678348700 1 -- start start 2023-11-15T06:25:44.287 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.278+0000 7fd678348700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6700fab80 0x7fd6700fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:44.287 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.278+0000 7fd678348700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6700fb490 con 0x7fd6700fab80 2023-11-15T06:25:44.288 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.278+0000 7fd6760e4700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6700fab80 0x7fd6700fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:44.288 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.278+0000 7fd6760e4700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6700fab80 0x7fd6700fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57372/0 (socket says 172.21.15.62:57372) 2023-11-15T06:25:44.288 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.278+0000 7fd6760e4700 1 -- 172.21.15.62:0/2420329451 learned_addr learned my addr 172.21.15.62:0/2420329451 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:44.288 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.279+0000 7fd6760e4700 1 -- 172.21.15.62:0/2420329451 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6700fb5d0 con 0x7fd6700fab80 2023-11-15T06:25:44.289 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.279+0000 7fd6760e4700 1 --2- 172.21.15.62:0/2420329451 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6700fab80 0x7fd6700fafa0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fd66c00b870 tx=0x7fd66c006f70).ready entity=mon.0 client_cookie=390c79255f99f328 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:44.289 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.279+0000 7fd6750e2700 1 -- 172.21.15.62:0/2420329451 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd66c00da20 con 0x7fd6700fab80 2023-11-15T06:25:44.289 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.279+0000 7fd6750e2700 1 -- 172.21.15.62:0/2420329451 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd66c010470 con 0x7fd6700fab80 2023-11-15T06:25:44.289 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.279+0000 7fd6750e2700 1 -- 172.21.15.62:0/2420329451 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd66c00f5b0 con 0x7fd6700fab80 2023-11-15T06:25:44.289 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.279+0000 7fd678348700 1 -- 172.21.15.62:0/2420329451 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6700fab80 msgr2=0x7fd6700fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:44.290 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.279+0000 7fd678348700 1 --2- 172.21.15.62:0/2420329451 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6700fab80 0x7fd6700fafa0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fd66c00b870 tx=0x7fd66c006f70).stop 2023-11-15T06:25:44.290 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.280+0000 7fd678348700 1 -- 172.21.15.62:0/2420329451 shutdown_connections 2023-11-15T06:25:44.290 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.280+0000 7fd678348700 1 --2- 172.21.15.62:0/2420329451 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6700fab80 0x7fd6700fafa0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:44.290 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.280+0000 7fd678348700 1 -- 172.21.15.62:0/2420329451 >> 172.21.15.62:0/2420329451 conn(0x7fd6700f63c0 msgr2=0x7fd6700f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:44.290 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.280+0000 7fd678348700 1 -- 172.21.15.62:0/2420329451 shutdown_connections 2023-11-15T06:25:44.291 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.280+0000 7fd678348700 1 -- 172.21.15.62:0/2420329451 wait complete. 2023-11-15T06:25:44.291 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.280+0000 7fd678348700 1 Processor -- start 2023-11-15T06:25:44.291 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.280+0000 7fd678348700 1 -- start start 2023-11-15T06:25:44.292 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.280+0000 7fd678348700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6700fab80 0x7fd670108770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:44.292 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.280+0000 7fd678348700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd670108c60 con 0x7fd6700fab80 2023-11-15T06:25:44.292 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.281+0000 7fd6760e4700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6700fab80 0x7fd670108770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:44.293 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.281+0000 7fd6760e4700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6700fab80 0x7fd670108770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57376/0 (socket says 172.21.15.62:57376) 2023-11-15T06:25:44.293 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.281+0000 7fd6760e4700 1 -- 172.21.15.62:0/2426167420 learned_addr learned my addr 172.21.15.62:0/2426167420 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:44.293 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.281+0000 7fd6760e4700 1 -- 172.21.15.62:0/2426167420 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd66c00b480 con 0x7fd6700fab80 2023-11-15T06:25:44.294 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.281+0000 7fd6760e4700 1 --2- 172.21.15.62:0/2426167420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6700fab80 0x7fd670108770 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fd66c006a40 tx=0x7fd66c006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:44.294 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.281+0000 7fd65effd700 1 -- 172.21.15.62:0/2426167420 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd66c00f3d0 con 0x7fd6700fab80 2023-11-15T06:25:44.295 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.281+0000 7fd65effd700 1 -- 172.21.15.62:0/2426167420 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd66c00fef0 con 0x7fd6700fab80 2023-11-15T06:25:44.295 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.281+0000 7fd678348700 1 -- 172.21.15.62:0/2426167420 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd670108e60 con 0x7fd6700fab80 2023-11-15T06:25:44.295 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.281+0000 7fd65effd700 1 -- 172.21.15.62:0/2426167420 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd66c019d40 con 0x7fd6700fab80 2023-11-15T06:25:44.296 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.281+0000 7fd678348700 1 -- 172.21.15.62:0/2426167420 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd670109270 con 0x7fd6700fab80 2023-11-15T06:25:44.296 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.282+0000 7fd65effd700 1 -- 172.21.15.62:0/2426167420 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fd66c02b060 con 0x7fd6700fab80 2023-11-15T06:25:44.296 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.282+0000 7fd65effd700 1 --2- 172.21.15.62:0/2426167420 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7fd65802fe60 0x7fd658032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:44.297 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.282+0000 7fd65effd700 1 -- 172.21.15.62:0/2426167420 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fd66c044250 con 0x7fd6700fab80 2023-11-15T06:25:44.297 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.282+0000 7fd678348700 1 -- 172.21.15.62:0/2426167420 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd670045ad0 con 0x7fd6700fab80 2023-11-15T06:25:44.297 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.282+0000 7fd6758e3700 1 --2- 172.21.15.62:0/2426167420 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7fd65802fe60 0x7fd658032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:44.298 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.283+0000 7fd6758e3700 1 --2- 172.21.15.62:0/2426167420 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7fd65802fe60 0x7fd658032320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fd6600096a0 tx=0x7fd660006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:44.298 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.287+0000 7fd65effd700 1 -- 172.21.15.62:0/2426167420 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd66c014020 con 0x7fd6700fab80 2023-11-15T06:25:44.473 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.471+0000 7fd678348700 1 -- 172.21.15.62:0/2426167420 --> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7fd670102710 con 0x7fd65802fe60 2023-11-15T06:25:44.477 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.475+0000 7fd65effd700 1 -- 172.21.15.62:0/2426167420 <== mgr.14144 v2:172.21.15.62:6800/1948381890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7fd670102710 con 0x7fd65802fe60 2023-11-15T06:25:44.477 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2023-11-15T06:25:44.479 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.476+0000 7fd678348700 1 -- 172.21.15.62:0/2426167420 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7fd65802fe60 msgr2=0x7fd658032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:44.480 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.476+0000 7fd678348700 1 --2- 172.21.15.62:0/2426167420 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7fd65802fe60 0x7fd658032320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fd6600096a0 tx=0x7fd660006b40).stop 2023-11-15T06:25:44.480 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.476+0000 7fd678348700 1 -- 172.21.15.62:0/2426167420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6700fab80 msgr2=0x7fd670108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:44.480 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.476+0000 7fd678348700 1 --2- 172.21.15.62:0/2426167420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6700fab80 0x7fd670108770 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fd66c006a40 tx=0x7fd66c006f70).stop 2023-11-15T06:25:44.480 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.476+0000 7fd678348700 1 -- 172.21.15.62:0/2426167420 shutdown_connections 2023-11-15T06:25:44.481 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.476+0000 7fd678348700 1 --2- 172.21.15.62:0/2426167420 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7fd65802fe60 0x7fd658032320 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:44.481 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.476+0000 7fd678348700 1 --2- 172.21.15.62:0/2426167420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6700fab80 0x7fd670108770 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:44.481 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.476+0000 7fd678348700 1 -- 172.21.15.62:0/2426167420 >> 172.21.15.62:0/2426167420 conn(0x7fd6700f63c0 msgr2=0x7fd6700ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:44.481 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.477+0000 7fd678348700 1 -- 172.21.15.62:0/2426167420 shutdown_connections 2023-11-15T06:25:44.481 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:44.477+0000 7fd678348700 1 -- 172.21.15.62:0/2426167420 wait complete. 2023-11-15T06:25:44.783 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:44 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:42.696434+0000 mgr.smithi062.rwurtk (mgr.14144) 3 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T06:25:44.784 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:44 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cephadm 2023-11-15T06:25:42.696943+0000 mgr.smithi062.rwurtk (mgr.14144) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2023-11-15T06:25:44.784 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:44 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:43.573240+0000 mgr.smithi062.rwurtk (mgr.14144) 5 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T06:25:44.784 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:44 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cephadm 2023-11-15T06:25:43.573893+0000 mgr.smithi062.rwurtk (mgr.14144) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2023-11-15T06:25:44.784 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:44 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:43.576112+0000 mon.smithi062 (mon.0) 80 : audit [INF] from='mgr.14144 172.21.15.62:0/1570055551' entity='mgr.smithi062.rwurtk' 2023-11-15T06:25:44.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:44 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:25:43.704262+0000 mon.smithi062 (mon.0) 81 : cluster [DBG] mgrmap e12: smithi062.rwurtk(active, since 2s) 2023-11-15T06:25:44.785 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:44 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:44.476144+0000 mon.smithi062 (mon.0) 82 : audit [INF] from='mgr.14144 172.21.15.62:0/1570055551' entity='mgr.smithi062.rwurtk' 2023-11-15T06:25:44.848 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi062 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptvq9ksio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpf12kfgcg:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2023-11-15T06:25:45.225 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.221+0000 7fe45e027700 1 Processor -- start 2023-11-15T06:25:45.226 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.221+0000 7fe45e027700 1 -- start start 2023-11-15T06:25:45.226 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.221+0000 7fe45e027700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4580703b0 0x7fe458100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:45.226 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.221+0000 7fe45e027700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4580fc750 con 0x7fe4580703b0 2023-11-15T06:25:45.226 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.221+0000 7fe4577fe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4580703b0 0x7fe458100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:45.226 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.222+0000 7fe4577fe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4580703b0 0x7fe458100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57392/0 (socket says 172.21.15.62:57392) 2023-11-15T06:25:45.227 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.222+0000 7fe4577fe700 1 -- 172.21.15.62:0/1735473456 learned_addr learned my addr 172.21.15.62:0/1735473456 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:45.227 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.222+0000 7fe4577fe700 1 -- 172.21.15.62:0/1735473456 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe458101250 con 0x7fe4580703b0 2023-11-15T06:25:45.227 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.222+0000 7fe4577fe700 1 --2- 172.21.15.62:0/1735473456 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4580703b0 0x7fe458100d60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fe44800b5f0 tx=0x7fe448006ee0).ready entity=mon.0 client_cookie=68d725d96690f128 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:45.227 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.222+0000 7fe4567fc700 1 -- 172.21.15.62:0/1735473456 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe44800d600 con 0x7fe4580703b0 2023-11-15T06:25:45.227 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.222+0000 7fe4567fc700 1 -- 172.21.15.62:0/1735473456 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe44800dbd0 con 0x7fe4580703b0 2023-11-15T06:25:45.229 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.222+0000 7fe4567fc700 1 -- 172.21.15.62:0/1735473456 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe448011c00 con 0x7fe4580703b0 2023-11-15T06:25:45.230 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.223+0000 7fe45e027700 1 -- 172.21.15.62:0/1735473456 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4580703b0 msgr2=0x7fe458100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:45.230 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.223+0000 7fe45e027700 1 --2- 172.21.15.62:0/1735473456 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4580703b0 0x7fe458100d60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fe44800b5f0 tx=0x7fe448006ee0).stop 2023-11-15T06:25:45.230 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.223+0000 7fe45e027700 1 -- 172.21.15.62:0/1735473456 shutdown_connections 2023-11-15T06:25:45.230 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.223+0000 7fe45e027700 1 --2- 172.21.15.62:0/1735473456 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4580703b0 0x7fe458100d60 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:45.230 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.223+0000 7fe45e027700 1 -- 172.21.15.62:0/1735473456 >> 172.21.15.62:0/1735473456 conn(0x7fe4580f63c0 msgr2=0x7fe4580f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:45.231 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.223+0000 7fe45e027700 1 -- 172.21.15.62:0/1735473456 shutdown_connections 2023-11-15T06:25:45.231 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.223+0000 7fe45e027700 1 -- 172.21.15.62:0/1735473456 wait complete. 2023-11-15T06:25:45.231 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.223+0000 7fe45e027700 1 Processor -- start 2023-11-15T06:25:45.231 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.224+0000 7fe45e027700 1 -- start start 2023-11-15T06:25:45.231 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.224+0000 7fe45e027700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4580703b0 0x7fe45810cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:45.232 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.224+0000 7fe45e027700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe45810d230 con 0x7fe4580703b0 2023-11-15T06:25:45.232 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.224+0000 7fe4577fe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4580703b0 0x7fe45810cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:45.232 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.224+0000 7fe4577fe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4580703b0 0x7fe45810cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57406/0 (socket says 172.21.15.62:57406) 2023-11-15T06:25:45.232 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.224+0000 7fe4577fe700 1 -- 172.21.15.62:0/1272535601 learned_addr learned my addr 172.21.15.62:0/1272535601 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:45.232 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.224+0000 7fe4577fe700 1 -- 172.21.15.62:0/1272535601 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe44800b200 con 0x7fe4580703b0 2023-11-15T06:25:45.233 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.224+0000 7fe4577fe700 1 --2- 172.21.15.62:0/1272535601 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4580703b0 0x7fe45810cd40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fe448002f40 tx=0x7fe448006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:45.233 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.225+0000 7fe43ffff700 1 -- 172.21.15.62:0/1272535601 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe448011690 con 0x7fe4580703b0 2023-11-15T06:25:45.233 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.225+0000 7fe45e027700 1 -- 172.21.15.62:0/1272535601 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe45810d430 con 0x7fe4580703b0 2023-11-15T06:25:45.233 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.225+0000 7fe45e027700 1 -- 172.21.15.62:0/1272535601 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe45810d840 con 0x7fe4580703b0 2023-11-15T06:25:45.233 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.225+0000 7fe43ffff700 1 -- 172.21.15.62:0/1272535601 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe448006a70 con 0x7fe4580703b0 2023-11-15T06:25:45.234 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.225+0000 7fe43ffff700 1 -- 172.21.15.62:0/1272535601 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe44801b5f0 con 0x7fe4580703b0 2023-11-15T06:25:45.234 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.226+0000 7fe43ffff700 1 -- 172.21.15.62:0/1272535601 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fe44800f070 con 0x7fe4580703b0 2023-11-15T06:25:45.234 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.226+0000 7fe45e027700 1 -- 172.21.15.62:0/1272535601 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe438004fa0 con 0x7fe4580703b0 2023-11-15T06:25:45.234 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.226+0000 7fe43ffff700 1 --2- 172.21.15.62:0/1272535601 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7fe44402feb0 0x7fe444032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:45.234 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.226+0000 7fe43ffff700 1 -- 172.21.15.62:0/1272535601 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fe4480421b0 con 0x7fe4580703b0 2023-11-15T06:25:45.235 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.227+0000 7fe456ffd700 1 --2- 172.21.15.62:0/1272535601 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7fe44402feb0 0x7fe444032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:45.235 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.227+0000 7fe456ffd700 1 --2- 172.21.15.62:0/1272535601 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7fe44402feb0 0x7fe444032370 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fe44c0096a0 tx=0x7fe44c006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:45.235 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.231+0000 7fe43ffff700 1 -- 172.21.15.62:0/1272535601 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe448022900 con 0x7fe4580703b0 2023-11-15T06:25:45.406 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.403+0000 7fe45e027700 1 -- 172.21.15.62:0/1272535601 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fe438005670 con 0x7fe4580703b0 2023-11-15T06:25:45.408 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.406+0000 7fe43ffff700 1 -- 172.21.15.62:0/1272535601 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v8)=0 v8) v1 ==== 142+0+0 (secure 0 0 0) 0x7fe448016030 con 0x7fe4580703b0 2023-11-15T06:25:45.415 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.408+0000 7fe45e027700 1 -- 172.21.15.62:0/1272535601 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7fe44402feb0 msgr2=0x7fe444032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:45.415 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.408+0000 7fe45e027700 1 --2- 172.21.15.62:0/1272535601 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7fe44402feb0 0x7fe444032370 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fe44c0096a0 tx=0x7fe44c006b40).stop 2023-11-15T06:25:45.415 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.408+0000 7fe45e027700 1 -- 172.21.15.62:0/1272535601 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4580703b0 msgr2=0x7fe45810cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:45.416 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.408+0000 7fe45e027700 1 --2- 172.21.15.62:0/1272535601 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4580703b0 0x7fe45810cd40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fe448002f40 tx=0x7fe448006ee0).stop 2023-11-15T06:25:45.416 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.408+0000 7fe45e027700 1 -- 172.21.15.62:0/1272535601 shutdown_connections 2023-11-15T06:25:45.416 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.408+0000 7fe45e027700 1 --2- 172.21.15.62:0/1272535601 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7fe44402feb0 0x7fe444032370 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:45.416 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.408+0000 7fe45e027700 1 --2- 172.21.15.62:0/1272535601 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4580703b0 0x7fe45810cd40 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:45.416 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.408+0000 7fe45e027700 1 -- 172.21.15.62:0/1272535601 >> 172.21.15.62:0/1272535601 conn(0x7fe4580f63c0 msgr2=0x7fe4580f7fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:45.417 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.408+0000 7fe45e027700 1 -- 172.21.15.62:0/1272535601 shutdown_connections 2023-11-15T06:25:45.417 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:45.408+0000 7fe45e027700 1 -- 172.21.15.62:0/1272535601 wait complete. 2023-11-15T06:25:45.850 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:45 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:44.473455+0000 mgr.smithi062.rwurtk (mgr.14144) 7 : audit [DBG] from='client.14156 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T06:25:45.850 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:45 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cephadm 2023-11-15T06:25:44.473979+0000 mgr.smithi062.rwurtk (mgr.14144) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2023-11-15T06:25:45.850 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:45 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:45.407830+0000 mon.smithi062 (mon.0) 83 : audit [INF] from='client.? 172.21.15.62:0/1272535601' entity='client.admin' 2023-11-15T06:25:45.915 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi062 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptvq9ksio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpf12kfgcg:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2023-11-15T06:25:46.427 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.424+0000 7f4df8be5700 1 Processor -- start 2023-11-15T06:25:46.427 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.424+0000 7f4df8be5700 1 -- start start 2023-11-15T06:25:46.428 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.425+0000 7f4df8be5700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4df40703d0 0x7f4df40fe940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:46.428 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.425+0000 7f4df8be5700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df40fee30 con 0x7f4df40703d0 2023-11-15T06:25:46.428 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.425+0000 7f4df259c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4df40703d0 0x7f4df40fe940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:46.430 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.425+0000 7f4df259c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4df40703d0 0x7f4df40fe940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57420/0 (socket says 172.21.15.62:57420) 2023-11-15T06:25:46.431 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.425+0000 7f4df259c700 1 -- 172.21.15.62:0/3212639206 learned_addr learned my addr 172.21.15.62:0/3212639206 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:46.431 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.425+0000 7f4df259c700 1 -- 172.21.15.62:0/3212639206 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df40fef70 con 0x7f4df40703d0 2023-11-15T06:25:46.431 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.425+0000 7f4df259c700 1 --2- 172.21.15.62:0/3212639206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4df40703d0 0x7f4df40fe940 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f4de400b5f0 tx=0x7f4de4006ee0).ready entity=mon.0 client_cookie=3eddd193ae5e1491 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:46.431 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.425+0000 7f4df159a700 1 -- 172.21.15.62:0/3212639206 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4de400d600 con 0x7f4df40703d0 2023-11-15T06:25:46.431 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.425+0000 7f4df159a700 1 -- 172.21.15.62:0/3212639206 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4de400dc10 con 0x7f4df40703d0 2023-11-15T06:25:46.432 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.425+0000 7f4df159a700 1 -- 172.21.15.62:0/3212639206 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4de4011cd0 con 0x7f4df40703d0 2023-11-15T06:25:46.432 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.426+0000 7f4df8be5700 1 -- 172.21.15.62:0/3212639206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4df40703d0 msgr2=0x7f4df40fe940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:46.432 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.426+0000 7f4df8be5700 1 --2- 172.21.15.62:0/3212639206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4df40703d0 0x7f4df40fe940 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f4de400b5f0 tx=0x7f4de4006ee0).stop 2023-11-15T06:25:46.432 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.426+0000 7f4df8be5700 1 -- 172.21.15.62:0/3212639206 shutdown_connections 2023-11-15T06:25:46.432 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.426+0000 7f4df8be5700 1 --2- 172.21.15.62:0/3212639206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4df40703d0 0x7f4df40fe940 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:46.432 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.426+0000 7f4df8be5700 1 -- 172.21.15.62:0/3212639206 >> 172.21.15.62:0/3212639206 conn(0x7f4df40f63c0 msgr2=0x7f4df40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:46.433 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.426+0000 7f4df8be5700 1 -- 172.21.15.62:0/3212639206 shutdown_connections 2023-11-15T06:25:46.433 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.426+0000 7f4df8be5700 1 -- 172.21.15.62:0/3212639206 wait complete. 2023-11-15T06:25:46.433 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.427+0000 7f4df8be5700 1 Processor -- start 2023-11-15T06:25:46.433 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.427+0000 7f4df8be5700 1 -- start start 2023-11-15T06:25:46.433 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.427+0000 7f4df8be5700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4df40703d0 0x7f4df410aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:46.434 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.427+0000 7f4df8be5700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df410af90 con 0x7f4df40703d0 2023-11-15T06:25:46.434 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.427+0000 7f4df259c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4df40703d0 0x7f4df410aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:46.434 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.427+0000 7f4df259c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4df40703d0 0x7f4df410aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57428/0 (socket says 172.21.15.62:57428) 2023-11-15T06:25:46.434 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.427+0000 7f4df259c700 1 -- 172.21.15.62:0/25653615 learned_addr learned my addr 172.21.15.62:0/25653615 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:46.435 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.428+0000 7f4df259c700 1 -- 172.21.15.62:0/25653615 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de400b200 con 0x7f4df40703d0 2023-11-15T06:25:46.435 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.428+0000 7f4df259c700 1 --2- 172.21.15.62:0/25653615 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4df40703d0 0x7f4df410aaa0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f4de400df90 tx=0x7f4de4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:46.435 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.428+0000 7f4de2ffd700 1 -- 172.21.15.62:0/25653615 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4de4011da0 con 0x7f4df40703d0 2023-11-15T06:25:46.435 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.428+0000 7f4df8be5700 1 -- 172.21.15.62:0/25653615 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4df410b190 con 0x7f4df40703d0 2023-11-15T06:25:46.436 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.428+0000 7f4de2ffd700 1 -- 172.21.15.62:0/25653615 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4de401b8c0 con 0x7f4df40703d0 2023-11-15T06:25:46.436 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.428+0000 7f4de2ffd700 1 -- 172.21.15.62:0/25653615 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4de401a930 con 0x7f4df40703d0 2023-11-15T06:25:46.436 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.428+0000 7f4df8be5700 1 -- 172.21.15.62:0/25653615 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4df410b5a0 con 0x7f4df40703d0 2023-11-15T06:25:46.436 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.429+0000 7f4de2ffd700 1 -- 172.21.15.62:0/25653615 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f4de400f070 con 0x7f4df40703d0 2023-11-15T06:25:46.436 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.429+0000 7f4de2ffd700 1 --2- 172.21.15.62:0/25653615 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7f4dd802feb0 0x7f4dd8032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:46.436 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.429+0000 7f4de2ffd700 1 -- 172.21.15.62:0/25653615 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f4de4041080 con 0x7f4df40703d0 2023-11-15T06:25:46.437 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.430+0000 7f4df8be5700 1 -- 172.21.15.62:0/25653615 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4df4059100 con 0x7f4df40703d0 2023-11-15T06:25:46.437 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.430+0000 7f4df1d9b700 1 --2- 172.21.15.62:0/25653615 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7f4dd802feb0 0x7f4dd8032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:46.437 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.430+0000 7f4df1d9b700 1 --2- 172.21.15.62:0/25653615 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7f4dd802feb0 0x7f4dd8032370 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f4de80096a0 tx=0x7f4de8006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:46.442 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.437+0000 7f4de2ffd700 1 -- 172.21.15.62:0/25653615 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4de4016070 con 0x7f4df40703d0 2023-11-15T06:25:46.600 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.597+0000 7f4df8be5700 1 -- 172.21.15.62:0/25653615 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f4df4045ad0 con 0x7f4df40703d0 2023-11-15T06:25:46.605 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.603+0000 7f4de2ffd700 1 -- 172.21.15.62:0/25653615 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v10)=0 v10) v1 ==== 130+0+0 (secure 0 0 0) 0x7f4de4024410 con 0x7f4df40703d0 2023-11-15T06:25:46.607 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.605+0000 7f4df8be5700 1 -- 172.21.15.62:0/25653615 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7f4dd802feb0 msgr2=0x7f4dd8032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:46.608 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.605+0000 7f4df8be5700 1 --2- 172.21.15.62:0/25653615 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7f4dd802feb0 0x7f4dd8032370 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f4de80096a0 tx=0x7f4de8006b40).stop 2023-11-15T06:25:46.608 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.605+0000 7f4df8be5700 1 -- 172.21.15.62:0/25653615 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4df40703d0 msgr2=0x7f4df410aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:46.608 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.605+0000 7f4df8be5700 1 --2- 172.21.15.62:0/25653615 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4df40703d0 0x7f4df410aaa0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f4de400df90 tx=0x7f4de4006ee0).stop 2023-11-15T06:25:46.609 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.605+0000 7f4df8be5700 1 -- 172.21.15.62:0/25653615 shutdown_connections 2023-11-15T06:25:46.609 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.605+0000 7f4df8be5700 1 --2- 172.21.15.62:0/25653615 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7f4dd802feb0 0x7f4dd8032370 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:46.610 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.605+0000 7f4df8be5700 1 --2- 172.21.15.62:0/25653615 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4df40703d0 0x7f4df410aaa0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:46.610 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.605+0000 7f4df8be5700 1 -- 172.21.15.62:0/25653615 >> 172.21.15.62:0/25653615 conn(0x7f4df40f63c0 msgr2=0x7f4df40f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:46.610 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.605+0000 7f4df8be5700 1 -- 172.21.15.62:0/25653615 shutdown_connections 2023-11-15T06:25:46.611 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:46.605+0000 7f4df8be5700 1 -- 172.21.15.62:0/25653615 wait complete. 2023-11-15T06:25:46.973 INFO:teuthology.orchestra.run.smithi062.stderr:Enabling the dashboard module... 2023-11-15T06:25:46.974 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi062 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptvq9ksio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpf12kfgcg:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2023-11-15T06:25:47.339 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:47.333+0000 7ff50f36a700 1 Processor -- start 2023-11-15T06:25:47.339 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:47.333+0000 7ff50f36a700 1 -- start start 2023-11-15T06:25:47.340 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:47.333+0000 7ff50f36a700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff508070420 0x7ff508100d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:47.340 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:47.333+0000 7ff50f36a700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5080fc760 con 0x7ff508070420 2023-11-15T06:25:47.340 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:47.333+0000 7ff50d106700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff508070420 0x7ff508100d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:47.340 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:47.333+0000 7ff50d106700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff508070420 0x7ff508100d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57440/0 (socket says 172.21.15.62:57440) 2023-11-15T06:25:47.341 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:47.333+0000 7ff50d106700 1 -- 172.21.15.62:0/2306325095 learned_addr learned my addr 172.21.15.62:0/2306325095 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:47.341 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:47.334+0000 7ff50d106700 1 -- 172.21.15.62:0/2306325095 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff508101260 con 0x7ff508070420 2023-11-15T06:25:47.341 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:47.334+0000 7ff50d106700 1 --2- 172.21.15.62:0/2306325095 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff508070420 0x7ff508100d70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7ff50400b5f0 tx=0x7ff504006ee0).ready entity=mon.0 client_cookie=ad8d0ad6d753e65c server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:47.341 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:47.334+0000 7ff4f7fff700 1 -- 172.21.15.62:0/2306325095 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff50400d600 con 0x7ff508070420 2023-11-15T06:25:47.342 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:47.334+0000 7ff4f7fff700 1 -- 172.21.15.62:0/2306325095 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff50400dc10 con 0x7ff508070420 2023-11-15T06:25:47.342 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:47.334+0000 7ff4f7fff700 1 -- 172.21.15.62:0/2306325095 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff504011cd0 con 0x7ff508070420 2023-11-15T06:25:47.342 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:47.335+0000 7ff50f36a700 1 -- 172.21.15.62:0/2306325095 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff508070420 msgr2=0x7ff508100d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:47.342 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:47.335+0000 7ff50f36a700 1 --2- 172.21.15.62:0/2306325095 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff508070420 0x7ff508100d70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7ff50400b5f0 tx=0x7ff504006ee0).stop 2023-11-15T06:25:47.343 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:47.335+0000 7ff50f36a700 1 -- 172.21.15.62:0/2306325095 shutdown_connections 2023-11-15T06:25:47.343 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:47.335+0000 7ff50f36a700 1 --2- 172.21.15.62:0/2306325095 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff508070420 0x7ff508100d70 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:47.343 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:47.335+0000 7ff50f36a700 1 -- 172.21.15.62:0/2306325095 >> 172.21.15.62:0/2306325095 conn(0x7ff5080f63b0 msgr2=0x7ff5080f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:47.343 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:47.335+0000 7ff50f36a700 1 -- 172.21.15.62:0/2306325095 shutdown_connections 2023-11-15T06:25:47.343 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:47.335+0000 7ff50f36a700 1 -- 172.21.15.62:0/2306325095 wait complete. 2023-11-15T06:25:47.344 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:47.336+0000 7ff50f36a700 1 Processor -- start 2023-11-15T06:25:47.344 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:47.336+0000 7ff50f36a700 1 -- start start 2023-11-15T06:25:47.344 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:47.336+0000 7ff50f36a700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff508070420 0x7ff508115490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:47.344 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:47.336+0000 7ff50f36a700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff508115980 con 0x7ff508070420 2023-11-15T06:25:47.345 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:47.336+0000 7ff50d106700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff508070420 0x7ff508115490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:47.345 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:47.336+0000 7ff50d106700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff508070420 0x7ff508115490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57444/0 (socket says 172.21.15.62:57444) 2023-11-15T06:25:47.345 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:47.336+0000 7ff50d106700 1 -- 172.21.15.62:0/2538985105 learned_addr learned my addr 172.21.15.62:0/2538985105 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:47.345 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:47.336+0000 7ff50d106700 1 -- 172.21.15.62:0/2538985105 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff50400b200 con 0x7ff508070420 2023-11-15T06:25:47.346 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:47.337+0000 7ff50d106700 1 --2- 172.21.15.62:0/2538985105 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff508070420 0x7ff508115490 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7ff50400dfb0 tx=0x7ff504006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:47.347 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:47.337+0000 7ff4f5ffb700 1 -- 172.21.15.62:0/2538985105 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff50401b410 con 0x7ff508070420 2023-11-15T06:25:47.347 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:47.337+0000 7ff4f5ffb700 1 -- 172.21.15.62:0/2538985105 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff50401ba20 con 0x7ff508070420 2023-11-15T06:25:47.348 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:47.337+0000 7ff4f5ffb700 1 -- 172.21.15.62:0/2538985105 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff50401a970 con 0x7ff508070420 2023-11-15T06:25:47.348 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:47.337+0000 7ff50f36a700 1 -- 172.21.15.62:0/2538985105 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff508115b80 con 0x7ff508070420 2023-11-15T06:25:47.348 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:47.337+0000 7ff50f36a700 1 -- 172.21.15.62:0/2538985105 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff508115f90 con 0x7ff508070420 2023-11-15T06:25:47.348 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:47.338+0000 7ff4f5ffb700 1 -- 172.21.15.62:0/2538985105 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7ff50400f070 con 0x7ff508070420 2023-11-15T06:25:47.348 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:47.338+0000 7ff4f5ffb700 1 --2- 172.21.15.62:0/2538985105 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7ff4f002feb0 0x7ff4f0032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:47.349 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:47.338+0000 7ff4f5ffb700 1 -- 172.21.15.62:0/2538985105 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7ff504041080 con 0x7ff508070420 2023-11-15T06:25:47.349 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:47.338+0000 7ff50f36a700 1 -- 172.21.15.62:0/2538985105 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff508059100 con 0x7ff508070420 2023-11-15T06:25:47.349 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:47.338+0000 7ff50c905700 1 --2- 172.21.15.62:0/2538985105 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7ff4f002feb0 0x7ff4f0032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:47.349 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:47.339+0000 7ff50c905700 1 --2- 172.21.15.62:0/2538985105 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7ff4f002feb0 0x7ff4f0032370 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7ff4f80096a0 tx=0x7ff4f8006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:47.350 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:47.343+0000 7ff4f5ffb700 1 -- 172.21.15.62:0/2538985105 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff504016070 con 0x7ff508070420 2023-11-15T06:25:47.548 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:47.546+0000 7ff50f36a700 1 -- 172.21.15.62:0/2538985105 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7ff508101e30 con 0x7ff508070420 2023-11-15T06:25:47.667 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:47 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:46.218576+0000 mon.smithi062 (mon.0) 84 : audit [INF] from='mgr.14144 172.21.15.62:0/1570055551' entity='mgr.smithi062.rwurtk' 2023-11-15T06:25:47.668 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:47 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:46.457300+0000 mon.smithi062 (mon.0) 85 : audit [INF] from='mgr.14144 172.21.15.62:0/1570055551' entity='mgr.smithi062.rwurtk' 2023-11-15T06:25:47.668 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:47 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:46.604469+0000 mon.smithi062 (mon.0) 86 : audit [INF] from='client.? 172.21.15.62:0/25653615' entity='client.admin' 2023-11-15T06:25:48.518 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:48.515+0000 7ff4f5ffb700 1 -- 172.21.15.62:0/2538985105 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7ff504024790 con 0x7ff508070420 2023-11-15T06:25:48.519 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:48.516+0000 7ff4f5ffb700 1 -- 172.21.15.62:0/2538985105 <== mon.0 v2:172.21.15.62:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v13) v1 ==== 88+0+0 (secure 0 0 0) 0x7ff50402d770 con 0x7ff508070420 2023-11-15T06:25:48.521 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:48.519+0000 7ff50f36a700 1 -- 172.21.15.62:0/2538985105 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7ff4f002feb0 msgr2=0x7ff4f0032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:48.521 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:48.519+0000 7ff50f36a700 1 --2- 172.21.15.62:0/2538985105 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7ff4f002feb0 0x7ff4f0032370 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7ff4f80096a0 tx=0x7ff4f8006b40).stop 2023-11-15T06:25:48.522 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:48.519+0000 7ff50f36a700 1 -- 172.21.15.62:0/2538985105 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff508070420 msgr2=0x7ff508115490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:48.522 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:48.519+0000 7ff50f36a700 1 --2- 172.21.15.62:0/2538985105 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff508070420 0x7ff508115490 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7ff50400dfb0 tx=0x7ff504006ee0).stop 2023-11-15T06:25:48.523 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:48.519+0000 7ff50f36a700 1 -- 172.21.15.62:0/2538985105 shutdown_connections 2023-11-15T06:25:48.523 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:48.519+0000 7ff50f36a700 1 --2- 172.21.15.62:0/2538985105 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7ff4f002feb0 0x7ff4f0032370 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:48.523 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:48.519+0000 7ff50f36a700 1 --2- 172.21.15.62:0/2538985105 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff508070420 0x7ff508115490 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:48.523 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:48.519+0000 7ff50f36a700 1 -- 172.21.15.62:0/2538985105 >> 172.21.15.62:0/2538985105 conn(0x7ff5080f63b0 msgr2=0x7ff5080f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:48.523 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:48.519+0000 7ff50f36a700 1 -- 172.21.15.62:0/2538985105 shutdown_connections 2023-11-15T06:25:48.524 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:48.519+0000 7ff50f36a700 1 -- 172.21.15.62:0/2538985105 wait complete. 2023-11-15T06:25:48.833 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:48 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:47.514361+0000 mon.smithi062 (mon.0) 87 : audit [INF] from='mgr.14144 172.21.15.62:0/1570055551' entity='mgr.smithi062.rwurtk' 2023-11-15T06:25:48.833 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:48 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:47.548228+0000 mon.smithi062 (mon.0) 88 : audit [INF] from='client.? 172.21.15.62:0/2538985105' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-11-15T06:25:48.864 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi062 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptvq9ksio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpf12kfgcg:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2023-11-15T06:25:49.238 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.234+0000 7fbde2f0b700 1 Processor -- start 2023-11-15T06:25:49.238 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.234+0000 7fbde2f0b700 1 -- start start 2023-11-15T06:25:49.238 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.234+0000 7fbde2f0b700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbddc0fd2a0 0x7fbddc0fd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:49.239 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.234+0000 7fbde2f0b700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbddc0fdbb0 con 0x7fbddc0fd2a0 2023-11-15T06:25:49.239 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.234+0000 7fbde0ca7700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbddc0fd2a0 0x7fbddc0fd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:49.239 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.234+0000 7fbde0ca7700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbddc0fd2a0 0x7fbddc0fd6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57480/0 (socket says 172.21.15.62:57480) 2023-11-15T06:25:49.239 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.234+0000 7fbde0ca7700 1 -- 172.21.15.62:0/328589857 learned_addr learned my addr 172.21.15.62:0/328589857 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:49.240 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.235+0000 7fbde0ca7700 1 -- 172.21.15.62:0/328589857 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbddc0fdcf0 con 0x7fbddc0fd2a0 2023-11-15T06:25:49.240 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.235+0000 7fbde0ca7700 1 --2- 172.21.15.62:0/328589857 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbddc0fd2a0 0x7fbddc0fd6c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fbdd000b5f0 tx=0x7fbdd0006ee0).ready entity=mon.0 client_cookie=73bf6f3b6b4358ac server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:49.240 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.235+0000 7fbddb7fe700 1 -- 172.21.15.62:0/328589857 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbdd000d600 con 0x7fbddc0fd2a0 2023-11-15T06:25:49.241 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.235+0000 7fbddb7fe700 1 -- 172.21.15.62:0/328589857 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbdd000dc10 con 0x7fbddc0fd2a0 2023-11-15T06:25:49.241 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.235+0000 7fbddb7fe700 1 -- 172.21.15.62:0/328589857 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbdd0011cd0 con 0x7fbddc0fd2a0 2023-11-15T06:25:49.241 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.236+0000 7fbde2f0b700 1 -- 172.21.15.62:0/328589857 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbddc0fd2a0 msgr2=0x7fbddc0fd6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:49.241 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.236+0000 7fbde2f0b700 1 --2- 172.21.15.62:0/328589857 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbddc0fd2a0 0x7fbddc0fd6c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fbdd000b5f0 tx=0x7fbdd0006ee0).stop 2023-11-15T06:25:49.242 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.236+0000 7fbde2f0b700 1 -- 172.21.15.62:0/328589857 shutdown_connections 2023-11-15T06:25:49.242 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.236+0000 7fbde2f0b700 1 --2- 172.21.15.62:0/328589857 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbddc0fd2a0 0x7fbddc0fd6c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:49.242 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.236+0000 7fbde2f0b700 1 -- 172.21.15.62:0/328589857 >> 172.21.15.62:0/328589857 conn(0x7fbddc0f5b40 msgr2=0x7fbddc0f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:49.242 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.236+0000 7fbde2f0b700 1 -- 172.21.15.62:0/328589857 shutdown_connections 2023-11-15T06:25:49.242 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.236+0000 7fbde2f0b700 1 -- 172.21.15.62:0/328589857 wait complete. 2023-11-15T06:25:49.243 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.237+0000 7fbde2f0b700 1 Processor -- start 2023-11-15T06:25:49.243 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.237+0000 7fbde2f0b700 1 -- start start 2023-11-15T06:25:49.243 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.237+0000 7fbde2f0b700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbddc0fd2a0 0x7fbddc10aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:49.243 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.237+0000 7fbde2f0b700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbddc10af20 con 0x7fbddc0fd2a0 2023-11-15T06:25:49.243 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.237+0000 7fbde0ca7700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbddc0fd2a0 0x7fbddc10aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:49.244 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.237+0000 7fbde0ca7700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbddc0fd2a0 0x7fbddc10aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57492/0 (socket says 172.21.15.62:57492) 2023-11-15T06:25:49.244 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.237+0000 7fbde0ca7700 1 -- 172.21.15.62:0/4226370220 learned_addr learned my addr 172.21.15.62:0/4226370220 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:49.244 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.238+0000 7fbde0ca7700 1 -- 172.21.15.62:0/4226370220 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd000b200 con 0x7fbddc0fd2a0 2023-11-15T06:25:49.244 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.238+0000 7fbde0ca7700 1 --2- 172.21.15.62:0/4226370220 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbddc0fd2a0 0x7fbddc10aa30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fbdd000e040 tx=0x7fbdd0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:49.244 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.238+0000 7fbdd97fa700 1 -- 172.21.15.62:0/4226370220 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbdd001b410 con 0x7fbddc0fd2a0 2023-11-15T06:25:49.245 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.238+0000 7fbdd97fa700 1 -- 172.21.15.62:0/4226370220 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbdd001ba20 con 0x7fbddc0fd2a0 2023-11-15T06:25:49.245 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.238+0000 7fbdd97fa700 1 -- 172.21.15.62:0/4226370220 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbdd001a910 con 0x7fbddc0fd2a0 2023-11-15T06:25:49.245 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.238+0000 7fbde2f0b700 1 -- 172.21.15.62:0/4226370220 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbddc10b120 con 0x7fbddc0fd2a0 2023-11-15T06:25:49.245 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.238+0000 7fbde2f0b700 1 -- 172.21.15.62:0/4226370220 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbddc10b530 con 0x7fbddc0fd2a0 2023-11-15T06:25:49.246 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.239+0000 7fbdd97fa700 1 -- 172.21.15.62:0/4226370220 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fbdd000f070 con 0x7fbddc0fd2a0 2023-11-15T06:25:49.246 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.239+0000 7fbde2f0b700 1 -- 172.21.15.62:0/4226370220 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbddc045ad0 con 0x7fbddc0fd2a0 2023-11-15T06:25:49.247 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.239+0000 7fbdd97fa700 1 --2- 172.21.15.62:0/4226370220 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7fbdc002feb0 0x7fbdc0032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:49.247 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.239+0000 7fbdd97fa700 1 -- 172.21.15.62:0/4226370220 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fbdd0041220 con 0x7fbddc0fd2a0 2023-11-15T06:25:49.247 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.239+0000 7fbddbfff700 1 -- 172.21.15.62:0/4226370220 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7fbdc002feb0 msgr2=0x7fbdc0032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/1948381890 2023-11-15T06:25:49.247 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.239+0000 7fbddbfff700 1 --2- 172.21.15.62:0/4226370220 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7fbdc002feb0 0x7fbdc0032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-15T06:25:49.248 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.243+0000 7fbdd97fa700 1 -- 172.21.15.62:0/4226370220 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbdd0019c90 con 0x7fbddc0fd2a0 2023-11-15T06:25:49.442 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.440+0000 7fbddbfff700 1 -- 172.21.15.62:0/4226370220 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7fbdc002feb0 msgr2=0x7fbdc0032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/1948381890 2023-11-15T06:25:49.443 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.440+0000 7fbddbfff700 1 --2- 172.21.15.62:0/4226370220 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7fbdc002feb0 0x7fbdc0032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-15T06:25:49.444 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.441+0000 7fbde2f0b700 1 -- 172.21.15.62:0/4226370220 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fbddc10c5e0 con 0x7fbddc0fd2a0 2023-11-15T06:25:49.445 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.442+0000 7fbdd97fa700 1 -- 172.21.15.62:0/4226370220 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v13) v1 ==== 56+0+104 (secure 0 0 0) 0x7fbdd0016070 con 0x7fbddc0fd2a0 2023-11-15T06:25:49.445 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: { 2023-11-15T06:25:49.445 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "epoch": 13, 2023-11-15T06:25:49.445 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "available": true, 2023-11-15T06:25:49.446 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "active_name": "smithi062.rwurtk", 2023-11-15T06:25:49.446 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_standby": 0 2023-11-15T06:25:49.446 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: } 2023-11-15T06:25:49.448 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.444+0000 7fbde2f0b700 1 -- 172.21.15.62:0/4226370220 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7fbdc002feb0 msgr2=0x7fbdc0032370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-15T06:25:49.448 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.444+0000 7fbde2f0b700 1 --2- 172.21.15.62:0/4226370220 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7fbdc002feb0 0x7fbdc0032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T06:25:49.448 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.444+0000 7fbde2f0b700 1 -- 172.21.15.62:0/4226370220 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbddc0fd2a0 msgr2=0x7fbddc10aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:49.448 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.444+0000 7fbde2f0b700 1 --2- 172.21.15.62:0/4226370220 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbddc0fd2a0 0x7fbddc10aa30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fbdd000e040 tx=0x7fbdd0006ee0).stop 2023-11-15T06:25:49.449 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.444+0000 7fbde2f0b700 1 -- 172.21.15.62:0/4226370220 shutdown_connections 2023-11-15T06:25:49.449 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.444+0000 7fbde2f0b700 1 --2- 172.21.15.62:0/4226370220 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7fbdc002feb0 0x7fbdc0032370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T06:25:49.449 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.444+0000 7fbde2f0b700 1 --2- 172.21.15.62:0/4226370220 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbddc0fd2a0 0x7fbddc10aa30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:49.449 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.444+0000 7fbde2f0b700 1 -- 172.21.15.62:0/4226370220 >> 172.21.15.62:0/4226370220 conn(0x7fbddc0f5b40 msgr2=0x7fbddc0ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:49.449 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.444+0000 7fbde2f0b700 1 -- 172.21.15.62:0/4226370220 shutdown_connections 2023-11-15T06:25:49.450 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:49.444+0000 7fbde2f0b700 1 -- 172.21.15.62:0/4226370220 wait complete. 2023-11-15T06:25:49.733 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:49 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:48.518097+0000 mon.smithi062 (mon.0) 89 : audit [INF] from='client.? 172.21.15.62:0/2538985105' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-11-15T06:25:49.733 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:49 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:25:48.518155+0000 mon.smithi062 (mon.0) 90 : cluster [DBG] mgrmap e13: smithi062.rwurtk(active, since 7s) 2023-11-15T06:25:49.733 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:49 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:49.443653+0000 mon.smithi062 (mon.0) 91 : audit [DBG] from='client.? 172.21.15.62:0/4226370220' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-11-15T06:25:49.755 INFO:teuthology.orchestra.run.smithi062.stderr:Waiting for the mgr to restart... 2023-11-15T06:25:49.756 INFO:teuthology.orchestra.run.smithi062.stderr:Waiting for mgr epoch 13... 2023-11-15T06:25:49.756 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi062 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptvq9ksio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpf12kfgcg:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2023-11-15T06:25:50.123 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:50.120+0000 7f8339d3e700 1 Processor -- start 2023-11-15T06:25:50.123 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:50.120+0000 7f8339d3e700 1 -- start start 2023-11-15T06:25:50.123 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:50.120+0000 7f8339d3e700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83340703b0 0x7f83340fe960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:50.124 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:50.120+0000 7f8339d3e700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83340fee50 con 0x7f83340703b0 2023-11-15T06:25:50.124 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:50.120+0000 7f83337fe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83340703b0 0x7f83340fe960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:50.124 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:50.120+0000 7f83337fe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83340703b0 0x7f83340fe960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33432/0 (socket says 172.21.15.62:33432) 2023-11-15T06:25:50.124 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:50.120+0000 7f83337fe700 1 -- 172.21.15.62:0/323943808 learned_addr learned my addr 172.21.15.62:0/323943808 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:50.124 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:50.120+0000 7f83337fe700 1 -- 172.21.15.62:0/323943808 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83340fef90 con 0x7f83340703b0 2023-11-15T06:25:50.127 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:50.121+0000 7f83337fe700 1 --2- 172.21.15.62:0/323943808 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83340703b0 0x7f83340fe960 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f832400b5f0 tx=0x7f8324006ee0).ready entity=mon.0 client_cookie=3f321f146a653d8c server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:50.128 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:50.121+0000 7f83327fc700 1 -- 172.21.15.62:0/323943808 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f832400d600 con 0x7f83340703b0 2023-11-15T06:25:50.128 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:50.121+0000 7f83327fc700 1 -- 172.21.15.62:0/323943808 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f832400dc10 con 0x7f83340703b0 2023-11-15T06:25:50.128 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:50.121+0000 7f83327fc700 1 -- 172.21.15.62:0/323943808 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8324011cd0 con 0x7f83340703b0 2023-11-15T06:25:50.128 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:50.121+0000 7f8339d3e700 1 -- 172.21.15.62:0/323943808 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83340703b0 msgr2=0x7f83340fe960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:50.129 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:50.121+0000 7f8339d3e700 1 --2- 172.21.15.62:0/323943808 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83340703b0 0x7f83340fe960 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f832400b5f0 tx=0x7f8324006ee0).stop 2023-11-15T06:25:50.129 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:50.121+0000 7f8339d3e700 1 -- 172.21.15.62:0/323943808 shutdown_connections 2023-11-15T06:25:50.129 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:50.121+0000 7f8339d3e700 1 --2- 172.21.15.62:0/323943808 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83340703b0 0x7f83340fe960 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:50.129 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:50.121+0000 7f8339d3e700 1 -- 172.21.15.62:0/323943808 >> 172.21.15.62:0/323943808 conn(0x7f83340f63c0 msgr2=0x7f83340f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:50.129 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:50.121+0000 7f8339d3e700 1 -- 172.21.15.62:0/323943808 shutdown_connections 2023-11-15T06:25:50.129 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:50.121+0000 7f8339d3e700 1 -- 172.21.15.62:0/323943808 wait complete. 2023-11-15T06:25:50.130 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:50.122+0000 7f8339d3e700 1 Processor -- start 2023-11-15T06:25:50.130 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:50.122+0000 7f8339d3e700 1 -- start start 2023-11-15T06:25:50.130 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:50.123+0000 7f8339d3e700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83340703b0 0x7f833410a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:50.130 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:50.123+0000 7f8339d3e700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833410aeb0 con 0x7f83340703b0 2023-11-15T06:25:50.130 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:50.123+0000 7f83337fe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83340703b0 0x7f833410a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:50.131 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:50.123+0000 7f83337fe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83340703b0 0x7f833410a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33444/0 (socket says 172.21.15.62:33444) 2023-11-15T06:25:50.131 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:50.123+0000 7f83337fe700 1 -- 172.21.15.62:0/274209466 learned_addr learned my addr 172.21.15.62:0/274209466 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:50.131 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:50.123+0000 7f83337fe700 1 -- 172.21.15.62:0/274209466 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f832400b200 con 0x7f83340703b0 2023-11-15T06:25:50.131 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:50.123+0000 7f83337fe700 1 --2- 172.21.15.62:0/274209466 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83340703b0 0x7f833410a9c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f832400b1d0 tx=0x7f8324006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:50.131 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:50.123+0000 7f831bfff700 1 -- 172.21.15.62:0/274209466 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8324011df0 con 0x7f83340703b0 2023-11-15T06:25:50.132 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:50.123+0000 7f8339d3e700 1 -- 172.21.15.62:0/274209466 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f833410b0b0 con 0x7f83340703b0 2023-11-15T06:25:50.132 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:50.123+0000 7f831bfff700 1 -- 172.21.15.62:0/274209466 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f832401b8c0 con 0x7f83340703b0 2023-11-15T06:25:50.132 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:50.123+0000 7f831bfff700 1 -- 172.21.15.62:0/274209466 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f832401a8a0 con 0x7f83340703b0 2023-11-15T06:25:50.132 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:50.124+0000 7f8339d3e700 1 -- 172.21.15.62:0/274209466 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f833410b4c0 con 0x7f83340703b0 2023-11-15T06:25:50.132 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:50.125+0000 7f831bfff700 1 -- 172.21.15.62:0/274209466 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f832400f070 con 0x7f83340703b0 2023-11-15T06:25:50.133 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:50.125+0000 7f831bfff700 1 --2- 172.21.15.62:0/274209466 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7f832002feb0 0x7f8320032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:50.133 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:50.125+0000 7f8339d3e700 1 -- 172.21.15.62:0/274209466 --> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8334059100 con 0x7f832002feb0 2023-11-15T06:25:50.133 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:50.125+0000 7f831bfff700 1 -- 172.21.15.62:0/274209466 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f8324040c00 con 0x7f83340703b0 2023-11-15T06:25:50.133 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:50.125+0000 7f8332ffd700 1 -- 172.21.15.62:0/274209466 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7f832002feb0 msgr2=0x7f8320032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/1948381890 2023-11-15T06:25:50.133 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:50.125+0000 7f8332ffd700 1 --2- 172.21.15.62:0/274209466 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7f832002feb0 0x7f8320032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-15T06:25:50.328 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:50.326+0000 7f8332ffd700 1 -- 172.21.15.62:0/274209466 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7f832002feb0 msgr2=0x7f8320032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/1948381890 2023-11-15T06:25:50.329 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:50.326+0000 7f8332ffd700 1 --2- 172.21.15.62:0/274209466 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7f832002feb0 0x7f8320032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-15T06:25:50.729 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:50.727+0000 7f8332ffd700 1 -- 172.21.15.62:0/274209466 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7f832002feb0 msgr2=0x7f8320032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/1948381890 2023-11-15T06:25:50.730 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:50.727+0000 7f8332ffd700 1 --2- 172.21.15.62:0/274209466 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7f832002feb0 0x7f8320032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-11-15T06:25:51.530 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:51.528+0000 7f8332ffd700 1 -- 172.21.15.62:0/274209466 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7f832002feb0 msgr2=0x7f8320032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/1948381890 2023-11-15T06:25:51.531 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:51.528+0000 7f8332ffd700 1 --2- 172.21.15.62:0/274209466 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7f832002feb0 0x7f8320032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-11-15T06:25:53.132 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:53.129+0000 7f8332ffd700 1 -- 172.21.15.62:0/274209466 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7f832002feb0 msgr2=0x7f8320032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/1948381890 2023-11-15T06:25:53.132 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:53.129+0000 7f8332ffd700 1 --2- 172.21.15.62:0/274209466 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7f832002feb0 0x7f8320032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-11-15T06:25:53.776 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:53.773+0000 7f831bfff700 1 -- 172.21.15.62:0/274209466 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mgrmap(e 14) v1 ==== 35997+0+0 (secure 0 0 0) 0x7f832401ab60 con 0x7f83340703b0 2023-11-15T06:25:53.777 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:53.773+0000 7f831bfff700 1 -- 172.21.15.62:0/274209466 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7f832002feb0 msgr2=0x7f8320032370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-15T06:25:53.777 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:53.773+0000 7f831bfff700 1 --2- 172.21.15.62:0/274209466 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7f832002feb0 0x7f8320032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T06:25:53.915 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:53 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:25:53.722418+0000 mon.smithi062 (mon.0) 92 : cluster [INF] Active manager daemon smithi062.rwurtk restarted 2023-11-15T06:25:53.916 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:53 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:25:53.722799+0000 mon.smithi062 (mon.0) 93 : cluster [INF] Activating manager daemon smithi062.rwurtk 2023-11-15T06:25:53.916 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:53 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:25:53.724963+0000 mon.smithi062 (mon.0) 94 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-11-15T06:25:54.780 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:54.779+0000 7f831bfff700 1 -- 172.21.15.62:0/274209466 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f8324041f00 con 0x7f83340703b0 2023-11-15T06:25:54.781 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:54.779+0000 7f831bfff700 1 --2- 172.21.15.62:0/274209466 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f8320033990 0x7f8320035d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:54.781 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:54.779+0000 7f831bfff700 1 -- 172.21.15.62:0/274209466 --> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8334059100 con 0x7f8320033990 2023-11-15T06:25:54.781 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:54.779+0000 7f8332ffd700 1 --2- 172.21.15.62:0/274209466 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f8320033990 0x7f8320035d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:54.781 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:54.780+0000 7f8332ffd700 1 --2- 172.21.15.62:0/274209466 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f8320033990 0x7f8320035d80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f8328006fd0 tx=0x7f8328006cf0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:54.782 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:54.780+0000 7f831bfff700 1 -- 172.21.15.62:0/274209466 <== mgr.14164 v2:172.21.15.62:6800/2759863993 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f8334059100 con 0x7f8320033990 2023-11-15T06:25:54.790 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:54.787+0000 7f8339d3e700 1 -- 172.21.15.62:0/274209466 --> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f8334045ad0 con 0x7f8320033990 2023-11-15T06:25:54.790 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:54.787+0000 7f831bfff700 1 -- 172.21.15.62:0/274209466 <== mgr.14164 v2:172.21.15.62:6800/2759863993 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f8334045ad0 con 0x7f8320033990 2023-11-15T06:25:54.790 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: { 2023-11-15T06:25:54.791 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "mgrmap_epoch": 15, 2023-11-15T06:25:54.791 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "initialized": true 2023-11-15T06:25:54.791 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: } 2023-11-15T06:25:54.791 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:54.788+0000 7f8339d3e700 1 -- 172.21.15.62:0/274209466 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f8320033990 msgr2=0x7f8320035d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:54.791 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:54.788+0000 7f8339d3e700 1 --2- 172.21.15.62:0/274209466 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f8320033990 0x7f8320035d80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f8328006fd0 tx=0x7f8328006cf0).stop 2023-11-15T06:25:54.792 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:54.788+0000 7f8339d3e700 1 -- 172.21.15.62:0/274209466 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83340703b0 msgr2=0x7f833410a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:54.792 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:54.788+0000 7f8339d3e700 1 --2- 172.21.15.62:0/274209466 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83340703b0 0x7f833410a9c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f832400b1d0 tx=0x7f8324006ee0).stop 2023-11-15T06:25:54.792 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:54.788+0000 7f8339d3e700 1 -- 172.21.15.62:0/274209466 shutdown_connections 2023-11-15T06:25:54.792 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:54.788+0000 7f8339d3e700 1 --2- 172.21.15.62:0/274209466 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f8320033990 0x7f8320035d80 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:54.793 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:54.788+0000 7f8339d3e700 1 --2- 172.21.15.62:0/274209466 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83340703b0 0x7f833410a9c0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:54.793 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:54.788+0000 7f8339d3e700 1 --2- 172.21.15.62:0/274209466 >> [v2:172.21.15.62:6800/1948381890,v1:172.21.15.62:6801/1948381890] conn(0x7f832002feb0 0x7f8320032370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T06:25:54.793 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:54.788+0000 7f8339d3e700 1 -- 172.21.15.62:0/274209466 >> 172.21.15.62:0/274209466 conn(0x7f83340f63c0 msgr2=0x7f83340f7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:54.794 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:54.788+0000 7f8339d3e700 1 -- 172.21.15.62:0/274209466 shutdown_connections 2023-11-15T06:25:54.794 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:54.788+0000 7f8339d3e700 1 -- 172.21.15.62:0/274209466 wait complete. 2023-11-15T06:25:54.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:54 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:25:53.775634+0000 mon.smithi062 (mon.0) 95 : cluster [DBG] mgrmap e14: smithi062.rwurtk(active, starting, since 0.0529241s) 2023-11-15T06:25:54.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:54 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:53.779108+0000 mon.smithi062 (mon.0) 96 : audit [DBG] from='mgr.14164 172.21.15.62:0/147479783' entity='mgr.smithi062.rwurtk' cmd=[{"prefix": "mon metadata", "id": "smithi062"}]: dispatch 2023-11-15T06:25:54.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:54 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:53.779312+0000 mon.smithi062 (mon.0) 97 : audit [DBG] from='mgr.14164 172.21.15.62:0/147479783' entity='mgr.smithi062.rwurtk' cmd=[{"prefix": "mgr metadata", "who": "smithi062.rwurtk", "id": "smithi062.rwurtk"}]: dispatch 2023-11-15T06:25:54.919 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:54 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:53.779496+0000 mon.smithi062 (mon.0) 98 : audit [DBG] from='mgr.14164 172.21.15.62:0/147479783' entity='mgr.smithi062.rwurtk' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-15T06:25:54.919 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:54 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:53.779641+0000 mon.smithi062 (mon.0) 99 : audit [DBG] from='mgr.14164 172.21.15.62:0/147479783' entity='mgr.smithi062.rwurtk' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-15T06:25:54.919 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:54 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:53.779760+0000 mon.smithi062 (mon.0) 100 : audit [DBG] from='mgr.14164 172.21.15.62:0/147479783' entity='mgr.smithi062.rwurtk' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-15T06:25:54.919 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:54 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:25:53.781626+0000 mon.smithi062 (mon.0) 101 : cluster [INF] Manager daemon smithi062.rwurtk is now available 2023-11-15T06:25:54.919 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:54 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:53.885934+0000 mon.smithi062 (mon.0) 102 : audit [DBG] from='mgr.14164 172.21.15.62:0/147479783' entity='mgr.smithi062.rwurtk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T06:25:54.920 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:54 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:53.887491+0000 mon.smithi062 (mon.0) 103 : audit [DBG] from='mgr.14164 172.21.15.62:0/147479783' entity='mgr.smithi062.rwurtk' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T06:25:54.920 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:54 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:53.911980+0000 mon.smithi062 (mon.0) 104 : audit [INF] from='mgr.14164 172.21.15.62:0/147479783' entity='mgr.smithi062.rwurtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi062.rwurtk/mirror_snapshot_schedule"}]: dispatch 2023-11-15T06:25:54.920 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:54 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:53.915185+0000 mon.smithi062 (mon.0) 105 : audit [INF] from='mgr.14164 172.21.15.62:0/147479783' entity='mgr.smithi062.rwurtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi062.rwurtk/trash_purge_schedule"}]: dispatch 2023-11-15T06:25:54.920 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:54 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:54.727037+0000 mon.smithi062 (mon.0) 106 : audit [INF] from='mgr.14164 172.21.15.62:0/147479783' entity='mgr.smithi062.rwurtk' 2023-11-15T06:25:55.181 INFO:teuthology.orchestra.run.smithi062.stderr:mgr epoch 13 is available 2023-11-15T06:25:55.181 INFO:teuthology.orchestra.run.smithi062.stderr:Generating a dashboard self-signed certificate... 2023-11-15T06:25:55.182 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi062 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptvq9ksio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpf12kfgcg:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2023-11-15T06:25:55.690 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.687+0000 7f33cf59e700 1 Processor -- start 2023-11-15T06:25:55.690 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.687+0000 7f33cf59e700 1 -- start start 2023-11-15T06:25:55.690 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.687+0000 7f33cf59e700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f33d0070420 0x7f33d0100bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:55.691 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.687+0000 7f33cf59e700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d00fc5c0 con 0x7f33d0070420 2023-11-15T06:25:55.691 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.688+0000 7f33ce59c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f33d0070420 0x7f33d0100bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:55.691 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.688+0000 7f33ce59c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f33d0070420 0x7f33d0100bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33484/0 (socket says 172.21.15.62:33484) 2023-11-15T06:25:55.691 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.688+0000 7f33ce59c700 1 -- 172.21.15.62:0/2524631524 learned_addr learned my addr 172.21.15.62:0/2524631524 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:55.691 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.688+0000 7f33ce59c700 1 -- 172.21.15.62:0/2524631524 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33d01010c0 con 0x7f33d0070420 2023-11-15T06:25:55.692 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.688+0000 7f33ce59c700 1 --2- 172.21.15.62:0/2524631524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f33d0070420 0x7f33d0100bd0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f33c0009960 tx=0x7f33c00092d0).ready entity=mon.0 client_cookie=3db15c1a1766f1be server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:55.692 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.688+0000 7f33cd59a700 1 -- 172.21.15.62:0/2524631524 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33c000b9c0 con 0x7f33d0070420 2023-11-15T06:25:55.692 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.688+0000 7f33cd59a700 1 -- 172.21.15.62:0/2524631524 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f33c0011400 con 0x7f33d0070420 2023-11-15T06:25:55.692 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.688+0000 7f33cd59a700 1 -- 172.21.15.62:0/2524631524 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33c0010580 con 0x7f33d0070420 2023-11-15T06:25:55.692 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.689+0000 7f33cf59e700 1 -- 172.21.15.62:0/2524631524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f33d0070420 msgr2=0x7f33d0100bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:55.693 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.689+0000 7f33cf59e700 1 --2- 172.21.15.62:0/2524631524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f33d0070420 0x7f33d0100bd0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f33c0009960 tx=0x7f33c00092d0).stop 2023-11-15T06:25:55.695 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.689+0000 7f33cf59e700 1 -- 172.21.15.62:0/2524631524 shutdown_connections 2023-11-15T06:25:55.695 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.689+0000 7f33cf59e700 1 --2- 172.21.15.62:0/2524631524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f33d0070420 0x7f33d0100bd0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:55.695 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.689+0000 7f33cf59e700 1 -- 172.21.15.62:0/2524631524 >> 172.21.15.62:0/2524631524 conn(0x7f33d00f6250 msgr2=0x7f33d00f8670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:55.695 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.689+0000 7f33cf59e700 1 -- 172.21.15.62:0/2524631524 shutdown_connections 2023-11-15T06:25:55.696 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.689+0000 7f33cf59e700 1 -- 172.21.15.62:0/2524631524 wait complete. 2023-11-15T06:25:55.696 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.689+0000 7f33cf59e700 1 Processor -- start 2023-11-15T06:25:55.696 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.690+0000 7f33cf59e700 1 -- start start 2023-11-15T06:25:55.696 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.690+0000 7f33cf59e700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f33d0070420 0x7f33d010caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:55.696 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.690+0000 7f33cf59e700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d010cfe0 con 0x7f33d0070420 2023-11-15T06:25:55.697 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.690+0000 7f33ce59c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f33d0070420 0x7f33d010caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:55.697 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.690+0000 7f33ce59c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f33d0070420 0x7f33d010caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33486/0 (socket says 172.21.15.62:33486) 2023-11-15T06:25:55.697 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.690+0000 7f33ce59c700 1 -- 172.21.15.62:0/1823088629 learned_addr learned my addr 172.21.15.62:0/1823088629 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:55.697 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.690+0000 7f33ce59c700 1 -- 172.21.15.62:0/1823088629 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c0009570 con 0x7f33d0070420 2023-11-15T06:25:55.697 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.690+0000 7f33ce59c700 1 --2- 172.21.15.62:0/1823088629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f33d0070420 0x7f33d010caf0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f33c000ba90 tx=0x7f33c00092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:55.698 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.690+0000 7f33c6ffd700 1 -- 172.21.15.62:0/1823088629 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33c00103a0 con 0x7f33d0070420 2023-11-15T06:25:55.698 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.690+0000 7f33c6ffd700 1 -- 172.21.15.62:0/1823088629 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f33c0010eb0 con 0x7f33d0070420 2023-11-15T06:25:55.698 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.690+0000 7f33cf59e700 1 -- 172.21.15.62:0/1823088629 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33d010d1e0 con 0x7f33d0070420 2023-11-15T06:25:55.698 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.690+0000 7f33c6ffd700 1 -- 172.21.15.62:0/1823088629 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33c001bc30 con 0x7f33d0070420 2023-11-15T06:25:55.698 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.690+0000 7f33cf59e700 1 -- 172.21.15.62:0/1823088629 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33d010d5f0 con 0x7f33d0070420 2023-11-15T06:25:55.698 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.691+0000 7f33c6ffd700 1 -- 172.21.15.62:0/1823088629 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f33c000f070 con 0x7f33d0070420 2023-11-15T06:25:55.699 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.691+0000 7f33c6ffd700 1 --2- 172.21.15.62:0/1823088629 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f33b002fdc0 0x7f33b0032280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:55.699 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.691+0000 7f33c6ffd700 1 -- 172.21.15.62:0/1823088629 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f33c0041c50 con 0x7f33d0070420 2023-11-15T06:25:55.699 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.691+0000 7f33cf59e700 1 -- 172.21.15.62:0/1823088629 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33d0045ad0 con 0x7f33d0070420 2023-11-15T06:25:55.699 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.691+0000 7f33cdd9b700 1 --2- 172.21.15.62:0/1823088629 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f33b002fdc0 0x7f33b0032280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:55.699 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.692+0000 7f33cdd9b700 1 --2- 172.21.15.62:0/1823088629 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f33b002fdc0 0x7f33b0032280 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f33b80096a0 tx=0x7f33b8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:55.700 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.696+0000 7f33c6ffd700 1 -- 172.21.15.62:0/1823088629 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f33c0019920 con 0x7f33d0070420 2023-11-15T06:25:55.787 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.785+0000 7f33c6ffd700 1 -- 172.21.15.62:0/1823088629 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f33c0025460 con 0x7f33d0070420 2023-11-15T06:25:55.864 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.862+0000 7f33cf59e700 1 -- 172.21.15.62:0/1823088629 --> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f33d01069f0 con 0x7f33b002fdc0 2023-11-15T06:25:55.981 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.980+0000 7f33c6ffd700 1 -- 172.21.15.62:0/1823088629 <== mgr.14164 v2:172.21.15.62:6800/2759863993 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f33d01069f0 con 0x7f33b002fdc0 2023-11-15T06:25:55.982 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: Self-signed certificate created 2023-11-15T06:25:55.984 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.982+0000 7f33cf59e700 1 -- 172.21.15.62:0/1823088629 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f33b002fdc0 msgr2=0x7f33b0032280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:55.984 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.982+0000 7f33cf59e700 1 --2- 172.21.15.62:0/1823088629 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f33b002fdc0 0x7f33b0032280 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f33b80096a0 tx=0x7f33b8006b40).stop 2023-11-15T06:25:55.984 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.982+0000 7f33cf59e700 1 -- 172.21.15.62:0/1823088629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f33d0070420 msgr2=0x7f33d010caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:55.984 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.982+0000 7f33cf59e700 1 --2- 172.21.15.62:0/1823088629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f33d0070420 0x7f33d010caf0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f33c000ba90 tx=0x7f33c00092d0).stop 2023-11-15T06:25:55.985 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.982+0000 7f33cf59e700 1 -- 172.21.15.62:0/1823088629 shutdown_connections 2023-11-15T06:25:55.985 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.982+0000 7f33cf59e700 1 --2- 172.21.15.62:0/1823088629 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f33b002fdc0 0x7f33b0032280 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:55.985 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.982+0000 7f33cf59e700 1 --2- 172.21.15.62:0/1823088629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f33d0070420 0x7f33d010caf0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:55.985 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.982+0000 7f33cf59e700 1 -- 172.21.15.62:0/1823088629 >> 172.21.15.62:0/1823088629 conn(0x7f33d00f6250 msgr2=0x7f33d00f7d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:55.985 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.982+0000 7f33cf59e700 1 -- 172.21.15.62:0/1823088629 shutdown_connections 2023-11-15T06:25:55.986 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:55.982+0000 7f33cf59e700 1 -- 172.21.15.62:0/1823088629 wait complete. 2023-11-15T06:25:56.011 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:55 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:54.780699+0000 mgr.smithi062.rwurtk (mgr.14164) 1 : audit [DBG] from='client.14168 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-11-15T06:25:56.011 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:55 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:25:54.781191+0000 mon.smithi062 (mon.0) 107 : cluster [DBG] mgrmap e15: smithi062.rwurtk(active, since 1.05848s) 2023-11-15T06:25:56.012 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:55 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:54.788178+0000 mgr.smithi062.rwurtk (mgr.14164) 2 : audit [DBG] from='client.14168 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-11-15T06:25:56.321 INFO:teuthology.orchestra.run.smithi062.stderr:Creating initial admin user... 2023-11-15T06:25:56.322 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi062 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptvq9ksio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpf12kfgcg:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpct41wpng:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2023-11-15T06:25:57.167 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:56 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:25:55.787051+0000 mon.smithi062 (mon.0) 108 : cluster [DBG] mgrmap e16: smithi062.rwurtk(active, since 2s) 2023-11-15T06:25:57.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:56 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:55.863733+0000 mgr.smithi062.rwurtk (mgr.14164) 3 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T06:25:57.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:56 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:55.978588+0000 mon.smithi062 (mon.0) 109 : audit [INF] from='mgr.14164 172.21.15.62:0/147479783' entity='mgr.smithi062.rwurtk' 2023-11-15T06:25:57.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:56 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:55.980514+0000 mon.smithi062 (mon.0) 110 : audit [INF] from='mgr.14164 172.21.15.62:0/147479783' entity='mgr.smithi062.rwurtk' 2023-11-15T06:25:57.178 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.176+0000 7f2b8759e700 1 Processor -- start 2023-11-15T06:25:57.178 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.176+0000 7f2b8759e700 1 -- start start 2023-11-15T06:25:57.178 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.176+0000 7f2b8759e700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b880703b0 0x7f2b88100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:57.178 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.176+0000 7f2b8759e700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b880fc5a0 con 0x7f2b880703b0 2023-11-15T06:25:57.179 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.176+0000 7f2b8659c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b880703b0 0x7f2b88100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:57.179 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.176+0000 7f2b8659c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b880703b0 0x7f2b88100bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33492/0 (socket says 172.21.15.62:33492) 2023-11-15T06:25:57.179 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.176+0000 7f2b8659c700 1 -- 172.21.15.62:0/2466302081 learned_addr learned my addr 172.21.15.62:0/2466302081 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:57.182 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.177+0000 7f2b8659c700 1 -- 172.21.15.62:0/2466302081 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b881010a0 con 0x7f2b880703b0 2023-11-15T06:25:57.183 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.177+0000 7f2b8659c700 1 --2- 172.21.15.62:0/2466302081 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b880703b0 0x7f2b88100bb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f2b7800b5f0 tx=0x7f2b78006ee0).ready entity=mon.0 client_cookie=45f7ce868851d4bf server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:57.183 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.177+0000 7f2b8559a700 1 -- 172.21.15.62:0/2466302081 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b7800d600 con 0x7f2b880703b0 2023-11-15T06:25:57.183 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.177+0000 7f2b8559a700 1 -- 172.21.15.62:0/2466302081 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2b7800dc10 con 0x7f2b880703b0 2023-11-15T06:25:57.183 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.177+0000 7f2b8559a700 1 -- 172.21.15.62:0/2466302081 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b78011cd0 con 0x7f2b880703b0 2023-11-15T06:25:57.183 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.178+0000 7f2b8759e700 1 -- 172.21.15.62:0/2466302081 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b880703b0 msgr2=0x7f2b88100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:57.184 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.178+0000 7f2b8759e700 1 --2- 172.21.15.62:0/2466302081 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b880703b0 0x7f2b88100bb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f2b7800b5f0 tx=0x7f2b78006ee0).stop 2023-11-15T06:25:57.184 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.178+0000 7f2b8759e700 1 -- 172.21.15.62:0/2466302081 shutdown_connections 2023-11-15T06:25:57.184 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.178+0000 7f2b8759e700 1 --2- 172.21.15.62:0/2466302081 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b880703b0 0x7f2b88100bb0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:57.184 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.178+0000 7f2b8759e700 1 -- 172.21.15.62:0/2466302081 >> 172.21.15.62:0/2466302081 conn(0x7f2b880f6210 msgr2=0x7f2b880f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:57.184 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.178+0000 7f2b8759e700 1 -- 172.21.15.62:0/2466302081 shutdown_connections 2023-11-15T06:25:57.185 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.178+0000 7f2b8759e700 1 -- 172.21.15.62:0/2466302081 wait complete. 2023-11-15T06:25:57.185 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.178+0000 7f2b8759e700 1 Processor -- start 2023-11-15T06:25:57.185 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.179+0000 7f2b8759e700 1 -- start start 2023-11-15T06:25:57.185 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.179+0000 7f2b8759e700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b880703b0 0x7f2b8810cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:57.185 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.179+0000 7f2b8759e700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8810cfc0 con 0x7f2b880703b0 2023-11-15T06:25:57.186 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.179+0000 7f2b8659c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b880703b0 0x7f2b8810cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:57.186 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.179+0000 7f2b8659c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b880703b0 0x7f2b8810cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33506/0 (socket says 172.21.15.62:33506) 2023-11-15T06:25:57.186 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.179+0000 7f2b8659c700 1 -- 172.21.15.62:0/2647047174 learned_addr learned my addr 172.21.15.62:0/2647047174 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:57.187 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.179+0000 7f2b8659c700 1 -- 172.21.15.62:0/2647047174 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b7800b200 con 0x7f2b880703b0 2023-11-15T06:25:57.187 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.179+0000 7f2b8659c700 1 --2- 172.21.15.62:0/2647047174 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b880703b0 0x7f2b8810cad0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f2b7800dfb0 tx=0x7f2b78006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:57.187 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.179+0000 7f2b7effd700 1 -- 172.21.15.62:0/2647047174 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b7801b410 con 0x7f2b880703b0 2023-11-15T06:25:57.187 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.180+0000 7f2b8759e700 1 -- 172.21.15.62:0/2647047174 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b8810d1c0 con 0x7f2b880703b0 2023-11-15T06:25:57.187 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.180+0000 7f2b7effd700 1 -- 172.21.15.62:0/2647047174 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2b7801ba20 con 0x7f2b880703b0 2023-11-15T06:25:57.188 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.180+0000 7f2b7effd700 1 -- 172.21.15.62:0/2647047174 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b7801a970 con 0x7f2b880703b0 2023-11-15T06:25:57.188 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.180+0000 7f2b8759e700 1 -- 172.21.15.62:0/2647047174 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b8810d5d0 con 0x7f2b880703b0 2023-11-15T06:25:57.188 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.181+0000 7f2b7effd700 1 -- 172.21.15.62:0/2647047174 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f2b7800f070 con 0x7f2b880703b0 2023-11-15T06:25:57.188 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.181+0000 7f2b7effd700 1 --2- 172.21.15.62:0/2647047174 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f2b6802feb0 0x7f2b68032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:57.188 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.181+0000 7f2b7effd700 1 -- 172.21.15.62:0/2647047174 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2b780411e0 con 0x7f2b880703b0 2023-11-15T06:25:57.189 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.181+0000 7f2b8759e700 1 -- 172.21.15.62:0/2647047174 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b88045ad0 con 0x7f2b880703b0 2023-11-15T06:25:57.189 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.181+0000 7f2b85d9b700 1 --2- 172.21.15.62:0/2647047174 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f2b6802feb0 0x7f2b68032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:57.189 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.181+0000 7f2b85d9b700 1 --2- 172.21.15.62:0/2647047174 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f2b6802feb0 0x7f2b68032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f2b700096a0 tx=0x7f2b70006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:57.190 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.185+0000 7f2b7effd700 1 -- 172.21.15.62:0/2647047174 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2b78016070 con 0x7f2b880703b0 2023-11-15T06:25:57.360 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.358+0000 7f2b8759e700 1 -- 172.21.15.62:0/2647047174 --> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f2b88101d30 con 0x7f2b6802feb0 2023-11-15T06:25:57.606 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.604+0000 7f2b7effd700 1 -- 172.21.15.62:0/2647047174 <== mgr.14164 v2:172.21.15.62:6800/2759863993 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f2b88101d30 con 0x7f2b6802feb0 2023-11-15T06:25:57.606 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$tQIs6IbSf.lFQUEnKQT1XuXM0jXZD.x0s4GTMVvWFhSbnShWphgS2", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1700029557, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-11-15T06:25:57.608 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.606+0000 7f2b8759e700 1 -- 172.21.15.62:0/2647047174 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f2b6802feb0 msgr2=0x7f2b68032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:57.609 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.606+0000 7f2b8759e700 1 --2- 172.21.15.62:0/2647047174 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f2b6802feb0 0x7f2b68032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f2b700096a0 tx=0x7f2b70006b40).stop 2023-11-15T06:25:57.609 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.606+0000 7f2b8759e700 1 -- 172.21.15.62:0/2647047174 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b880703b0 msgr2=0x7f2b8810cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:57.609 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.606+0000 7f2b8759e700 1 --2- 172.21.15.62:0/2647047174 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b880703b0 0x7f2b8810cad0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f2b7800dfb0 tx=0x7f2b78006ee0).stop 2023-11-15T06:25:57.609 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.606+0000 7f2b8759e700 1 -- 172.21.15.62:0/2647047174 shutdown_connections 2023-11-15T06:25:57.609 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.606+0000 7f2b8759e700 1 --2- 172.21.15.62:0/2647047174 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f2b6802feb0 0x7f2b68032370 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:57.610 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.606+0000 7f2b8759e700 1 --2- 172.21.15.62:0/2647047174 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b880703b0 0x7f2b8810cad0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:57.610 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.606+0000 7f2b8759e700 1 -- 172.21.15.62:0/2647047174 >> 172.21.15.62:0/2647047174 conn(0x7f2b880f6210 msgr2=0x7f2b880f7db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:57.610 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.607+0000 7f2b8759e700 1 -- 172.21.15.62:0/2647047174 shutdown_connections 2023-11-15T06:25:57.610 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:57.607+0000 7f2b8759e700 1 -- 172.21.15.62:0/2647047174 wait complete. 2023-11-15T06:25:57.917 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:57 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:57.359950+0000 mgr.smithi062.rwurtk (mgr.14164) 4 : audit [DBG] from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T06:25:57.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:57 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:57.604834+0000 mon.smithi062 (mon.0) 111 : audit [INF] from='mgr.14164 172.21.15.62:0/147479783' entity='mgr.smithi062.rwurtk' 2023-11-15T06:25:57.971 INFO:teuthology.orchestra.run.smithi062.stderr:Fetching dashboard port number... 2023-11-15T06:25:57.971 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi062 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptvq9ksio:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpf12kfgcg:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2023-11-15T06:25:58.314 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.312+0000 7f3754db0700 1 Processor -- start 2023-11-15T06:25:58.315 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.312+0000 7f3754db0700 1 -- start start 2023-11-15T06:25:58.315 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.312+0000 7f3754db0700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37500703b0 0x7f3750100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:58.315 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.312+0000 7f3754db0700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37500fc750 con 0x7f37500703b0 2023-11-15T06:25:58.315 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.313+0000 7f374e59c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37500703b0 0x7f3750100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:58.316 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.313+0000 7f374e59c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37500703b0 0x7f3750100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33514/0 (socket says 172.21.15.62:33514) 2023-11-15T06:25:58.316 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.313+0000 7f374e59c700 1 -- 172.21.15.62:0/4091647158 learned_addr learned my addr 172.21.15.62:0/4091647158 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:58.316 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.313+0000 7f374e59c700 1 -- 172.21.15.62:0/4091647158 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3750101250 con 0x7f37500703b0 2023-11-15T06:25:58.316 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.313+0000 7f374e59c700 1 --2- 172.21.15.62:0/4091647158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37500703b0 0x7f3750100d60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f374400b5f0 tx=0x7f3744006ee0).ready entity=mon.0 client_cookie=c89b642d81319bea server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:58.316 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.313+0000 7f374d59a700 1 -- 172.21.15.62:0/4091647158 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f374400d600 con 0x7f37500703b0 2023-11-15T06:25:58.319 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.313+0000 7f374d59a700 1 -- 172.21.15.62:0/4091647158 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f374400dc10 con 0x7f37500703b0 2023-11-15T06:25:58.319 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.313+0000 7f374d59a700 1 -- 172.21.15.62:0/4091647158 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3744011cd0 con 0x7f37500703b0 2023-11-15T06:25:58.320 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.314+0000 7f3754db0700 1 -- 172.21.15.62:0/4091647158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37500703b0 msgr2=0x7f3750100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:58.320 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.314+0000 7f3754db0700 1 --2- 172.21.15.62:0/4091647158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37500703b0 0x7f3750100d60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f374400b5f0 tx=0x7f3744006ee0).stop 2023-11-15T06:25:58.320 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.314+0000 7f3754db0700 1 -- 172.21.15.62:0/4091647158 shutdown_connections 2023-11-15T06:25:58.320 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.314+0000 7f3754db0700 1 --2- 172.21.15.62:0/4091647158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37500703b0 0x7f3750100d60 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:58.321 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.314+0000 7f3754db0700 1 -- 172.21.15.62:0/4091647158 >> 172.21.15.62:0/4091647158 conn(0x7f37500f63c0 msgr2=0x7f37500f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:58.321 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.314+0000 7f3754db0700 1 -- 172.21.15.62:0/4091647158 shutdown_connections 2023-11-15T06:25:58.321 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.314+0000 7f3754db0700 1 -- 172.21.15.62:0/4091647158 wait complete. 2023-11-15T06:25:58.321 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.315+0000 7f3754db0700 1 Processor -- start 2023-11-15T06:25:58.321 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.315+0000 7f3754db0700 1 -- start start 2023-11-15T06:25:58.321 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.315+0000 7f3754db0700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37500703b0 0x7f375010cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:58.322 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.315+0000 7f3754db0700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f375010d230 con 0x7f37500703b0 2023-11-15T06:25:58.322 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.315+0000 7f374e59c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37500703b0 0x7f375010cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:58.322 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.315+0000 7f374e59c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37500703b0 0x7f375010cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33516/0 (socket says 172.21.15.62:33516) 2023-11-15T06:25:58.322 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.315+0000 7f374e59c700 1 -- 172.21.15.62:0/180688099 learned_addr learned my addr 172.21.15.62:0/180688099 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:25:58.322 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.315+0000 7f374e59c700 1 -- 172.21.15.62:0/180688099 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f374400b200 con 0x7f37500703b0 2023-11-15T06:25:58.323 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.315+0000 7f374e59c700 1 --2- 172.21.15.62:0/180688099 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37500703b0 0x7f375010cd40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f374400dfb0 tx=0x7f3744006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:58.323 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.316+0000 7f373effd700 1 -- 172.21.15.62:0/180688099 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f374401b410 con 0x7f37500703b0 2023-11-15T06:25:58.323 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.316+0000 7f373effd700 1 -- 172.21.15.62:0/180688099 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f374401ba20 con 0x7f37500703b0 2023-11-15T06:25:58.323 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.316+0000 7f3754db0700 1 -- 172.21.15.62:0/180688099 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f375010d430 con 0x7f37500703b0 2023-11-15T06:25:58.323 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.316+0000 7f373effd700 1 -- 172.21.15.62:0/180688099 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f374401a970 con 0x7f37500703b0 2023-11-15T06:25:58.324 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.316+0000 7f3754db0700 1 -- 172.21.15.62:0/180688099 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f375010d840 con 0x7f37500703b0 2023-11-15T06:25:58.324 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.316+0000 7f373effd700 1 -- 172.21.15.62:0/180688099 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f374400f070 con 0x7f37500703b0 2023-11-15T06:25:58.324 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.317+0000 7f373effd700 1 --2- 172.21.15.62:0/180688099 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f373402ff00 0x7f37340323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:25:58.324 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.317+0000 7f3754db0700 1 -- 172.21.15.62:0/180688099 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3750045ad0 con 0x7f37500703b0 2023-11-15T06:25:58.324 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.317+0000 7f373effd700 1 -- 172.21.15.62:0/180688099 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3744041110 con 0x7f37500703b0 2023-11-15T06:25:58.325 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.317+0000 7f374dd9b700 1 --2- 172.21.15.62:0/180688099 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f373402ff00 0x7f37340323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:25:58.325 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.317+0000 7f374dd9b700 1 --2- 172.21.15.62:0/180688099 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f373402ff00 0x7f37340323c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f37400096a0 tx=0x7f3740006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:25:58.325 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.321+0000 7f373effd700 1 -- 172.21.15.62:0/180688099 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3744016070 con 0x7f37500703b0 2023-11-15T06:25:58.479 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.477+0000 7f3754db0700 1 -- 172.21.15.62:0/180688099 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f3750101e20 con 0x7f37500703b0 2023-11-15T06:25:58.479 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.478+0000 7f373effd700 1 -- 172.21.15.62:0/180688099 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v10) v1 ==== 112+0+5 (secure 0 0 0) 0x7f3744019cd0 con 0x7f37500703b0 2023-11-15T06:25:58.480 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 8443 2023-11-15T06:25:58.481 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.479+0000 7f3754db0700 1 -- 172.21.15.62:0/180688099 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f373402ff00 msgr2=0x7f37340323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:58.482 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.479+0000 7f3754db0700 1 --2- 172.21.15.62:0/180688099 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f373402ff00 0x7f37340323c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f37400096a0 tx=0x7f3740006b40).stop 2023-11-15T06:25:58.482 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.479+0000 7f3754db0700 1 -- 172.21.15.62:0/180688099 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37500703b0 msgr2=0x7f375010cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:25:58.482 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.479+0000 7f3754db0700 1 --2- 172.21.15.62:0/180688099 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37500703b0 0x7f375010cd40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f374400dfb0 tx=0x7f3744006ee0).stop 2023-11-15T06:25:58.482 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.479+0000 7f3754db0700 1 -- 172.21.15.62:0/180688099 shutdown_connections 2023-11-15T06:25:58.483 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.479+0000 7f3754db0700 1 --2- 172.21.15.62:0/180688099 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f373402ff00 0x7f37340323c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:58.483 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.479+0000 7f3754db0700 1 --2- 172.21.15.62:0/180688099 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37500703b0 0x7f375010cd40 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:25:58.483 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.480+0000 7f3754db0700 1 -- 172.21.15.62:0/180688099 >> 172.21.15.62:0/180688099 conn(0x7f37500f63c0 msgr2=0x7f37500f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:25:58.483 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.480+0000 7f3754db0700 1 -- 172.21.15.62:0/180688099 shutdown_connections 2023-11-15T06:25:58.483 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2023-11-15T06:25:58.480+0000 7f3754db0700 1 -- 172.21.15.62:0/180688099 wait complete. 2023-11-15T06:25:58.822 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: systemctl is-enabled firewalld.service 2023-11-15T06:25:58.830 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: disabled 2023-11-15T06:25:58.831 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: systemctl is-active firewalld.service 2023-11-15T06:25:58.838 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: inactive 2023-11-15T06:25:58.839 INFO:teuthology.orchestra.run.smithi062.stderr:firewalld.service is not enabled 2023-11-15T06:25:58.839 INFO:teuthology.orchestra.run.smithi062.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2023-11-15T06:25:58.842 INFO:teuthology.orchestra.run.smithi062.stderr:Ceph Dashboard is now available at: 2023-11-15T06:25:58.842 INFO:teuthology.orchestra.run.smithi062.stderr: 2023-11-15T06:25:58.843 INFO:teuthology.orchestra.run.smithi062.stderr: URL: https://smithi062.front.sepia.ceph.com:8443/ 2023-11-15T06:25:58.843 INFO:teuthology.orchestra.run.smithi062.stderr: User: admin 2023-11-15T06:25:58.843 INFO:teuthology.orchestra.run.smithi062.stderr: Password: qk39wwn3xs 2023-11-15T06:25:58.844 INFO:teuthology.orchestra.run.smithi062.stderr: 2023-11-15T06:25:58.844 INFO:teuthology.orchestra.run.smithi062.stderr:You can access the Ceph CLI with: 2023-11-15T06:25:58.844 INFO:teuthology.orchestra.run.smithi062.stderr: 2023-11-15T06:25:58.845 INFO:teuthology.orchestra.run.smithi062.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-11-15T06:25:58.845 INFO:teuthology.orchestra.run.smithi062.stderr: 2023-11-15T06:25:58.846 INFO:teuthology.orchestra.run.smithi062.stderr:Please consider enabling telemetry to help improve Ceph: 2023-11-15T06:25:58.846 INFO:teuthology.orchestra.run.smithi062.stderr: 2023-11-15T06:25:58.846 INFO:teuthology.orchestra.run.smithi062.stderr: ceph telemetry on 2023-11-15T06:25:58.847 INFO:teuthology.orchestra.run.smithi062.stderr: 2023-11-15T06:25:58.847 INFO:teuthology.orchestra.run.smithi062.stderr:For more information see: 2023-11-15T06:25:58.847 INFO:teuthology.orchestra.run.smithi062.stderr: 2023-11-15T06:25:58.847 INFO:teuthology.orchestra.run.smithi062.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2023-11-15T06:25:58.848 INFO:teuthology.orchestra.run.smithi062.stderr: 2023-11-15T06:25:58.848 INFO:teuthology.orchestra.run.smithi062.stderr:Bootstrap complete. 2023-11-15T06:25:58.848 INFO:teuthology.orchestra.run.smithi062.stderr:Releasing lock 140428132735464 on /run/cephadm/a3c2da76-837f-11ee-8dbe-212e2dc638e7.lock 2023-11-15T06:25:58.848 INFO:teuthology.orchestra.run.smithi062.stderr:Lock 140428132735464 released on /run/cephadm/a3c2da76-837f-11ee-8dbe-212e2dc638e7.lock 2023-11-15T06:25:58.879 INFO:tasks.cephadm:Fetching config... 2023-11-15T06:25:58.880 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-11-15T06:25:58.880 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-11-15T06:25:58.941 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-11-15T06:25:58.941 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-11-15T06:25:58.942 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-11-15T06:25:59.002 INFO:tasks.cephadm:Fetching mon keyring... 2023-11-15T06:25:59.003 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-11-15T06:25:59.003 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/mon.smithi062/keyring of=/dev/stdout 2023-11-15T06:25:59.033 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:25:58 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:25:58.478391+0000 mon.smithi062 (mon.0) 112 : audit [DBG] from='client.? 172.21.15.62:0/180688099' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-11-15T06:25:59.064 INFO:tasks.cephadm:Fetching pub ssh key... 2023-11-15T06:25:59.064 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2023-11-15T06:25:59.065 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-11-15T06:25:59.122 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-11-15T06:25:59.123 DEBUG:teuthology.orchestra.run.smithi062:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5+aqffi9x3Pq9kjlnQYPUXA/1lo4stG9BltXRno7qEJ/+OWtYt5Wmb9jCJ33mRCfVPPj1QKZDTHse1niDh1wX7rkYZocmeUMS5P7xr5YA/DP5sqqP87WsOjWBMiRDK5yUWlD2T7bVUO8QhLGHJFYfhrARhnqUxnhFwF2jkKao9Z1pB+PG8bTmv+S8nio8EfDEwOzLtuSjo8Q9ZLekPhc4UpWuUIqB0xLQxwSIVst/0EE2sYME+DfXCZMiYSEZuvf1Fs0cEOXdfR6JpR10YR3HIhvQ2h7P8csIq842USQCOsWhIXRh8/L8S2wiprO0vb9tnZ96tiX7sj1YrKLoMlgnLc7BbHlBOwceF999ljHiHPGG1Xhd8X9Gp6kMoXIpcBzcN3kkeW/EeLsDp+v3YJlU4JcObznE7Xf7g+9XgIAbTKse5lP6k9rr8dhjByeW9zLxTVxtcKWX6uhKDGPjEdbbL1QDVPhNhlPWhGZhS8+WxZwp7sPgN1yYNJRP9+cwOeU= ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-11-15T06:25:59.199 INFO:teuthology.orchestra.run.smithi062.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5+aqffi9x3Pq9kjlnQYPUXA/1lo4stG9BltXRno7qEJ/+OWtYt5Wmb9jCJ33mRCfVPPj1QKZDTHse1niDh1wX7rkYZocmeUMS5P7xr5YA/DP5sqqP87WsOjWBMiRDK5yUWlD2T7bVUO8QhLGHJFYfhrARhnqUxnhFwF2jkKao9Z1pB+PG8bTmv+S8nio8EfDEwOzLtuSjo8Q9ZLekPhc4UpWuUIqB0xLQxwSIVst/0EE2sYME+DfXCZMiYSEZuvf1Fs0cEOXdfR6JpR10YR3HIhvQ2h7P8csIq842USQCOsWhIXRh8/L8S2wiprO0vb9tnZ96tiX7sj1YrKLoMlgnLc7BbHlBOwceF999ljHiHPGG1Xhd8X9Gp6kMoXIpcBzcN3kkeW/EeLsDp+v3YJlU4JcObznE7Xf7g+9XgIAbTKse5lP6k9rr8dhjByeW9zLxTVxtcKWX6uhKDGPjEdbbL1QDVPhNhlPWhGZhS8+WxZwp7sPgN1yYNJRP9+cwOeU= ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7 2023-11-15T06:25:59.210 DEBUG:teuthology.orchestra.run.smithi081:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5+aqffi9x3Pq9kjlnQYPUXA/1lo4stG9BltXRno7qEJ/+OWtYt5Wmb9jCJ33mRCfVPPj1QKZDTHse1niDh1wX7rkYZocmeUMS5P7xr5YA/DP5sqqP87WsOjWBMiRDK5yUWlD2T7bVUO8QhLGHJFYfhrARhnqUxnhFwF2jkKao9Z1pB+PG8bTmv+S8nio8EfDEwOzLtuSjo8Q9ZLekPhc4UpWuUIqB0xLQxwSIVst/0EE2sYME+DfXCZMiYSEZuvf1Fs0cEOXdfR6JpR10YR3HIhvQ2h7P8csIq842USQCOsWhIXRh8/L8S2wiprO0vb9tnZ96tiX7sj1YrKLoMlgnLc7BbHlBOwceF999ljHiHPGG1Xhd8X9Gp6kMoXIpcBzcN3kkeW/EeLsDp+v3YJlU4JcObznE7Xf7g+9XgIAbTKse5lP6k9rr8dhjByeW9zLxTVxtcKWX6uhKDGPjEdbbL1QDVPhNhlPWhGZhS8+WxZwp7sPgN1yYNJRP9+cwOeU= ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-11-15T06:25:59.255 INFO:teuthology.orchestra.run.smithi081.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5+aqffi9x3Pq9kjlnQYPUXA/1lo4stG9BltXRno7qEJ/+OWtYt5Wmb9jCJ33mRCfVPPj1QKZDTHse1niDh1wX7rkYZocmeUMS5P7xr5YA/DP5sqqP87WsOjWBMiRDK5yUWlD2T7bVUO8QhLGHJFYfhrARhnqUxnhFwF2jkKao9Z1pB+PG8bTmv+S8nio8EfDEwOzLtuSjo8Q9ZLekPhc4UpWuUIqB0xLQxwSIVst/0EE2sYME+DfXCZMiYSEZuvf1Fs0cEOXdfR6JpR10YR3HIhvQ2h7P8csIq842USQCOsWhIXRh8/L8S2wiprO0vb9tnZ96tiX7sj1YrKLoMlgnLc7BbHlBOwceF999ljHiHPGG1Xhd8X9Gp6kMoXIpcBzcN3kkeW/EeLsDp+v3YJlU4JcObznE7Xf7g+9XgIAbTKse5lP6k9rr8dhjByeW9zLxTVxtcKWX6uhKDGPjEdbbL1QDVPhNhlPWhGZhS8+WxZwp7sPgN1yYNJRP9+cwOeU= ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7 2023-11-15T06:25:59.267 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-11-15T06:26:00.668 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:00.630+0000 7f226f816700 1 -- 172.21.15.62:0/4290314288 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2268060280 msgr2=0x7f22680f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:26:00.669 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:00.630+0000 7f226f816700 1 --2- 172.21.15.62:0/4290314288 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2268060280 0x7f22680f90f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f2264009fb0 tx=0x7f2264009b30).stop 2023-11-15T06:26:00.669 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:00.630+0000 7f226f816700 1 -- 172.21.15.62:0/4290314288 shutdown_connections 2023-11-15T06:26:00.669 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:00.630+0000 7f226f816700 1 --2- 172.21.15.62:0/4290314288 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2268060280 0x7f22680f90f0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:26:00.669 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:00.630+0000 7f226f816700 1 -- 172.21.15.62:0/4290314288 >> 172.21.15.62:0/4290314288 conn(0x7f22680f0b70 msgr2=0x7f22680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:26:00.669 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:00.630+0000 7f226f816700 1 -- 172.21.15.62:0/4290314288 shutdown_connections 2023-11-15T06:26:00.670 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:00.630+0000 7f226f816700 1 -- 172.21.15.62:0/4290314288 wait complete. 2023-11-15T06:26:00.670 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:00.630+0000 7f226f816700 1 Processor -- start 2023-11-15T06:26:00.670 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:00.630+0000 7f226f816700 1 -- start start 2023-11-15T06:26:00.670 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:00.631+0000 7f226f816700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2268060280 0x7f2268108900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:26:00.670 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:00.631+0000 7f226f816700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2268108df0 con 0x7f2268060280 2023-11-15T06:26:00.671 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:00.631+0000 7f226d5b2700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2268060280 0x7f2268108900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:26:00.671 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:00.631+0000 7f226d5b2700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2268060280 0x7f2268108900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45666/0 (socket says 172.21.15.62:45666) 2023-11-15T06:26:00.671 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:00.631+0000 7f226d5b2700 1 -- 172.21.15.62:0/1889298753 learned_addr learned my addr 172.21.15.62:0/1889298753 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:26:00.671 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:00.631+0000 7f226d5b2700 1 -- 172.21.15.62:0/1889298753 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2264005040 con 0x7f2268060280 2023-11-15T06:26:00.671 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:00.631+0000 7f226d5b2700 1 --2- 172.21.15.62:0/1889298753 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2268060280 0x7f2268108900 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f2264013510 tx=0x7f226400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:26:00.672 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:00.631+0000 7f225dffb700 1 -- 172.21.15.62:0/1889298753 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2264009320 con 0x7f2268060280 2023-11-15T06:26:00.672 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:00.631+0000 7f225dffb700 1 -- 172.21.15.62:0/1889298753 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2264013f00 con 0x7f2268060280 2023-11-15T06:26:00.672 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:00.631+0000 7f226f816700 1 -- 172.21.15.62:0/1889298753 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2268108ff0 con 0x7f2268060280 2023-11-15T06:26:00.672 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:00.631+0000 7f225dffb700 1 -- 172.21.15.62:0/1889298753 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2264014dc0 con 0x7f2268060280 2023-11-15T06:26:00.672 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:00.632+0000 7f226f816700 1 -- 172.21.15.62:0/1889298753 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2268109480 con 0x7f2268060280 2023-11-15T06:26:00.673 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:00.633+0000 7f225dffb700 1 -- 172.21.15.62:0/1889298753 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f226401d3d0 con 0x7f2268060280 2023-11-15T06:26:00.673 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:00.633+0000 7f226f816700 1 -- 172.21.15.62:0/1889298753 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2268045ad0 con 0x7f2268060280 2023-11-15T06:26:00.673 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:00.633+0000 7f225dffb700 1 --2- 172.21.15.62:0/1889298753 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f225402ff60 0x7f2254032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:26:00.673 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:00.633+0000 7f225dffb700 1 -- 172.21.15.62:0/1889298753 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2264041060 con 0x7f2268060280 2023-11-15T06:26:00.673 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:00.633+0000 7f226cdb1700 1 --2- 172.21.15.62:0/1889298753 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f225402ff60 0x7f2254032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:26:00.674 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:00.634+0000 7f226cdb1700 1 --2- 172.21.15.62:0/1889298753 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f225402ff60 0x7f2254032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f22580096a0 tx=0x7f2258006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:26:00.674 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:00.640+0000 7f225dffb700 1 -- 172.21.15.62:0/1889298753 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f22640095f0 con 0x7f2268060280 2023-11-15T06:26:00.805 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:00.804+0000 7f226f816700 1 -- 172.21.15.62:0/1889298753 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f22680fbf50 con 0x7f2268060280 2023-11-15T06:26:00.812 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:00.811+0000 7f225dffb700 1 -- 172.21.15.62:0/1889298753 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v11)=0 v11) v1 ==== 125+0+0 (secure 0 0 0) 0x7f226401cb10 con 0x7f2268060280 2023-11-15T06:26:00.815 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:00.814+0000 7f226f816700 1 -- 172.21.15.62:0/1889298753 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f225402ff60 msgr2=0x7f2254032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:26:00.816 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:00.814+0000 7f226f816700 1 --2- 172.21.15.62:0/1889298753 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f225402ff60 0x7f2254032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f22580096a0 tx=0x7f2258006b40).stop 2023-11-15T06:26:00.816 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:00.814+0000 7f226f816700 1 -- 172.21.15.62:0/1889298753 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2268060280 msgr2=0x7f2268108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:26:00.816 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:00.814+0000 7f226f816700 1 --2- 172.21.15.62:0/1889298753 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2268060280 0x7f2268108900 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f2264013510 tx=0x7f226400b040).stop 2023-11-15T06:26:00.816 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:00.814+0000 7f226f816700 1 -- 172.21.15.62:0/1889298753 shutdown_connections 2023-11-15T06:26:00.816 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:00.814+0000 7f226f816700 1 --2- 172.21.15.62:0/1889298753 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f225402ff60 0x7f2254032420 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:26:00.817 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:00.814+0000 7f226f816700 1 --2- 172.21.15.62:0/1889298753 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2268060280 0x7f2268108900 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:26:00.817 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:00.814+0000 7f226f816700 1 -- 172.21.15.62:0/1889298753 >> 172.21.15.62:0/1889298753 conn(0x7f22680f0b70 msgr2=0x7f22680f15d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:26:00.817 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:00.814+0000 7f226f816700 1 -- 172.21.15.62:0/1889298753 shutdown_connections 2023-11-15T06:26:00.817 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:00.814+0000 7f226f816700 1 -- 172.21.15.62:0/1889298753 wait complete. 2023-11-15T06:26:01.045 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:00 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:25:59.790131+0000 mon.smithi062 (mon.0) 113 : cluster [DBG] mgrmap e17: smithi062.rwurtk(active, since 6s) 2023-11-15T06:26:01.045 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:00 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:26:00.664444+0000 mon.smithi062 (mon.0) 114 : audit [INF] from='mgr.14164 172.21.15.62:0/147479783' entity='mgr.smithi062.rwurtk' 2023-11-15T06:26:01.215 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi081 2023-11-15T06:26:01.215 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2023-11-15T06:26:01.216 DEBUG:teuthology.orchestra.run.smithi081:> dd of=/etc/ceph/ceph.conf 2023-11-15T06:26:01.239 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2023-11-15T06:26:01.240 DEBUG:teuthology.orchestra.run.smithi081:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-11-15T06:26:01.300 INFO:tasks.cephadm:Adding host smithi081 to orchestrator... 2023-11-15T06:26:01.301 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph orch host add smithi081 2023-11-15T06:26:02.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:01 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:26:00.811743+0000 mon.smithi062 (mon.0) 115 : audit [INF] from='client.? 172.21.15.62:0/1889298753' entity='client.admin' 2023-11-15T06:26:02.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:01 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:26:01.020412+0000 mon.smithi062 (mon.0) 116 : audit [INF] from='mgr.14164 172.21.15.62:0/147479783' entity='mgr.smithi062.rwurtk' 2023-11-15T06:26:02.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:01 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cephadm 2023-11-15T06:26:01.027081+0000 mgr.smithi062.rwurtk (mgr.14164) 5 : cephadm [INF] Deploying daemon alertmanager.smithi062 on smithi062 2023-11-15T06:26:15.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:14 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:26:13.779564+0000 mgr.smithi062.rwurtk (mgr.14164) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:26:17.167 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:16 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:26:15.779898+0000 mgr.smithi062.rwurtk (mgr.14164) 7 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:26:18.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:26:17.780266+0000 mgr.smithi062.rwurtk (mgr.14164) 8 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:26:21.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:20 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:26:19.780671+0000 mgr.smithi062.rwurtk (mgr.14164) 9 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:26:22.357 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:22.353+0000 7fcfb98b4700 1 -- 172.21.15.81:0/2534525968 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcfb40f6510 msgr2=0x7fcfb40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:26:22.358 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:22.353+0000 7fcfb98b4700 1 --2- 172.21.15.81:0/2534525968 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcfb40f6510 0x7fcfb40f6930 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fcfa8009fb0 tx=0x7fcfa8009b30).stop 2023-11-15T06:26:22.358 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:22.353+0000 7fcfb98b4700 1 -- 172.21.15.81:0/2534525968 shutdown_connections 2023-11-15T06:26:22.358 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:22.353+0000 7fcfb98b4700 1 --2- 172.21.15.81:0/2534525968 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcfb40f6510 0x7fcfb40f6930 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:26:22.358 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:22.353+0000 7fcfb98b4700 1 -- 172.21.15.81:0/2534525968 >> 172.21.15.81:0/2534525968 conn(0x7fcfb40f1c70 msgr2=0x7fcfb40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:26:22.359 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:22.353+0000 7fcfb98b4700 1 -- 172.21.15.81:0/2534525968 shutdown_connections 2023-11-15T06:26:22.359 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:22.353+0000 7fcfb98b4700 1 -- 172.21.15.81:0/2534525968 wait complete. 2023-11-15T06:26:22.359 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:22.354+0000 7fcfb98b4700 1 Processor -- start 2023-11-15T06:26:22.359 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:22.354+0000 7fcfb98b4700 1 -- start start 2023-11-15T06:26:22.359 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:22.354+0000 7fcfb98b4700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcfb40f6510 0x7fcfb4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:26:22.360 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:22.354+0000 7fcfb98b4700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb4106c20 con 0x7fcfb40f6510 2023-11-15T06:26:22.360 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:22.354+0000 7fcfb2ffd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcfb40f6510 0x7fcfb4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:26:22.360 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:22.354+0000 7fcfb2ffd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcfb40f6510 0x7fcfb4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:44604/0 (socket says 172.21.15.81:44604) 2023-11-15T06:26:22.360 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:22.354+0000 7fcfb2ffd700 1 -- 172.21.15.81:0/704887121 learned_addr learned my addr 172.21.15.81:0/704887121 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:26:22.360 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:22.355+0000 7fcfb2ffd700 1 -- 172.21.15.81:0/704887121 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa8005040 con 0x7fcfb40f6510 2023-11-15T06:26:22.361 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:22.355+0000 7fcfb2ffd700 1 --2- 172.21.15.81:0/704887121 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcfb40f6510 0x7fcfb4106730 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fcfa8013510 tx=0x7fcfa800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:26:22.361 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:22.355+0000 7fcf9b7fe700 1 -- 172.21.15.81:0/704887121 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfa8009320 con 0x7fcfb40f6510 2023-11-15T06:26:22.361 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:22.355+0000 7fcf9b7fe700 1 -- 172.21.15.81:0/704887121 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcfa8013f00 con 0x7fcfb40f6510 2023-11-15T06:26:22.361 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:22.355+0000 7fcf9b7fe700 1 -- 172.21.15.81:0/704887121 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfa8014d90 con 0x7fcfb40f6510 2023-11-15T06:26:22.361 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:22.355+0000 7fcfb98b4700 1 -- 172.21.15.81:0/704887121 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfb4106e20 con 0x7fcfb40f6510 2023-11-15T06:26:22.365 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:22.355+0000 7fcfb98b4700 1 -- 172.21.15.81:0/704887121 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfb41072b0 con 0x7fcfb40f6510 2023-11-15T06:26:22.366 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:22.356+0000 7fcf9b7fe700 1 -- 172.21.15.81:0/704887121 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcfa80095f0 con 0x7fcfb40f6510 2023-11-15T06:26:22.367 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:22.356+0000 7fcfb98b4700 1 -- 172.21.15.81:0/704887121 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfb4045ad0 con 0x7fcfb40f6510 2023-11-15T06:26:22.368 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:22.356+0000 7fcf9b7fe700 1 --2- 172.21.15.81:0/704887121 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fcf9c02ff60 0x7fcf9c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:26:22.368 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:22.356+0000 7fcf9b7fe700 1 -- 172.21.15.81:0/704887121 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcfa8041000 con 0x7fcfb40f6510 2023-11-15T06:26:22.368 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:22.356+0000 7fcfb27fc700 1 --2- 172.21.15.81:0/704887121 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fcf9c02ff60 0x7fcf9c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:26:22.368 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:22.356+0000 7fcfb27fc700 1 --2- 172.21.15.81:0/704887121 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fcf9c02ff60 0x7fcf9c032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fcfa40096a0 tx=0x7fcfa4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:26:22.368 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:22.361+0000 7fcf9b7fe700 1 -- 172.21.15.81:0/704887121 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcfa801d7f0 con 0x7fcfb40f6510 2023-11-15T06:26:22.534 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:22.532+0000 7fcfb98b4700 1 -- 172.21.15.81:0/704887121 --> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi081", "target": ["mon-mgr", ""]}) v1 -- 0x7fcfb40f2e70 con 0x7fcf9c02ff60 2023-11-15T06:26:23.144 INFO:teuthology.orchestra.run.smithi081.stdout:Added host 'smithi081' 2023-11-15T06:26:23.145 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:23.141+0000 7fcf9b7fe700 1 -- 172.21.15.81:0/704887121 <== mgr.14164 v2:172.21.15.62:6800/2759863993 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7fcfb40f2e70 con 0x7fcf9c02ff60 2023-11-15T06:26:23.146 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:23.143+0000 7fcfb98b4700 1 -- 172.21.15.81:0/704887121 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fcf9c02ff60 msgr2=0x7fcf9c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:26:23.147 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:23.143+0000 7fcfb98b4700 1 --2- 172.21.15.81:0/704887121 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fcf9c02ff60 0x7fcf9c032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fcfa40096a0 tx=0x7fcfa4006b40).stop 2023-11-15T06:26:23.147 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:23.144+0000 7fcfb98b4700 1 -- 172.21.15.81:0/704887121 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcfb40f6510 msgr2=0x7fcfb4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:26:23.147 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:23.144+0000 7fcfb98b4700 1 --2- 172.21.15.81:0/704887121 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcfb40f6510 0x7fcfb4106730 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fcfa8013510 tx=0x7fcfa800b040).stop 2023-11-15T06:26:23.147 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:23.144+0000 7fcfb98b4700 1 -- 172.21.15.81:0/704887121 shutdown_connections 2023-11-15T06:26:23.147 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:23.144+0000 7fcfb98b4700 1 --2- 172.21.15.81:0/704887121 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fcf9c02ff60 0x7fcf9c032420 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:26:23.148 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:23.144+0000 7fcfb98b4700 1 --2- 172.21.15.81:0/704887121 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcfb40f6510 0x7fcfb4106730 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:26:23.148 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:23.144+0000 7fcfb98b4700 1 -- 172.21.15.81:0/704887121 >> 172.21.15.81:0/704887121 conn(0x7fcfb40f1c70 msgr2=0x7fcfb40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:26:23.148 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:23.144+0000 7fcfb98b4700 1 -- 172.21.15.81:0/704887121 shutdown_connections 2023-11-15T06:26:23.148 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:23.144+0000 7fcfb98b4700 1 -- 172.21.15.81:0/704887121 wait complete. 2023-11-15T06:26:23.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:22 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:26:21.780997+0000 mgr.smithi062.rwurtk (mgr.14164) 10 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:26:23.169 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:22 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:26:22.534481+0000 mgr.smithi062.rwurtk (mgr.14164) 11 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi081", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T06:26:24.092 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph orch host ls --format=json 2023-11-15T06:26:24.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:24 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:26:23.142298+0000 mon.smithi062 (mon.0) 117 : audit [INF] from='mgr.14164 172.21.15.62:0/147479783' entity='mgr.smithi062.rwurtk' 2023-11-15T06:26:24.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:24 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cephadm 2023-11-15T06:26:23.142715+0000 mgr.smithi062.rwurtk (mgr.14164) 12 : cephadm [INF] Added host smithi081 2023-11-15T06:26:25.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:25 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:26:23.781256+0000 mgr.smithi062.rwurtk (mgr.14164) 13 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:26:27.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:27 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:26:25.781578+0000 mgr.smithi062.rwurtk (mgr.14164) 14 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:26:28.709 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:28.575+0000 7fdfcef0e700 1 -- 172.21.15.81:0/2424751162 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfc80f62a0 msgr2=0x7fdfc80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:26:28.710 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:28.575+0000 7fdfcef0e700 1 --2- 172.21.15.81:0/2424751162 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfc80f62a0 0x7fdfc80f66c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fdfb8009fb0 tx=0x7fdfb8009b30).stop 2023-11-15T06:26:28.710 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:28.575+0000 7fdfcef0e700 1 -- 172.21.15.81:0/2424751162 shutdown_connections 2023-11-15T06:26:28.710 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:28.575+0000 7fdfcef0e700 1 --2- 172.21.15.81:0/2424751162 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfc80f62a0 0x7fdfc80f66c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:26:28.710 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:28.575+0000 7fdfcef0e700 1 -- 172.21.15.81:0/2424751162 >> 172.21.15.81:0/2424751162 conn(0x7fdfc80f1aa0 msgr2=0x7fdfc80f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:26:28.711 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:28.575+0000 7fdfcef0e700 1 -- 172.21.15.81:0/2424751162 shutdown_connections 2023-11-15T06:26:28.711 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:28.575+0000 7fdfcef0e700 1 -- 172.21.15.81:0/2424751162 wait complete. 2023-11-15T06:26:28.711 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:28.575+0000 7fdfcef0e700 1 Processor -- start 2023-11-15T06:26:28.711 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:28.575+0000 7fdfcef0e700 1 -- start start 2023-11-15T06:26:28.711 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:28.575+0000 7fdfcef0e700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfc80f62a0 0x7fdfc81064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:26:28.712 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:28.575+0000 7fdfcef0e700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc81069a0 con 0x7fdfc80f62a0 2023-11-15T06:26:28.712 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:28.576+0000 7fdfcdf0c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfc80f62a0 0x7fdfc81064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:26:28.712 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:28.576+0000 7fdfcdf0c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfc80f62a0 0x7fdfc81064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:44616/0 (socket says 172.21.15.81:44616) 2023-11-15T06:26:28.712 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:28.576+0000 7fdfcdf0c700 1 -- 172.21.15.81:0/3638122200 learned_addr learned my addr 172.21.15.81:0/3638122200 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:26:28.712 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:28.576+0000 7fdfcdf0c700 1 -- 172.21.15.81:0/3638122200 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb8005040 con 0x7fdfc80f62a0 2023-11-15T06:26:28.713 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:28.576+0000 7fdfcdf0c700 1 --2- 172.21.15.81:0/3638122200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfc80f62a0 0x7fdfc81064b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fdfb8013510 tx=0x7fdfb800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:26:28.713 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:28.576+0000 7fdfbe7fc700 1 -- 172.21.15.81:0/3638122200 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfb8009320 con 0x7fdfc80f62a0 2023-11-15T06:26:28.713 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:28.576+0000 7fdfcef0e700 1 -- 172.21.15.81:0/3638122200 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfc8106ba0 con 0x7fdfc80f62a0 2023-11-15T06:26:28.713 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:28.576+0000 7fdfbe7fc700 1 -- 172.21.15.81:0/3638122200 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdfb8013f00 con 0x7fdfc80f62a0 2023-11-15T06:26:28.713 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:28.576+0000 7fdfbe7fc700 1 -- 172.21.15.81:0/3638122200 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfb8014dc0 con 0x7fdfc80f62a0 2023-11-15T06:26:28.714 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:28.576+0000 7fdfcef0e700 1 -- 172.21.15.81:0/3638122200 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfc8107030 con 0x7fdfc80f62a0 2023-11-15T06:26:28.714 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:28.577+0000 7fdfbe7fc700 1 -- 172.21.15.81:0/3638122200 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdfb801d3d0 con 0x7fdfc80f62a0 2023-11-15T06:26:28.714 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:28.577+0000 7fdfbe7fc700 1 --2- 172.21.15.81:0/3638122200 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fdfb402ff60 0x7fdfb4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:26:28.714 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:28.577+0000 7fdfcef0e700 1 -- 172.21.15.81:0/3638122200 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfc8045ad0 con 0x7fdfc80f62a0 2023-11-15T06:26:28.714 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:28.577+0000 7fdfbe7fc700 1 -- 172.21.15.81:0/3638122200 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdfb8041320 con 0x7fdfc80f62a0 2023-11-15T06:26:28.715 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:28.577+0000 7fdfcd70b700 1 --2- 172.21.15.81:0/3638122200 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fdfb402ff60 0x7fdfb4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:26:28.715 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:28.578+0000 7fdfcd70b700 1 --2- 172.21.15.81:0/3638122200 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fdfb402ff60 0x7fdfb4032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fdfc40096a0 tx=0x7fdfc4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:26:28.715 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:28.583+0000 7fdfbe7fc700 1 -- 172.21.15.81:0/3638122200 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdfb8012920 con 0x7fdfc80f62a0 2023-11-15T06:26:28.757 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:28.754+0000 7fdfcef0e700 1 -- 172.21.15.81:0/3638122200 --> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdfc80f2c20 con 0x7fdfb402ff60 2023-11-15T06:26:28.758 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:28.757+0000 7fdfbe7fc700 1 -- 172.21.15.81:0/3638122200 <== mgr.14164 v2:172.21.15.62:6800/2759863993 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+156 (secure 0 0 0) 0x7fdfc80f2c20 con 0x7fdfb402ff60 2023-11-15T06:26:28.759 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:26:28.759 INFO:teuthology.orchestra.run.smithi081.stdout:[{"addr": "172.21.15.62", "hostname": "smithi062", "labels": [], "status": ""}, {"addr": "smithi081", "hostname": "smithi081", "labels": [], "status": ""}] 2023-11-15T06:26:28.760 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:28.758+0000 7fdfcef0e700 1 -- 172.21.15.81:0/3638122200 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fdfb402ff60 msgr2=0x7fdfb4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:26:28.760 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:28.758+0000 7fdfcef0e700 1 --2- 172.21.15.81:0/3638122200 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fdfb402ff60 0x7fdfb4032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fdfc40096a0 tx=0x7fdfc4006b40).stop 2023-11-15T06:26:28.760 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:28.758+0000 7fdfcef0e700 1 -- 172.21.15.81:0/3638122200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfc80f62a0 msgr2=0x7fdfc81064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:26:28.761 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:28.758+0000 7fdfcef0e700 1 --2- 172.21.15.81:0/3638122200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfc80f62a0 0x7fdfc81064b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fdfb8013510 tx=0x7fdfb800b040).stop 2023-11-15T06:26:28.761 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:28.758+0000 7fdfcef0e700 1 -- 172.21.15.81:0/3638122200 shutdown_connections 2023-11-15T06:26:28.761 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:28.758+0000 7fdfcef0e700 1 --2- 172.21.15.81:0/3638122200 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fdfb402ff60 0x7fdfb4032420 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:26:28.761 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:28.758+0000 7fdfcef0e700 1 --2- 172.21.15.81:0/3638122200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdfc80f62a0 0x7fdfc81064b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:26:28.761 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:28.758+0000 7fdfcef0e700 1 -- 172.21.15.81:0/3638122200 >> 172.21.15.81:0/3638122200 conn(0x7fdfc80f1aa0 msgr2=0x7fdfc80f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:26:28.761 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:28.758+0000 7fdfcef0e700 1 -- 172.21.15.81:0/3638122200 shutdown_connections 2023-11-15T06:26:28.762 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:28.758+0000 7fdfcef0e700 1 -- 172.21.15.81:0/3638122200 wait complete. 2023-11-15T06:26:28.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:28 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:26:27.781843+0000 mgr.smithi062.rwurtk (mgr.14164) 15 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:26:29.692 INFO:tasks.cephadm:Setting crush tunables to default 2023-11-15T06:26:29.692 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph osd crush tunables default 2023-11-15T06:26:29.917 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:29 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:26:28.757041+0000 mgr.smithi062.rwurtk (mgr.14164) 16 : audit [DBG] from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-15T06:26:30.668 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:30 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:26:29.782106+0000 mgr.smithi062.rwurtk (mgr.14164) 17 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:26:30.895 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:30.893+0000 7f5ab0226700 1 -- 172.21.15.62:0/3101897708 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5aa80f5410 msgr2=0x7f5aa80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:26:30.896 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:30.893+0000 7f5ab0226700 1 --2- 172.21.15.62:0/3101897708 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5aa80f5410 0x7f5aa80f5830 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f5a98009fb0 tx=0x7f5a98009b30).stop 2023-11-15T06:26:30.896 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:30.893+0000 7f5ab0226700 1 -- 172.21.15.62:0/3101897708 shutdown_connections 2023-11-15T06:26:30.896 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:30.893+0000 7f5ab0226700 1 --2- 172.21.15.62:0/3101897708 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5aa80f5410 0x7f5aa80f5830 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:26:30.896 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:30.893+0000 7f5ab0226700 1 -- 172.21.15.62:0/3101897708 >> 172.21.15.62:0/3101897708 conn(0x7f5aa80f0b70 msgr2=0x7f5aa80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:26:30.897 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:30.893+0000 7f5ab0226700 1 -- 172.21.15.62:0/3101897708 shutdown_connections 2023-11-15T06:26:30.897 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:30.893+0000 7f5ab0226700 1 -- 172.21.15.62:0/3101897708 wait complete. 2023-11-15T06:26:30.899 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:30.893+0000 7f5ab0226700 1 Processor -- start 2023-11-15T06:26:30.899 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:30.894+0000 7f5ab0226700 1 -- start start 2023-11-15T06:26:30.899 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:30.894+0000 7f5ab0226700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5aa80f5410 0x7f5aa8109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:26:30.899 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:30.894+0000 7f5ab0226700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa8060240 con 0x7f5aa80f5410 2023-11-15T06:26:30.900 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:30.894+0000 7f5aadfc2700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5aa80f5410 0x7f5aa8109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:26:30.900 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:30.894+0000 7f5aadfc2700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5aa80f5410 0x7f5aa8109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50338/0 (socket says 172.21.15.62:50338) 2023-11-15T06:26:30.900 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:30.894+0000 7f5aadfc2700 1 -- 172.21.15.62:0/1579034565 learned_addr learned my addr 172.21.15.62:0/1579034565 (peer_addr_for_me v2:172.21.15.62:0/0) 2023-11-15T06:26:30.900 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:30.894+0000 7f5aadfc2700 1 -- 172.21.15.62:0/1579034565 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a98005040 con 0x7f5aa80f5410 2023-11-15T06:26:30.900 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:30.894+0000 7f5aadfc2700 1 --2- 172.21.15.62:0/1579034565 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5aa80f5410 0x7f5aa8109430 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f5a98013510 tx=0x7f5a9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:26:30.901 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:30.895+0000 7f5a9e7fc700 1 -- 172.21.15.62:0/1579034565 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a98009320 con 0x7f5aa80f5410 2023-11-15T06:26:30.901 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:30.895+0000 7f5a9e7fc700 1 -- 172.21.15.62:0/1579034565 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5a98013f00 con 0x7f5aa80f5410 2023-11-15T06:26:30.901 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:30.895+0000 7f5a9e7fc700 1 -- 172.21.15.62:0/1579034565 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a98014d90 con 0x7f5aa80f5410 2023-11-15T06:26:30.901 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:30.895+0000 7f5ab0226700 1 -- 172.21.15.62:0/1579034565 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aa8060440 con 0x7f5aa80f5410 2023-11-15T06:26:30.901 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:30.895+0000 7f5ab0226700 1 -- 172.21.15.62:0/1579034565 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aa8109ae0 con 0x7f5aa80f5410 2023-11-15T06:26:30.902 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:30.895+0000 7f5a9e7fc700 1 -- 172.21.15.62:0/1579034565 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5a980095f0 con 0x7f5aa80f5410 2023-11-15T06:26:30.902 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:30.896+0000 7f5a9e7fc700 1 --2- 172.21.15.62:0/1579034565 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f5a9402ff60 0x7f5a94032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:26:30.902 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:30.896+0000 7f5a9e7fc700 1 -- 172.21.15.62:0/1579034565 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5a98040fe0 con 0x7f5aa80f5410 2023-11-15T06:26:30.902 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:30.896+0000 7f5ab0226700 1 -- 172.21.15.62:0/1579034565 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5aa8045ad0 con 0x7f5aa80f5410 2023-11-15T06:26:30.902 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:30.896+0000 7f5aad7c1700 1 --2- 172.21.15.62:0/1579034565 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f5a9402ff60 0x7f5a94032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:26:30.903 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:30.896+0000 7f5aad7c1700 1 --2- 172.21.15.62:0/1579034565 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f5a9402ff60 0x7f5a94032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f5aa40096a0 tx=0x7f5aa4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:26:30.903 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:30.900+0000 7f5a9e7fc700 1 -- 172.21.15.62:0/1579034565 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5a98014420 con 0x7f5aa80f5410 2023-11-15T06:26:31.066 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:31.064+0000 7f5ab0226700 1 -- 172.21.15.62:0/1579034565 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f5aa80fae60 con 0x7f5aa80f5410 2023-11-15T06:26:31.608 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:31.606+0000 7f5a9e7fc700 1 -- 172.21.15.62:0/1579034565 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7f5aa80fae60 con 0x7f5aa80f5410 2023-11-15T06:26:31.610 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:31.608+0000 7f5ab0226700 1 -- 172.21.15.62:0/1579034565 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f5a9402ff60 msgr2=0x7f5a94032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:26:31.610 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:31.608+0000 7f5ab0226700 1 --2- 172.21.15.62:0/1579034565 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f5a9402ff60 0x7f5a94032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f5aa40096a0 tx=0x7f5aa4006b40).stop 2023-11-15T06:26:31.611 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:31.608+0000 7f5ab0226700 1 -- 172.21.15.62:0/1579034565 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5aa80f5410 msgr2=0x7f5aa8109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:26:31.611 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:31.608+0000 7f5ab0226700 1 --2- 172.21.15.62:0/1579034565 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5aa80f5410 0x7f5aa8109430 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f5a98013510 tx=0x7f5a9800b040).stop 2023-11-15T06:26:31.611 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:31.608+0000 7f5ab0226700 1 -- 172.21.15.62:0/1579034565 shutdown_connections 2023-11-15T06:26:31.611 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:31.608+0000 7f5ab0226700 1 --2- 172.21.15.62:0/1579034565 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f5a9402ff60 0x7f5a94032420 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:26:31.611 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:31.608+0000 7f5ab0226700 1 --2- 172.21.15.62:0/1579034565 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5aa80f5410 0x7f5aa8109430 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:26:31.612 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:31.608+0000 7f5ab0226700 1 -- 172.21.15.62:0/1579034565 >> 172.21.15.62:0/1579034565 conn(0x7f5aa80f0b70 msgr2=0x7f5aa80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:26:31.612 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:31.608+0000 7f5ab0226700 1 -- 172.21.15.62:0/1579034565 shutdown_connections 2023-11-15T06:26:31.612 INFO:teuthology.orchestra.run.smithi062.stderr:2023-11-15T06:26:31.608+0000 7f5ab0226700 1 -- 172.21.15.62:0/1579034565 wait complete. 2023-11-15T06:26:31.612 INFO:teuthology.orchestra.run.smithi062.stderr:adjusted tunables profile to default 2023-11-15T06:26:31.911 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:31 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:26:31.066003+0000 mon.smithi062 (mon.0) 118 : audit [INF] from='client.? 172.21.15.62:0/1579034565' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-11-15T06:26:31.991 INFO:tasks.cephadm:Adding mon.smithi062 on smithi062 2023-11-15T06:26:31.992 INFO:tasks.cephadm:Adding mon.smithi081 on smithi081 2023-11-15T06:26:31.992 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph orch apply mon '2;smithi062:172.21.15.62=smithi062;smithi081:172.21.15.81=smithi081' 2023-11-15T06:26:32.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:32 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:26:31.607391+0000 mon.smithi062 (mon.0) 119 : audit [INF] from='client.? 172.21.15.62:0/1579034565' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-11-15T06:26:32.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:32 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:26:31.607459+0000 mon.smithi062 (mon.0) 120 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2023-11-15T06:26:32.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:32 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:26:31.782667+0000 mgr.smithi062.rwurtk (mgr.14164) 18 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:26:33.147 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:33.145+0000 7f2234ad5700 1 -- 172.21.15.81:0/1836888114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22300f5410 msgr2=0x7f22300f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:26:33.147 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:33.145+0000 7f2234ad5700 1 --2- 172.21.15.81:0/1836888114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22300f5410 0x7f22300f5830 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f2224009fb0 tx=0x7f2224009b30).stop 2023-11-15T06:26:33.147 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:33.145+0000 7f2234ad5700 1 -- 172.21.15.81:0/1836888114 shutdown_connections 2023-11-15T06:26:33.147 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:33.145+0000 7f2234ad5700 1 --2- 172.21.15.81:0/1836888114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22300f5410 0x7f22300f5830 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:26:33.148 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:33.145+0000 7f2234ad5700 1 -- 172.21.15.81:0/1836888114 >> 172.21.15.81:0/1836888114 conn(0x7f22300f0b70 msgr2=0x7f22300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:26:33.148 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:33.145+0000 7f2234ad5700 1 -- 172.21.15.81:0/1836888114 shutdown_connections 2023-11-15T06:26:33.148 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:33.145+0000 7f2234ad5700 1 -- 172.21.15.81:0/1836888114 wait complete. 2023-11-15T06:26:33.150 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:33.146+0000 7f2234ad5700 1 Processor -- start 2023-11-15T06:26:33.150 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:33.146+0000 7f2234ad5700 1 -- start start 2023-11-15T06:26:33.150 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:33.146+0000 7f2234ad5700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22300f5410 0x7f2230106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:26:33.151 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:33.146+0000 7f2234ad5700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2230106c10 con 0x7f22300f5410 2023-11-15T06:26:33.151 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:33.146+0000 7f222e59c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22300f5410 0x7f2230106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:26:33.151 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:33.146+0000 7f222e59c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22300f5410 0x7f2230106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:42806/0 (socket says 172.21.15.81:42806) 2023-11-15T06:26:33.151 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:33.146+0000 7f222e59c700 1 -- 172.21.15.81:0/3562983985 learned_addr learned my addr 172.21.15.81:0/3562983985 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:26:33.151 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:33.146+0000 7f222e59c700 1 -- 172.21.15.81:0/3562983985 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2224005040 con 0x7f22300f5410 2023-11-15T06:26:33.152 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:33.146+0000 7f222e59c700 1 --2- 172.21.15.81:0/3562983985 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22300f5410 0x7f2230106720 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f2224013510 tx=0x7f222400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:26:33.152 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:33.147+0000 7f221effd700 1 -- 172.21.15.81:0/3562983985 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2224009320 con 0x7f22300f5410 2023-11-15T06:26:33.152 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:33.147+0000 7f2234ad5700 1 -- 172.21.15.81:0/3562983985 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2230106e10 con 0x7f22300f5410 2023-11-15T06:26:33.152 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:33.147+0000 7f221effd700 1 -- 172.21.15.81:0/3562983985 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2224013f00 con 0x7f22300f5410 2023-11-15T06:26:33.152 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:33.147+0000 7f221effd700 1 -- 172.21.15.81:0/3562983985 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2224014d90 con 0x7f22300f5410 2023-11-15T06:26:33.153 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:33.147+0000 7f2234ad5700 1 -- 172.21.15.81:0/3562983985 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22301072a0 con 0x7f22300f5410 2023-11-15T06:26:33.153 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:33.147+0000 7f221effd700 1 -- 172.21.15.81:0/3562983985 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f22240263a0 con 0x7f22300f5410 2023-11-15T06:26:33.153 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:33.148+0000 7f221effd700 1 --2- 172.21.15.81:0/3562983985 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f221802ff60 0x7f2218032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:26:33.153 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:33.148+0000 7f221effd700 1 -- 172.21.15.81:0/3562983985 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f22240410f0 con 0x7f22300f5410 2023-11-15T06:26:33.153 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:33.148+0000 7f2234ad5700 1 -- 172.21.15.81:0/3562983985 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2230045ad0 con 0x7f22300f5410 2023-11-15T06:26:33.154 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:33.148+0000 7f222dd9b700 1 --2- 172.21.15.81:0/3562983985 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f221802ff60 0x7f2218032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:26:33.154 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:33.148+0000 7f222dd9b700 1 --2- 172.21.15.81:0/3562983985 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f221802ff60 0x7f2218032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f22200096a0 tx=0x7f2220006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:26:33.154 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:33.153+0000 7f221effd700 1 -- 172.21.15.81:0/3562983985 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2224012920 con 0x7f22300f5410 2023-11-15T06:26:33.330 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:33.328+0000 7f2234ad5700 1 -- 172.21.15.81:0/3562983985 --> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi062:172.21.15.62=smithi062;smithi081:172.21.15.81=smithi081", "target": ["mon-mgr", ""]}) v1 -- 0x7f22300f2a80 con 0x7f221802ff60 2023-11-15T06:26:33.335 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:33.334+0000 7f221effd700 1 -- 172.21.15.81:0/3562983985 <== mgr.14164 v2:172.21.15.62:6800/2759863993 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f22300f2a80 con 0x7f221802ff60 2023-11-15T06:26:33.336 INFO:teuthology.orchestra.run.smithi081.stdout:Scheduled mon update... 2023-11-15T06:26:33.337 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:33.335+0000 7f2234ad5700 1 -- 172.21.15.81:0/3562983985 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f221802ff60 msgr2=0x7f2218032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:26:33.337 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:33.335+0000 7f2234ad5700 1 --2- 172.21.15.81:0/3562983985 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f221802ff60 0x7f2218032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f22200096a0 tx=0x7f2220006b40).stop 2023-11-15T06:26:33.338 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:33.335+0000 7f2234ad5700 1 -- 172.21.15.81:0/3562983985 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22300f5410 msgr2=0x7f2230106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:26:33.338 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:33.335+0000 7f2234ad5700 1 --2- 172.21.15.81:0/3562983985 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22300f5410 0x7f2230106720 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f2224013510 tx=0x7f222400b040).stop 2023-11-15T06:26:33.338 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:33.335+0000 7f2234ad5700 1 -- 172.21.15.81:0/3562983985 shutdown_connections 2023-11-15T06:26:33.338 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:33.335+0000 7f2234ad5700 1 --2- 172.21.15.81:0/3562983985 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f221802ff60 0x7f2218032420 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:26:33.338 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:33.335+0000 7f2234ad5700 1 --2- 172.21.15.81:0/3562983985 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22300f5410 0x7f2230106720 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:26:33.339 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:33.335+0000 7f2234ad5700 1 -- 172.21.15.81:0/3562983985 >> 172.21.15.81:0/3562983985 conn(0x7f22300f0b70 msgr2=0x7f223005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:26:33.339 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:33.335+0000 7f2234ad5700 1 -- 172.21.15.81:0/3562983985 shutdown_connections 2023-11-15T06:26:33.339 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:33.335+0000 7f2234ad5700 1 -- 172.21.15.81:0/3562983985 wait complete. 2023-11-15T06:26:33.720 DEBUG:teuthology.orchestra.run.smithi081:mon.smithi081> sudo journalctl -f -n 0 -u ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7@mon.smithi081.service 2023-11-15T06:26:33.723 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:26:33.723 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:26:33.751 INFO:journalctl@ceph.mon.smithi081.smithi081.stdout:-- Logs begin at Wed 2023-11-15 06:12:43 UTC. -- 2023-11-15T06:26:34.668 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:34 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:26:33.330071+0000 mgr.smithi062.rwurtk (mgr.14164) 19 : audit [DBG] from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi062:172.21.15.62=smithi062;smithi081:172.21.15.81=smithi081", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T06:26:34.668 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:34 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cephadm 2023-11-15T06:26:33.331535+0000 mgr.smithi062.rwurtk (mgr.14164) 20 : cephadm [INF] Saving service mon spec with placement smithi062:172.21.15.62=smithi062;smithi081:172.21.15.81=smithi081;count:2 2023-11-15T06:26:34.669 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:34 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:26:33.334113+0000 mon.smithi062 (mon.0) 121 : audit [INF] from='mgr.14164 172.21.15.62:0/147479783' entity='mgr.smithi062.rwurtk' 2023-11-15T06:26:34.799 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:34.796+0000 7f4017afd700 1 -- 172.21.15.81:0/4219364413 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40100f5410 msgr2=0x7f40100f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:26:34.800 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:34.796+0000 7f4017afd700 1 --2- 172.21.15.81:0/4219364413 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40100f5410 0x7f40100f5830 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f4000009fb0 tx=0x7f4000009b30).stop 2023-11-15T06:26:34.800 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:34.796+0000 7f4017afd700 1 -- 172.21.15.81:0/4219364413 shutdown_connections 2023-11-15T06:26:34.800 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:34.796+0000 7f4017afd700 1 --2- 172.21.15.81:0/4219364413 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40100f5410 0x7f40100f5830 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:26:34.800 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:34.796+0000 7f4017afd700 1 -- 172.21.15.81:0/4219364413 >> 172.21.15.81:0/4219364413 conn(0x7f40100f0b70 msgr2=0x7f40100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:26:34.800 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:34.796+0000 7f4017afd700 1 -- 172.21.15.81:0/4219364413 shutdown_connections 2023-11-15T06:26:34.801 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:34.796+0000 7f4017afd700 1 -- 172.21.15.81:0/4219364413 wait complete. 2023-11-15T06:26:34.801 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:34.797+0000 7f4017afd700 1 Processor -- start 2023-11-15T06:26:34.801 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:34.797+0000 7f4017afd700 1 -- start start 2023-11-15T06:26:34.801 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:34.797+0000 7f4017afd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40100f5410 0x7f40101088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:26:34.801 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:34.797+0000 7f4017afd700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4010108dc0 con 0x7f40100f5410 2023-11-15T06:26:34.802 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:34.797+0000 7f4015899700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40100f5410 0x7f40101088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:26:34.802 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:34.797+0000 7f4015899700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40100f5410 0x7f40101088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:42818/0 (socket says 172.21.15.81:42818) 2023-11-15T06:26:34.802 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:34.797+0000 7f4015899700 1 -- 172.21.15.81:0/926561254 learned_addr learned my addr 172.21.15.81:0/926561254 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:26:34.802 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:34.797+0000 7f4015899700 1 -- 172.21.15.81:0/926561254 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4000005040 con 0x7f40100f5410 2023-11-15T06:26:34.803 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:34.798+0000 7f4015899700 1 --2- 172.21.15.81:0/926561254 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40100f5410 0x7f40101088d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f4000013510 tx=0x7f400000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:26:34.803 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:34.798+0000 7f40067fc700 1 -- 172.21.15.81:0/926561254 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4000009320 con 0x7f40100f5410 2023-11-15T06:26:34.803 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:34.798+0000 7f4017afd700 1 -- 172.21.15.81:0/926561254 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4010108fc0 con 0x7f40100f5410 2023-11-15T06:26:34.803 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:34.798+0000 7f40067fc700 1 -- 172.21.15.81:0/926561254 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4000013f00 con 0x7f40100f5410 2023-11-15T06:26:34.803 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:34.798+0000 7f4017afd700 1 -- 172.21.15.81:0/926561254 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4010109450 con 0x7f40100f5410 2023-11-15T06:26:34.804 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:34.798+0000 7f40067fc700 1 -- 172.21.15.81:0/926561254 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4000014d90 con 0x7f40100f5410 2023-11-15T06:26:34.805 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:34.798+0000 7f40067fc700 1 -- 172.21.15.81:0/926561254 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f40000263a0 con 0x7f40100f5410 2023-11-15T06:26:34.805 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:34.799+0000 7f40067fc700 1 --2- 172.21.15.81:0/926561254 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f3ffc02ffb0 0x7f3ffc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:26:34.805 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:34.799+0000 7f40067fc700 1 -- 172.21.15.81:0/926561254 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4000041260 con 0x7f40100f5410 2023-11-15T06:26:34.806 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:34.799+0000 7f4017afd700 1 -- 172.21.15.81:0/926561254 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4010059100 con 0x7f40100f5410 2023-11-15T06:26:34.806 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:34.799+0000 7f4015098700 1 --2- 172.21.15.81:0/926561254 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f3ffc02ffb0 0x7f3ffc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:26:34.806 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:34.799+0000 7f4015098700 1 --2- 172.21.15.81:0/926561254 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f3ffc02ffb0 0x7f3ffc032470 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f400c0096a0 tx=0x7f400c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:26:34.806 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:34.804+0000 7f40067fc700 1 -- 172.21.15.81:0/926561254 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4000012920 con 0x7f40100f5410 2023-11-15T06:26:35.011 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:35.009+0000 7f4017afd700 1 -- 172.21.15.81:0/926561254 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4010045ad0 con 0x7f40100f5410 2023-11-15T06:26:35.011 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:35.010+0000 7f40067fc700 1 -- 172.21.15.81:0/926561254 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f400002b020 con 0x7f40100f5410 2023-11-15T06:26:35.012 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:26:35.012 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:26:35.013 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:35.011+0000 7f4017afd700 1 -- 172.21.15.81:0/926561254 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f3ffc02ffb0 msgr2=0x7f3ffc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:26:35.013 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:35.011+0000 7f4017afd700 1 --2- 172.21.15.81:0/926561254 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f3ffc02ffb0 0x7f3ffc032470 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f400c0096a0 tx=0x7f400c006b40).stop 2023-11-15T06:26:35.013 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:35.011+0000 7f4017afd700 1 -- 172.21.15.81:0/926561254 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40100f5410 msgr2=0x7f40101088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:26:35.013 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:35.011+0000 7f4017afd700 1 --2- 172.21.15.81:0/926561254 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40100f5410 0x7f40101088d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f4000013510 tx=0x7f400000b040).stop 2023-11-15T06:26:35.014 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:35.011+0000 7f4017afd700 1 -- 172.21.15.81:0/926561254 shutdown_connections 2023-11-15T06:26:35.014 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:35.011+0000 7f4017afd700 1 --2- 172.21.15.81:0/926561254 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f3ffc02ffb0 0x7f3ffc032470 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:26:35.014 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:35.011+0000 7f4017afd700 1 --2- 172.21.15.81:0/926561254 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40100f5410 0x7f40101088d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:26:35.014 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:35.011+0000 7f4017afd700 1 -- 172.21.15.81:0/926561254 >> 172.21.15.81:0/926561254 conn(0x7f40100f0b70 msgr2=0x7f401005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:26:35.014 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:35.011+0000 7f4017afd700 1 -- 172.21.15.81:0/926561254 shutdown_connections 2023-11-15T06:26:35.015 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:35.011+0000 7f4017afd700 1 -- 172.21.15.81:0/926561254 wait complete. 2023-11-15T06:26:35.016 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:26:35.668 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:35 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:26:33.782997+0000 mgr.smithi062.rwurtk (mgr.14164) 21 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:26:35.668 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:35 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:26:35.010507+0000 mon.smithi062 (mon.0) 122 : audit [DBG] from='client.? 172.21.15.81:0/926561254' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:26:36.382 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:26:36.383 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:26:37.503 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:37.501+0000 7f51d1258700 1 -- 172.21.15.81:0/3163137512 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51cc0f59d0 msgr2=0x7f51cc0f5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:26:37.503 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:37.501+0000 7f51d1258700 1 --2- 172.21.15.81:0/3163137512 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51cc0f59d0 0x7f51cc0f5df0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f51b4009fb0 tx=0x7f51b4009b30).stop 2023-11-15T06:26:37.503 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:37.501+0000 7f51d1258700 1 -- 172.21.15.81:0/3163137512 shutdown_connections 2023-11-15T06:26:37.504 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:37.501+0000 7f51d1258700 1 --2- 172.21.15.81:0/3163137512 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51cc0f59d0 0x7f51cc0f5df0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:26:37.504 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:37.501+0000 7f51d1258700 1 -- 172.21.15.81:0/3163137512 >> 172.21.15.81:0/3163137512 conn(0x7f51cc0f02e0 msgr2=0x7f51cc0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:26:37.504 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:37.501+0000 7f51d1258700 1 -- 172.21.15.81:0/3163137512 shutdown_connections 2023-11-15T06:26:37.504 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:37.501+0000 7f51d1258700 1 -- 172.21.15.81:0/3163137512 wait complete. 2023-11-15T06:26:37.506 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:37.502+0000 7f51d1258700 1 Processor -- start 2023-11-15T06:26:37.506 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:37.502+0000 7f51d1258700 1 -- start start 2023-11-15T06:26:37.506 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:37.502+0000 7f51d1258700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51cc0f59d0 0x7f51cc109380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:26:37.507 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:37.502+0000 7f51d1258700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51cc0f53f0 con 0x7f51cc0f59d0 2023-11-15T06:26:37.507 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:37.503+0000 7f51cad9d700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51cc0f59d0 0x7f51cc109380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:26:37.507 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:37.503+0000 7f51cad9d700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51cc0f59d0 0x7f51cc109380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:42836/0 (socket says 172.21.15.81:42836) 2023-11-15T06:26:37.507 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:37.503+0000 7f51cad9d700 1 -- 172.21.15.81:0/1267653670 learned_addr learned my addr 172.21.15.81:0/1267653670 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:26:37.507 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:37.503+0000 7f51cad9d700 1 -- 172.21.15.81:0/1267653670 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51b4005040 con 0x7f51cc0f59d0 2023-11-15T06:26:37.508 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:37.503+0000 7f51cad9d700 1 --2- 172.21.15.81:0/1267653670 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51cc0f59d0 0x7f51cc109380 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f51b4013510 tx=0x7f51b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:26:37.508 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:37.503+0000 7f51c37fe700 1 -- 172.21.15.81:0/1267653670 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f51b4009320 con 0x7f51cc0f59d0 2023-11-15T06:26:37.508 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:37.503+0000 7f51d1258700 1 -- 172.21.15.81:0/1267653670 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51cc0f55f0 con 0x7f51cc0f59d0 2023-11-15T06:26:37.508 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:37.503+0000 7f51c37fe700 1 -- 172.21.15.81:0/1267653670 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f51b4013f00 con 0x7f51cc0f59d0 2023-11-15T06:26:37.508 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:37.503+0000 7f51c37fe700 1 -- 172.21.15.81:0/1267653670 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f51b4014d90 con 0x7f51cc0f59d0 2023-11-15T06:26:37.508 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:37.503+0000 7f51d1258700 1 -- 172.21.15.81:0/1267653670 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51cc109ab0 con 0x7f51cc0f59d0 2023-11-15T06:26:37.509 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:37.504+0000 7f51c37fe700 1 -- 172.21.15.81:0/1267653670 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f51b40095f0 con 0x7f51cc0f59d0 2023-11-15T06:26:37.509 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:37.504+0000 7f51d1258700 1 -- 172.21.15.81:0/1267653670 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51cc045ad0 con 0x7f51cc0f59d0 2023-11-15T06:26:37.509 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:37.504+0000 7f51c37fe700 1 --2- 172.21.15.81:0/1267653670 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f51b802ff60 0x7f51b8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:26:37.509 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:37.504+0000 7f51c37fe700 1 -- 172.21.15.81:0/1267653670 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f51b40410a0 con 0x7f51cc0f59d0 2023-11-15T06:26:37.509 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:37.504+0000 7f51ca59c700 1 --2- 172.21.15.81:0/1267653670 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f51b802ff60 0x7f51b8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:26:37.510 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:37.505+0000 7f51ca59c700 1 --2- 172.21.15.81:0/1267653670 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f51b802ff60 0x7f51b8032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f51bc0096a0 tx=0x7f51bc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:26:37.510 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:37.509+0000 7f51c37fe700 1 -- 172.21.15.81:0/1267653670 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f51b406d2b0 con 0x7f51cc0f59d0 2023-11-15T06:26:37.668 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:37 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:26:35.783305+0000 mgr.smithi062.rwurtk (mgr.14164) 22 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:26:37.715 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:37.713+0000 7f51d1258700 1 -- 172.21.15.81:0/1267653670 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f51cc0fae50 con 0x7f51cc0f59d0 2023-11-15T06:26:37.715 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:37.714+0000 7f51c37fe700 1 -- 172.21.15.81:0/1267653670 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f51b401d380 con 0x7f51cc0f59d0 2023-11-15T06:26:37.716 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:26:37.716 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:26:37.717 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:37.715+0000 7f51d1258700 1 -- 172.21.15.81:0/1267653670 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f51b802ff60 msgr2=0x7f51b8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:26:37.717 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:37.715+0000 7f51d1258700 1 --2- 172.21.15.81:0/1267653670 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f51b802ff60 0x7f51b8032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f51bc0096a0 tx=0x7f51bc006b40).stop 2023-11-15T06:26:37.718 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:37.715+0000 7f51d1258700 1 -- 172.21.15.81:0/1267653670 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51cc0f59d0 msgr2=0x7f51cc109380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:26:37.718 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:37.715+0000 7f51d1258700 1 --2- 172.21.15.81:0/1267653670 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51cc0f59d0 0x7f51cc109380 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f51b4013510 tx=0x7f51b400b040).stop 2023-11-15T06:26:37.718 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:37.715+0000 7f51d1258700 1 -- 172.21.15.81:0/1267653670 shutdown_connections 2023-11-15T06:26:37.718 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:37.715+0000 7f51d1258700 1 --2- 172.21.15.81:0/1267653670 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f51b802ff60 0x7f51b8032420 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:26:37.718 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:37.715+0000 7f51d1258700 1 --2- 172.21.15.81:0/1267653670 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51cc0f59d0 0x7f51cc109380 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:26:37.719 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:37.715+0000 7f51d1258700 1 -- 172.21.15.81:0/1267653670 >> 172.21.15.81:0/1267653670 conn(0x7f51cc0f02e0 msgr2=0x7f51cc0f0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:26:37.719 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:37.716+0000 7f51d1258700 1 -- 172.21.15.81:0/1267653670 shutdown_connections 2023-11-15T06:26:37.720 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:37.716+0000 7f51d1258700 1 -- 172.21.15.81:0/1267653670 wait complete. 2023-11-15T06:26:37.720 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:26:38.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:38 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:26:37.714755+0000 mon.smithi062 (mon.0) 123 : audit [DBG] from='client.? 172.21.15.81:0/1267653670' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:26:38.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:38 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:26:37.783598+0000 mgr.smithi062.rwurtk (mgr.14164) 23 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:26:39.091 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:26:39.092 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:26:40.317 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:40.314+0000 7f337135b700 1 -- 172.21.15.81:0/2858888345 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f336c0f5410 msgr2=0x7f336c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:26:40.317 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:40.314+0000 7f337135b700 1 --2- 172.21.15.81:0/2858888345 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f336c0f5410 0x7f336c0f5830 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f3360009fb0 tx=0x7f3360009b30).stop 2023-11-15T06:26:40.317 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:40.314+0000 7f337135b700 1 -- 172.21.15.81:0/2858888345 shutdown_connections 2023-11-15T06:26:40.318 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:40.314+0000 7f337135b700 1 --2- 172.21.15.81:0/2858888345 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f336c0f5410 0x7f336c0f5830 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:26:40.318 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:40.314+0000 7f337135b700 1 -- 172.21.15.81:0/2858888345 >> 172.21.15.81:0/2858888345 conn(0x7f336c0f0b70 msgr2=0x7f336c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:26:40.318 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:40.314+0000 7f337135b700 1 -- 172.21.15.81:0/2858888345 shutdown_connections 2023-11-15T06:26:40.318 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:40.314+0000 7f337135b700 1 -- 172.21.15.81:0/2858888345 wait complete. 2023-11-15T06:26:40.318 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:40.315+0000 7f337135b700 1 Processor -- start 2023-11-15T06:26:40.319 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:40.315+0000 7f337135b700 1 -- start start 2023-11-15T06:26:40.319 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:40.315+0000 7f337135b700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f336c0f5410 0x7f336c10b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:26:40.319 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:40.315+0000 7f337135b700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f336c107240 con 0x7f336c0f5410 2023-11-15T06:26:40.319 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:40.315+0000 7f336affd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f336c0f5410 0x7f336c10b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:26:40.319 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:40.315+0000 7f336affd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f336c0f5410 0x7f336c10b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:42856/0 (socket says 172.21.15.81:42856) 2023-11-15T06:26:40.320 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:40.315+0000 7f336affd700 1 -- 172.21.15.81:0/1523146757 learned_addr learned my addr 172.21.15.81:0/1523146757 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:26:40.320 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:40.316+0000 7f336affd700 1 -- 172.21.15.81:0/1523146757 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3360005040 con 0x7f336c0f5410 2023-11-15T06:26:40.320 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:40.316+0000 7f336affd700 1 --2- 172.21.15.81:0/1523146757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f336c0f5410 0x7f336c10b850 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f3360013510 tx=0x7f336000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:26:40.322 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:40.316+0000 7f33537fe700 1 -- 172.21.15.81:0/1523146757 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3360009320 con 0x7f336c0f5410 2023-11-15T06:26:40.322 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:40.316+0000 7f33537fe700 1 -- 172.21.15.81:0/1523146757 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3360013f00 con 0x7f336c0f5410 2023-11-15T06:26:40.322 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:40.316+0000 7f337135b700 1 -- 172.21.15.81:0/1523146757 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f336c060220 con 0x7f336c0f5410 2023-11-15T06:26:40.323 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:40.316+0000 7f33537fe700 1 -- 172.21.15.81:0/1523146757 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3360014d90 con 0x7f336c0f5410 2023-11-15T06:26:40.323 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:40.316+0000 7f337135b700 1 -- 172.21.15.81:0/1523146757 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f336c10bd40 con 0x7f336c0f5410 2023-11-15T06:26:40.323 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:40.317+0000 7f33537fe700 1 -- 172.21.15.81:0/1523146757 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3360014420 con 0x7f336c0f5410 2023-11-15T06:26:40.323 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:40.317+0000 7f337135b700 1 -- 172.21.15.81:0/1523146757 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f336c045ad0 con 0x7f336c0f5410 2023-11-15T06:26:40.323 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:40.317+0000 7f33537fe700 1 --2- 172.21.15.81:0/1523146757 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f335402ff60 0x7f3354032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:26:40.323 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:40.317+0000 7f33537fe700 1 -- 172.21.15.81:0/1523146757 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3360041000 con 0x7f336c0f5410 2023-11-15T06:26:40.324 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:40.317+0000 7f336a7fc700 1 --2- 172.21.15.81:0/1523146757 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f335402ff60 0x7f3354032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:26:40.324 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:40.318+0000 7f336a7fc700 1 --2- 172.21.15.81:0/1523146757 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f335402ff60 0x7f3354032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f335c0096a0 tx=0x7f335c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:26:40.324 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:40.322+0000 7f33537fe700 1 -- 172.21.15.81:0/1523146757 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f33600095f0 con 0x7f336c0f5410 2023-11-15T06:26:40.528 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:40.526+0000 7f337135b700 1 -- 172.21.15.81:0/1523146757 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f336c0fae60 con 0x7f336c0f5410 2023-11-15T06:26:40.528 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:40.527+0000 7f33537fe700 1 -- 172.21.15.81:0/1523146757 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f336001db70 con 0x7f336c0f5410 2023-11-15T06:26:40.529 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:26:40.529 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:26:40.530 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:40.528+0000 7f337135b700 1 -- 172.21.15.81:0/1523146757 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f335402ff60 msgr2=0x7f3354032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:26:40.530 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:40.528+0000 7f337135b700 1 --2- 172.21.15.81:0/1523146757 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f335402ff60 0x7f3354032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f335c0096a0 tx=0x7f335c006b40).stop 2023-11-15T06:26:40.530 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:40.528+0000 7f337135b700 1 -- 172.21.15.81:0/1523146757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f336c0f5410 msgr2=0x7f336c10b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:26:40.530 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:40.528+0000 7f337135b700 1 --2- 172.21.15.81:0/1523146757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f336c0f5410 0x7f336c10b850 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f3360013510 tx=0x7f336000b040).stop 2023-11-15T06:26:40.531 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:40.528+0000 7f337135b700 1 -- 172.21.15.81:0/1523146757 shutdown_connections 2023-11-15T06:26:40.531 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:40.528+0000 7f337135b700 1 --2- 172.21.15.81:0/1523146757 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f335402ff60 0x7f3354032420 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:26:40.531 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:40.528+0000 7f337135b700 1 --2- 172.21.15.81:0/1523146757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f336c0f5410 0x7f336c10b850 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:26:40.531 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:40.528+0000 7f337135b700 1 -- 172.21.15.81:0/1523146757 >> 172.21.15.81:0/1523146757 conn(0x7f336c0f0b70 msgr2=0x7f336c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:26:40.531 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:40.528+0000 7f337135b700 1 -- 172.21.15.81:0/1523146757 shutdown_connections 2023-11-15T06:26:40.532 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:40.528+0000 7f337135b700 1 -- 172.21.15.81:0/1523146757 wait complete. 2023-11-15T06:26:40.532 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:26:41.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:40 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:26:39.783806+0000 mgr.smithi062.rwurtk (mgr.14164) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:26:41.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:40 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:26:40.527527+0000 mon.smithi062 (mon.0) 124 : audit [DBG] from='client.? 172.21.15.81:0/1523146757' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:26:41.888 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:26:41.888 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:26:43.086 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:43.083+0000 7f38d0c77700 1 -- 172.21.15.81:0/1323931778 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38cc0f5410 msgr2=0x7f38cc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:26:43.087 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:43.083+0000 7f38d0c77700 1 --2- 172.21.15.81:0/1323931778 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38cc0f5410 0x7f38cc0f5830 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f38bc009fb0 tx=0x7f38bc009b30).stop 2023-11-15T06:26:43.087 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:43.083+0000 7f38d0c77700 1 -- 172.21.15.81:0/1323931778 shutdown_connections 2023-11-15T06:26:43.087 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:43.083+0000 7f38d0c77700 1 --2- 172.21.15.81:0/1323931778 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38cc0f5410 0x7f38cc0f5830 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:26:43.088 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:43.083+0000 7f38d0c77700 1 -- 172.21.15.81:0/1323931778 >> 172.21.15.81:0/1323931778 conn(0x7f38cc0f0b70 msgr2=0x7f38cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:26:43.088 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:43.083+0000 7f38d0c77700 1 -- 172.21.15.81:0/1323931778 shutdown_connections 2023-11-15T06:26:43.088 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:43.083+0000 7f38d0c77700 1 -- 172.21.15.81:0/1323931778 wait complete. 2023-11-15T06:26:43.088 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:43.084+0000 7f38d0c77700 1 Processor -- start 2023-11-15T06:26:43.088 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:43.084+0000 7f38d0c77700 1 -- start start 2023-11-15T06:26:43.089 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:43.084+0000 7f38d0c77700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38cc0f5410 0x7f38cc0fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:26:43.089 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:43.084+0000 7f38d0c77700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38cc0fb820 con 0x7f38cc0f5410 2023-11-15T06:26:43.089 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:43.084+0000 7f38ca59c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38cc0f5410 0x7f38cc0fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:26:43.089 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:43.084+0000 7f38ca59c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38cc0f5410 0x7f38cc0fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:42406/0 (socket says 172.21.15.81:42406) 2023-11-15T06:26:43.089 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:43.084+0000 7f38ca59c700 1 -- 172.21.15.81:0/217880896 learned_addr learned my addr 172.21.15.81:0/217880896 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:26:43.090 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:43.084+0000 7f38ca59c700 1 -- 172.21.15.81:0/217880896 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38bc005040 con 0x7f38cc0f5410 2023-11-15T06:26:43.090 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:43.085+0000 7f38ca59c700 1 --2- 172.21.15.81:0/217880896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38cc0f5410 0x7f38cc0fd3a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f38bc009f80 tx=0x7f38bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:26:43.090 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:43.085+0000 7f38baffd700 1 -- 172.21.15.81:0/217880896 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38bc009320 con 0x7f38cc0f5410 2023-11-15T06:26:43.090 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:43.085+0000 7f38d0c77700 1 -- 172.21.15.81:0/217880896 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38cc0fb9c0 con 0x7f38cc0f5410 2023-11-15T06:26:43.090 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:43.085+0000 7f38baffd700 1 -- 172.21.15.81:0/217880896 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f38bc013ec0 con 0x7f38cc0f5410 2023-11-15T06:26:43.091 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:43.085+0000 7f38baffd700 1 -- 172.21.15.81:0/217880896 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38bc01ece0 con 0x7f38cc0f5410 2023-11-15T06:26:43.091 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:43.085+0000 7f38d0c77700 1 -- 172.21.15.81:0/217880896 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38cc0fbe50 con 0x7f38cc0f5410 2023-11-15T06:26:43.092 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:43.085+0000 7f38baffd700 1 -- 172.21.15.81:0/217880896 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f38bc01e420 con 0x7f38cc0f5410 2023-11-15T06:26:43.092 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:43.086+0000 7f38d0c77700 1 -- 172.21.15.81:0/217880896 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38cc059100 con 0x7f38cc0f5410 2023-11-15T06:26:43.092 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:43.086+0000 7f38baffd700 1 --2- 172.21.15.81:0/217880896 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f38b402ff60 0x7f38b4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:26:43.093 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:43.086+0000 7f38baffd700 1 -- 172.21.15.81:0/217880896 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f38bc040e10 con 0x7f38cc0f5410 2023-11-15T06:26:43.093 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:43.086+0000 7f38c9d9b700 1 --2- 172.21.15.81:0/217880896 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f38b402ff60 0x7f38b4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:26:43.093 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:43.086+0000 7f38c9d9b700 1 --2- 172.21.15.81:0/217880896 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f38b402ff60 0x7f38b4032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f38c00096a0 tx=0x7f38c0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:26:43.093 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:43.090+0000 7f38baffd700 1 -- 172.21.15.81:0/217880896 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f38bc01e6f0 con 0x7f38cc0f5410 2023-11-15T06:26:43.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:42 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:26:41.784149+0000 mgr.smithi062.rwurtk (mgr.14164) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:26:43.297 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:43.296+0000 7f38d0c77700 1 -- 172.21.15.81:0/217880896 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f38cc0fae60 con 0x7f38cc0f5410 2023-11-15T06:26:43.298 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:43.296+0000 7f38baffd700 1 -- 172.21.15.81:0/217880896 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f38bc004490 con 0x7f38cc0f5410 2023-11-15T06:26:43.299 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:26:43.299 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:26:43.300 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:43.298+0000 7f38d0c77700 1 -- 172.21.15.81:0/217880896 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f38b402ff60 msgr2=0x7f38b4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:26:43.300 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:43.298+0000 7f38d0c77700 1 --2- 172.21.15.81:0/217880896 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f38b402ff60 0x7f38b4032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f38c00096a0 tx=0x7f38c0006b40).stop 2023-11-15T06:26:43.300 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:43.298+0000 7f38d0c77700 1 -- 172.21.15.81:0/217880896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38cc0f5410 msgr2=0x7f38cc0fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:26:43.301 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:43.298+0000 7f38d0c77700 1 --2- 172.21.15.81:0/217880896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38cc0f5410 0x7f38cc0fd3a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f38bc009f80 tx=0x7f38bc00b040).stop 2023-11-15T06:26:43.301 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:43.298+0000 7f38d0c77700 1 -- 172.21.15.81:0/217880896 shutdown_connections 2023-11-15T06:26:43.301 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:43.298+0000 7f38d0c77700 1 --2- 172.21.15.81:0/217880896 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f38b402ff60 0x7f38b4032420 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:26:43.301 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:43.298+0000 7f38d0c77700 1 --2- 172.21.15.81:0/217880896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38cc0f5410 0x7f38cc0fd3a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:26:43.301 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:43.298+0000 7f38d0c77700 1 -- 172.21.15.81:0/217880896 >> 172.21.15.81:0/217880896 conn(0x7f38cc0f0b70 msgr2=0x7f38cc05fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:26:43.302 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:43.298+0000 7f38d0c77700 1 -- 172.21.15.81:0/217880896 shutdown_connections 2023-11-15T06:26:43.302 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:43.298+0000 7f38d0c77700 1 -- 172.21.15.81:0/217880896 wait complete. 2023-11-15T06:26:43.302 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:26:44.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:43 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:26:43.296996+0000 mon.smithi062 (mon.0) 125 : audit [DBG] from='client.? 172.21.15.81:0/217880896' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:26:44.717 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:26:44.718 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:26:45.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:44 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:26:43.784541+0000 mgr.smithi062.rwurtk (mgr.14164) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:26:45.956 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:45.953+0000 7f2329d23700 1 -- 172.21.15.81:0/2361500711 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23240f6510 msgr2=0x7f23240f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:26:45.956 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:45.953+0000 7f2329d23700 1 --2- 172.21.15.81:0/2361500711 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23240f6510 0x7f23240f6930 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f230c009fb0 tx=0x7f230c009b30).stop 2023-11-15T06:26:45.956 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:45.953+0000 7f2329d23700 1 -- 172.21.15.81:0/2361500711 shutdown_connections 2023-11-15T06:26:45.956 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:45.953+0000 7f2329d23700 1 --2- 172.21.15.81:0/2361500711 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23240f6510 0x7f23240f6930 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:26:45.957 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:45.953+0000 7f2329d23700 1 -- 172.21.15.81:0/2361500711 >> 172.21.15.81:0/2361500711 conn(0x7f23240f1c70 msgr2=0x7f23240f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:26:45.957 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:45.953+0000 7f2329d23700 1 -- 172.21.15.81:0/2361500711 shutdown_connections 2023-11-15T06:26:45.957 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:45.953+0000 7f2329d23700 1 -- 172.21.15.81:0/2361500711 wait complete. 2023-11-15T06:26:45.957 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:45.954+0000 7f2329d23700 1 Processor -- start 2023-11-15T06:26:45.957 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:45.954+0000 7f2329d23700 1 -- start start 2023-11-15T06:26:45.958 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:45.954+0000 7f2329d23700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23240f6510 0x7f2324106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:26:45.958 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:45.954+0000 7f2329d23700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2324106c20 con 0x7f23240f6510 2023-11-15T06:26:45.958 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:45.954+0000 7f23237fe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23240f6510 0x7f2324106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:26:45.958 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:45.954+0000 7f23237fe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23240f6510 0x7f2324106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:42414/0 (socket says 172.21.15.81:42414) 2023-11-15T06:26:45.958 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:45.954+0000 7f23237fe700 1 -- 172.21.15.81:0/2764448202 learned_addr learned my addr 172.21.15.81:0/2764448202 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:26:45.960 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:45.955+0000 7f23237fe700 1 -- 172.21.15.81:0/2764448202 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f230c005040 con 0x7f23240f6510 2023-11-15T06:26:45.960 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:45.955+0000 7f23237fe700 1 --2- 172.21.15.81:0/2764448202 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23240f6510 0x7f2324106730 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f230c013510 tx=0x7f230c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:26:45.960 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:45.955+0000 7f231bfff700 1 -- 172.21.15.81:0/2764448202 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f230c009320 con 0x7f23240f6510 2023-11-15T06:26:45.961 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:45.955+0000 7f231bfff700 1 -- 172.21.15.81:0/2764448202 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f230c013f00 con 0x7f23240f6510 2023-11-15T06:26:45.961 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:45.955+0000 7f2329d23700 1 -- 172.21.15.81:0/2764448202 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2324106e20 con 0x7f23240f6510 2023-11-15T06:26:45.961 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:45.955+0000 7f231bfff700 1 -- 172.21.15.81:0/2764448202 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f230c014d90 con 0x7f23240f6510 2023-11-15T06:26:45.961 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:45.955+0000 7f2329d23700 1 -- 172.21.15.81:0/2764448202 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23241072b0 con 0x7f23240f6510 2023-11-15T06:26:45.962 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:45.956+0000 7f231bfff700 1 -- 172.21.15.81:0/2764448202 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f230c0095f0 con 0x7f23240f6510 2023-11-15T06:26:45.962 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:45.956+0000 7f2329d23700 1 -- 172.21.15.81:0/2764448202 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2324045ad0 con 0x7f23240f6510 2023-11-15T06:26:45.962 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:45.956+0000 7f231bfff700 1 --2- 172.21.15.81:0/2764448202 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f231002ff60 0x7f2310032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:26:45.962 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:45.956+0000 7f231bfff700 1 -- 172.21.15.81:0/2764448202 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f230c041000 con 0x7f23240f6510 2023-11-15T06:26:45.962 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:45.956+0000 7f2322ffd700 1 --2- 172.21.15.81:0/2764448202 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f231002ff60 0x7f2310032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:26:45.963 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:45.957+0000 7f2322ffd700 1 --2- 172.21.15.81:0/2764448202 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f231002ff60 0x7f2310032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f23140096a0 tx=0x7f2314006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:26:45.963 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:45.961+0000 7f231bfff700 1 -- 172.21.15.81:0/2764448202 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f230c01d7f0 con 0x7f23240f6510 2023-11-15T06:26:46.165 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:46.163+0000 7f2329d23700 1 -- 172.21.15.81:0/2764448202 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f23240fbf60 con 0x7f23240f6510 2023-11-15T06:26:46.165 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:46.164+0000 7f231bfff700 1 -- 172.21.15.81:0/2764448202 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f230c010230 con 0x7f23240f6510 2023-11-15T06:26:46.166 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:26:46.166 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:26:46.167 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:46.165+0000 7f2329d23700 1 -- 172.21.15.81:0/2764448202 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f231002ff60 msgr2=0x7f2310032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:26:46.167 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:46.165+0000 7f2329d23700 1 --2- 172.21.15.81:0/2764448202 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f231002ff60 0x7f2310032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f23140096a0 tx=0x7f2314006b40).stop 2023-11-15T06:26:46.168 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:46.165+0000 7f2329d23700 1 -- 172.21.15.81:0/2764448202 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23240f6510 msgr2=0x7f2324106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:26:46.168 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:46.165+0000 7f2329d23700 1 --2- 172.21.15.81:0/2764448202 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23240f6510 0x7f2324106730 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f230c013510 tx=0x7f230c00b040).stop 2023-11-15T06:26:46.168 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:46.165+0000 7f2329d23700 1 -- 172.21.15.81:0/2764448202 shutdown_connections 2023-11-15T06:26:46.168 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:46.165+0000 7f2329d23700 1 --2- 172.21.15.81:0/2764448202 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f231002ff60 0x7f2310032420 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:26:46.169 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:46.165+0000 7f2329d23700 1 --2- 172.21.15.81:0/2764448202 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23240f6510 0x7f2324106730 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:26:46.169 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:46.165+0000 7f2329d23700 1 -- 172.21.15.81:0/2764448202 >> 172.21.15.81:0/2764448202 conn(0x7f23240f1c70 msgr2=0x7f23240f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:26:46.169 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:46.165+0000 7f2329d23700 1 -- 172.21.15.81:0/2764448202 shutdown_connections 2023-11-15T06:26:46.169 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:46.165+0000 7f2329d23700 1 -- 172.21.15.81:0/2764448202 wait complete. 2023-11-15T06:26:46.170 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:26:47.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:46 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:26:45.784945+0000 mgr.smithi062.rwurtk (mgr.14164) 27 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:26:47.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:46 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:26:46.164631+0000 mon.smithi062 (mon.0) 126 : audit [DBG] from='client.? 172.21.15.81:0/2764448202' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:26:47.553 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:26:47.554 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:26:48.738 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:48.736+0000 7f1697cde700 1 -- 172.21.15.81:0/1677855548 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16900f5410 msgr2=0x7f16900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:26:48.738 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:48.736+0000 7f1697cde700 1 --2- 172.21.15.81:0/1677855548 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16900f5410 0x7f16900f5830 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f1680009fb0 tx=0x7f1680009b30).stop 2023-11-15T06:26:48.739 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:48.736+0000 7f1697cde700 1 -- 172.21.15.81:0/1677855548 shutdown_connections 2023-11-15T06:26:48.739 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:48.736+0000 7f1697cde700 1 --2- 172.21.15.81:0/1677855548 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16900f5410 0x7f16900f5830 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:26:48.739 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:48.736+0000 7f1697cde700 1 -- 172.21.15.81:0/1677855548 >> 172.21.15.81:0/1677855548 conn(0x7f16900f0b70 msgr2=0x7f16900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:26:48.739 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:48.736+0000 7f1697cde700 1 -- 172.21.15.81:0/1677855548 shutdown_connections 2023-11-15T06:26:48.740 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:48.736+0000 7f1697cde700 1 -- 172.21.15.81:0/1677855548 wait complete. 2023-11-15T06:26:48.740 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:48.736+0000 7f1697cde700 1 Processor -- start 2023-11-15T06:26:48.740 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:48.736+0000 7f1697cde700 1 -- start start 2023-11-15T06:26:48.740 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:48.737+0000 7f1697cde700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16900f5410 0x7f16901093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:26:48.740 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:48.737+0000 7f1697cde700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1690060280 con 0x7f16900f5410 2023-11-15T06:26:48.742 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:48.737+0000 7f1695a7a700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16900f5410 0x7f16901093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:26:48.742 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:48.737+0000 7f1695a7a700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16900f5410 0x7f16901093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:42430/0 (socket says 172.21.15.81:42430) 2023-11-15T06:26:48.743 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:48.737+0000 7f1695a7a700 1 -- 172.21.15.81:0/879678248 learned_addr learned my addr 172.21.15.81:0/879678248 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:26:48.743 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:48.737+0000 7f1695a7a700 1 -- 172.21.15.81:0/879678248 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1680005040 con 0x7f16900f5410 2023-11-15T06:26:48.743 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:48.737+0000 7f1695a7a700 1 --2- 172.21.15.81:0/879678248 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16900f5410 0x7f16901093f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f1680013510 tx=0x7f168000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:26:48.743 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:48.738+0000 7f16867fc700 1 -- 172.21.15.81:0/879678248 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1680009320 con 0x7f16900f5410 2023-11-15T06:26:48.743 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:48.738+0000 7f16867fc700 1 -- 172.21.15.81:0/879678248 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1680013f00 con 0x7f16900f5410 2023-11-15T06:26:48.744 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:48.738+0000 7f1697cde700 1 -- 172.21.15.81:0/879678248 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1690060480 con 0x7f16900f5410 2023-11-15T06:26:48.744 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:48.738+0000 7f16867fc700 1 -- 172.21.15.81:0/879678248 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1680014d90 con 0x7f16900f5410 2023-11-15T06:26:48.744 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:48.738+0000 7f1697cde700 1 -- 172.21.15.81:0/879678248 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1690109b20 con 0x7f16900f5410 2023-11-15T06:26:48.744 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:48.739+0000 7f1697cde700 1 -- 172.21.15.81:0/879678248 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1690045ad0 con 0x7f16900f5410 2023-11-15T06:26:48.745 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:48.739+0000 7f16867fc700 1 -- 172.21.15.81:0/879678248 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f16800095f0 con 0x7f16900f5410 2023-11-15T06:26:48.745 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:48.739+0000 7f16867fc700 1 --2- 172.21.15.81:0/879678248 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f167c02ffb0 0x7f167c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:26:48.745 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:48.739+0000 7f16867fc700 1 -- 172.21.15.81:0/879678248 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1680040f90 con 0x7f16900f5410 2023-11-15T06:26:48.745 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:48.739+0000 7f1695279700 1 --2- 172.21.15.81:0/879678248 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f167c02ffb0 0x7f167c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:26:48.745 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:48.740+0000 7f1695279700 1 --2- 172.21.15.81:0/879678248 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f167c02ffb0 0x7f167c032470 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f168c0096a0 tx=0x7f168c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:26:48.746 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:48.744+0000 7f16867fc700 1 -- 172.21.15.81:0/879678248 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f168001d7f0 con 0x7f16900f5410 2023-11-15T06:26:48.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:48 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:26:47.785282+0000 mgr.smithi062.rwurtk (mgr.14164) 28 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:26:48.949 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:48.947+0000 7f1697cde700 1 -- 172.21.15.81:0/879678248 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f169010a370 con 0x7f16900f5410 2023-11-15T06:26:48.949 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:48.948+0000 7f16867fc700 1 -- 172.21.15.81:0/879678248 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1680010230 con 0x7f16900f5410 2023-11-15T06:26:48.950 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:26:48.950 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:26:48.952 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:48.950+0000 7f1697cde700 1 -- 172.21.15.81:0/879678248 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f167c02ffb0 msgr2=0x7f167c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:26:48.952 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:48.950+0000 7f1697cde700 1 --2- 172.21.15.81:0/879678248 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f167c02ffb0 0x7f167c032470 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f168c0096a0 tx=0x7f168c006b40).stop 2023-11-15T06:26:48.952 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:48.950+0000 7f1697cde700 1 -- 172.21.15.81:0/879678248 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16900f5410 msgr2=0x7f16901093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:26:48.953 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:48.950+0000 7f1697cde700 1 --2- 172.21.15.81:0/879678248 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16900f5410 0x7f16901093f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f1680013510 tx=0x7f168000b040).stop 2023-11-15T06:26:48.953 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:48.950+0000 7f1697cde700 1 -- 172.21.15.81:0/879678248 shutdown_connections 2023-11-15T06:26:48.953 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:48.950+0000 7f1697cde700 1 --2- 172.21.15.81:0/879678248 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f167c02ffb0 0x7f167c032470 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:26:48.953 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:48.950+0000 7f1697cde700 1 --2- 172.21.15.81:0/879678248 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16900f5410 0x7f16901093f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:26:48.953 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:48.950+0000 7f1697cde700 1 -- 172.21.15.81:0/879678248 >> 172.21.15.81:0/879678248 conn(0x7f16900f0b70 msgr2=0x7f16900f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:26:48.954 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:48.950+0000 7f1697cde700 1 -- 172.21.15.81:0/879678248 shutdown_connections 2023-11-15T06:26:48.954 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:48.950+0000 7f1697cde700 1 -- 172.21.15.81:0/879678248 wait complete. 2023-11-15T06:26:48.955 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:26:49.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:49 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:26:48.948661+0000 mon.smithi062 (mon.0) 127 : audit [DBG] from='client.? 172.21.15.81:0/879678248' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:26:50.354 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:26:50.354 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:26:50.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:50 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:26:49.785568+0000 mgr.smithi062.rwurtk (mgr.14164) 29 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:26:53.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:52 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:26:51.785945+0000 mgr.smithi062.rwurtk (mgr.14164) 30 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:26:55.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:54 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:26:53.786190+0000 mgr.smithi062.rwurtk (mgr.14164) 31 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:26:55.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:54 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:26:53.914107+0000 mon.smithi062 (mon.0) 128 : audit [INF] from='mgr.14164 172.21.15.62:0/147479783' entity='mgr.smithi062.rwurtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi062.rwurtk/mirror_snapshot_schedule"}]: dispatch 2023-11-15T06:26:55.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:54 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:26:53.916520+0000 mon.smithi062 (mon.0) 129 : audit [INF] from='mgr.14164 172.21.15.62:0/147479783' entity='mgr.smithi062.rwurtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi062.rwurtk/trash_purge_schedule"}]: dispatch 2023-11-15T06:26:55.195 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:55.193+0000 7f4554934700 1 -- 172.21.15.81:0/1272207029 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45500f6510 msgr2=0x7f45500f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:26:55.195 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:55.193+0000 7f4554934700 1 --2- 172.21.15.81:0/1272207029 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45500f6510 0x7f45500f6930 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f4540009fb0 tx=0x7f4540009b30).stop 2023-11-15T06:26:55.196 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:55.193+0000 7f4554934700 1 -- 172.21.15.81:0/1272207029 shutdown_connections 2023-11-15T06:26:55.196 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:55.193+0000 7f4554934700 1 --2- 172.21.15.81:0/1272207029 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45500f6510 0x7f45500f6930 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:26:55.196 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:55.193+0000 7f4554934700 1 -- 172.21.15.81:0/1272207029 >> 172.21.15.81:0/1272207029 conn(0x7f45500f1c70 msgr2=0x7f45500f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:26:55.196 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:55.194+0000 7f4554934700 1 -- 172.21.15.81:0/1272207029 shutdown_connections 2023-11-15T06:26:55.196 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:55.194+0000 7f4554934700 1 -- 172.21.15.81:0/1272207029 wait complete. 2023-11-15T06:26:55.198 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:55.194+0000 7f4554934700 1 Processor -- start 2023-11-15T06:26:55.199 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:55.194+0000 7f4554934700 1 -- start start 2023-11-15T06:26:55.199 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:55.194+0000 7f4554934700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45500f6510 0x7f4550106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:26:55.199 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:55.194+0000 7f4554934700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4550106c20 con 0x7f45500f6510 2023-11-15T06:26:55.200 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:55.194+0000 7f454e59c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45500f6510 0x7f4550106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:26:55.200 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:55.194+0000 7f454e59c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45500f6510 0x7f4550106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:59170/0 (socket says 172.21.15.81:59170) 2023-11-15T06:26:55.200 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:55.194+0000 7f454e59c700 1 -- 172.21.15.81:0/2857655131 learned_addr learned my addr 172.21.15.81:0/2857655131 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:26:55.200 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:55.195+0000 7f454e59c700 1 -- 172.21.15.81:0/2857655131 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4540005040 con 0x7f45500f6510 2023-11-15T06:26:55.200 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:55.195+0000 7f454e59c700 1 --2- 172.21.15.81:0/2857655131 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45500f6510 0x7f4550106730 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f4540013510 tx=0x7f454000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:26:55.201 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:55.195+0000 7f4546ffd700 1 -- 172.21.15.81:0/2857655131 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4540009320 con 0x7f45500f6510 2023-11-15T06:26:55.201 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:55.195+0000 7f4546ffd700 1 -- 172.21.15.81:0/2857655131 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4540013f00 con 0x7f45500f6510 2023-11-15T06:26:55.201 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:55.195+0000 7f4554934700 1 -- 172.21.15.81:0/2857655131 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4550106e20 con 0x7f45500f6510 2023-11-15T06:26:55.201 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:55.195+0000 7f4546ffd700 1 -- 172.21.15.81:0/2857655131 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4540014d90 con 0x7f45500f6510 2023-11-15T06:26:55.201 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:55.195+0000 7f4554934700 1 -- 172.21.15.81:0/2857655131 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45501072b0 con 0x7f45500f6510 2023-11-15T06:26:55.202 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:55.196+0000 7f4546ffd700 1 -- 172.21.15.81:0/2857655131 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f45400263a0 con 0x7f45500f6510 2023-11-15T06:26:55.202 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:55.196+0000 7f4546ffd700 1 --2- 172.21.15.81:0/2857655131 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f453c02ff60 0x7f453c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:26:55.202 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:55.196+0000 7f4546ffd700 1 -- 172.21.15.81:0/2857655131 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f45400413b0 con 0x7f45500f6510 2023-11-15T06:26:55.202 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:55.196+0000 7f4554934700 1 -- 172.21.15.81:0/2857655131 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4550045ad0 con 0x7f45500f6510 2023-11-15T06:26:55.202 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:55.196+0000 7f454dd9b700 1 --2- 172.21.15.81:0/2857655131 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f453c02ff60 0x7f453c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:26:55.203 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:55.197+0000 7f454dd9b700 1 --2- 172.21.15.81:0/2857655131 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f453c02ff60 0x7f453c032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f45380096a0 tx=0x7f4538006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:26:55.203 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:55.201+0000 7f4546ffd700 1 -- 172.21.15.81:0/2857655131 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4540012920 con 0x7f45500f6510 2023-11-15T06:26:55.410 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:55.408+0000 7f4554934700 1 -- 172.21.15.81:0/2857655131 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f45500fc050 con 0x7f45500f6510 2023-11-15T06:26:55.411 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:55.409+0000 7f4546ffd700 1 -- 172.21.15.81:0/2857655131 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f454002b030 con 0x7f45500f6510 2023-11-15T06:26:55.411 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:26:55.411 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:26:55.412 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:55.411+0000 7f4554934700 1 -- 172.21.15.81:0/2857655131 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f453c02ff60 msgr2=0x7f453c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:26:55.413 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:55.411+0000 7f4554934700 1 --2- 172.21.15.81:0/2857655131 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f453c02ff60 0x7f453c032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f45380096a0 tx=0x7f4538006b40).stop 2023-11-15T06:26:55.413 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:55.411+0000 7f4554934700 1 -- 172.21.15.81:0/2857655131 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45500f6510 msgr2=0x7f4550106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:26:55.413 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:55.411+0000 7f4554934700 1 --2- 172.21.15.81:0/2857655131 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45500f6510 0x7f4550106730 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f4540013510 tx=0x7f454000b040).stop 2023-11-15T06:26:55.413 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:55.411+0000 7f4554934700 1 -- 172.21.15.81:0/2857655131 shutdown_connections 2023-11-15T06:26:55.413 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:55.411+0000 7f4554934700 1 --2- 172.21.15.81:0/2857655131 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f453c02ff60 0x7f453c032420 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:26:55.414 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:55.411+0000 7f4554934700 1 --2- 172.21.15.81:0/2857655131 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45500f6510 0x7f4550106730 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:26:55.414 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:55.411+0000 7f4554934700 1 -- 172.21.15.81:0/2857655131 >> 172.21.15.81:0/2857655131 conn(0x7f45500f1c70 msgr2=0x7f45500f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:26:55.414 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:55.411+0000 7f4554934700 1 -- 172.21.15.81:0/2857655131 shutdown_connections 2023-11-15T06:26:55.414 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:55.411+0000 7f4554934700 1 -- 172.21.15.81:0/2857655131 wait complete. 2023-11-15T06:26:55.415 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:26:56.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:55 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:26:55.410008+0000 mon.smithi062 (mon.0) 130 : audit [DBG] from='client.? 172.21.15.81:0/2857655131' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:26:56.821 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:26:56.822 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:26:57.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:56 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:26:55.786534+0000 mgr.smithi062.rwurtk (mgr.14164) 32 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:26:58.034 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:58.032+0000 7f5434c82700 1 -- 172.21.15.81:0/2889390641 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54300601c0 msgr2=0x7f54300f91b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:26:58.035 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:58.032+0000 7f5434c82700 1 --2- 172.21.15.81:0/2889390641 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54300601c0 0x7f54300f91b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f5420009fb0 tx=0x7f5420009b30).stop 2023-11-15T06:26:58.035 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:58.032+0000 7f5434c82700 1 -- 172.21.15.81:0/2889390641 shutdown_connections 2023-11-15T06:26:58.035 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:58.032+0000 7f5434c82700 1 --2- 172.21.15.81:0/2889390641 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54300601c0 0x7f54300f91b0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:26:58.036 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:58.032+0000 7f5434c82700 1 -- 172.21.15.81:0/2889390641 >> 172.21.15.81:0/2889390641 conn(0x7f54300f0b70 msgr2=0x7f54300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:26:58.036 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:58.032+0000 7f5434c82700 1 -- 172.21.15.81:0/2889390641 shutdown_connections 2023-11-15T06:26:58.036 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:58.032+0000 7f5434c82700 1 -- 172.21.15.81:0/2889390641 wait complete. 2023-11-15T06:26:58.036 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:58.033+0000 7f5434c82700 1 Processor -- start 2023-11-15T06:26:58.036 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:58.033+0000 7f5434c82700 1 -- start start 2023-11-15T06:26:58.037 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:58.033+0000 7f5434c82700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54300601c0 0x7f54301066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:26:58.037 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:58.033+0000 7f5434c82700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5430106bc0 con 0x7f54300601c0 2023-11-15T06:26:58.037 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:58.033+0000 7f542e59c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54300601c0 0x7f54301066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:26:58.037 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:58.033+0000 7f542e59c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54300601c0 0x7f54301066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:59192/0 (socket says 172.21.15.81:59192) 2023-11-15T06:26:58.037 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:58.033+0000 7f542e59c700 1 -- 172.21.15.81:0/1328515523 learned_addr learned my addr 172.21.15.81:0/1328515523 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:26:58.038 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:58.033+0000 7f542e59c700 1 -- 172.21.15.81:0/1328515523 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5420005040 con 0x7f54300601c0 2023-11-15T06:26:58.039 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:58.033+0000 7f542e59c700 1 --2- 172.21.15.81:0/1328515523 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54300601c0 0x7f54301066d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f5420013510 tx=0x7f542000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:26:58.039 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:58.034+0000 7f541effd700 1 -- 172.21.15.81:0/1328515523 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5420009320 con 0x7f54300601c0 2023-11-15T06:26:58.040 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:58.034+0000 7f5434c82700 1 -- 172.21.15.81:0/1328515523 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5430106dc0 con 0x7f54300601c0 2023-11-15T06:26:58.040 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:58.034+0000 7f541effd700 1 -- 172.21.15.81:0/1328515523 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5420013f00 con 0x7f54300601c0 2023-11-15T06:26:58.040 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:58.034+0000 7f541effd700 1 -- 172.21.15.81:0/1328515523 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5420014dc0 con 0x7f54300601c0 2023-11-15T06:26:58.040 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:58.034+0000 7f5434c82700 1 -- 172.21.15.81:0/1328515523 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5430107250 con 0x7f54300601c0 2023-11-15T06:26:58.040 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:58.034+0000 7f541effd700 1 -- 172.21.15.81:0/1328515523 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f54200095f0 con 0x7f54300601c0 2023-11-15T06:26:58.041 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:58.034+0000 7f5434c82700 1 -- 172.21.15.81:0/1328515523 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5430045ad0 con 0x7f54300601c0 2023-11-15T06:26:58.041 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:58.035+0000 7f541effd700 1 --2- 172.21.15.81:0/1328515523 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f541802ffb0 0x7f5418032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:26:58.041 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:58.035+0000 7f541effd700 1 -- 172.21.15.81:0/1328515523 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5420040ff0 con 0x7f54300601c0 2023-11-15T06:26:58.041 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:58.035+0000 7f542dd9b700 1 --2- 172.21.15.81:0/1328515523 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f541802ffb0 0x7f5418032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:26:58.041 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:58.035+0000 7f542dd9b700 1 --2- 172.21.15.81:0/1328515523 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f541802ffb0 0x7f5418032470 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f54240096a0 tx=0x7f5424006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:26:58.042 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:58.039+0000 7f541effd700 1 -- 172.21.15.81:0/1328515523 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f542006d2b0 con 0x7f54300601c0 2023-11-15T06:26:58.247 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:58.245+0000 7f5434c82700 1 -- 172.21.15.81:0/1328515523 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5430108100 con 0x7f54300601c0 2023-11-15T06:26:58.248 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:58.246+0000 7f541effd700 1 -- 172.21.15.81:0/1328515523 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5420014420 con 0x7f54300601c0 2023-11-15T06:26:58.248 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:26:58.249 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:26:58.250 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:58.247+0000 7f5434c82700 1 -- 172.21.15.81:0/1328515523 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f541802ffb0 msgr2=0x7f5418032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:26:58.250 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:58.247+0000 7f5434c82700 1 --2- 172.21.15.81:0/1328515523 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f541802ffb0 0x7f5418032470 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f54240096a0 tx=0x7f5424006b40).stop 2023-11-15T06:26:58.251 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:58.247+0000 7f5434c82700 1 -- 172.21.15.81:0/1328515523 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54300601c0 msgr2=0x7f54301066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:26:58.251 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:58.247+0000 7f5434c82700 1 --2- 172.21.15.81:0/1328515523 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54300601c0 0x7f54301066d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f5420013510 tx=0x7f542000b040).stop 2023-11-15T06:26:58.251 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:58.247+0000 7f5434c82700 1 -- 172.21.15.81:0/1328515523 shutdown_connections 2023-11-15T06:26:58.253 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:58.247+0000 7f5434c82700 1 --2- 172.21.15.81:0/1328515523 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f541802ffb0 0x7f5418032470 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:26:58.253 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:58.247+0000 7f5434c82700 1 --2- 172.21.15.81:0/1328515523 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54300601c0 0x7f54301066d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:26:58.254 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:58.247+0000 7f5434c82700 1 -- 172.21.15.81:0/1328515523 >> 172.21.15.81:0/1328515523 conn(0x7f54300f0b70 msgr2=0x7f54300f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:26:58.254 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:58.247+0000 7f5434c82700 1 -- 172.21.15.81:0/1328515523 shutdown_connections 2023-11-15T06:26:58.254 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:26:58.248+0000 7f5434c82700 1 -- 172.21.15.81:0/1328515523 wait complete. 2023-11-15T06:26:58.255 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:26:58.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:58 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:26:57.786858+0000 mgr.smithi062.rwurtk (mgr.14164) 33 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:26:58.919 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:26:58 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:26:58.246747+0000 mon.smithi062 (mon.0) 131 : audit [DBG] from='client.? 172.21.15.81:0/1328515523' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:26:59.662 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:26:59.663 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:27:01.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:00 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:26:59.787090+0000 mgr.smithi062.rwurtk (mgr.14164) 34 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:27:01.213 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:01.211+0000 7f3136fb1700 1 -- 172.21.15.81:0/1139769748 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f31300f5410 msgr2=0x7f31300f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:01.213 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:01.211+0000 7f3136fb1700 1 --2- 172.21.15.81:0/1139769748 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f31300f5410 0x7f31300f5830 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f3124009fb0 tx=0x7f3124009b30).stop 2023-11-15T06:27:01.214 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:01.211+0000 7f3136fb1700 1 -- 172.21.15.81:0/1139769748 shutdown_connections 2023-11-15T06:27:01.214 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:01.211+0000 7f3136fb1700 1 --2- 172.21.15.81:0/1139769748 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f31300f5410 0x7f31300f5830 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:01.214 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:01.211+0000 7f3136fb1700 1 -- 172.21.15.81:0/1139769748 >> 172.21.15.81:0/1139769748 conn(0x7f31300f0b70 msgr2=0x7f31300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:27:01.214 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:01.211+0000 7f3136fb1700 1 -- 172.21.15.81:0/1139769748 shutdown_connections 2023-11-15T06:27:01.215 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:01.211+0000 7f3136fb1700 1 -- 172.21.15.81:0/1139769748 wait complete. 2023-11-15T06:27:01.215 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:01.212+0000 7f3136fb1700 1 Processor -- start 2023-11-15T06:27:01.215 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:01.212+0000 7f3136fb1700 1 -- start start 2023-11-15T06:27:01.217 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:01.212+0000 7f3136fb1700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f31300f5410 0x7f3130108940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:27:01.217 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:01.212+0000 7f3136fb1700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3130108e30 con 0x7f31300f5410 2023-11-15T06:27:01.217 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:01.212+0000 7f3134d4d700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f31300f5410 0x7f3130108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:27:01.217 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:01.212+0000 7f3134d4d700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f31300f5410 0x7f3130108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:59218/0 (socket says 172.21.15.81:59218) 2023-11-15T06:27:01.218 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:01.212+0000 7f3134d4d700 1 -- 172.21.15.81:0/2479229466 learned_addr learned my addr 172.21.15.81:0/2479229466 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:27:01.218 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:01.213+0000 7f3134d4d700 1 -- 172.21.15.81:0/2479229466 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3124005040 con 0x7f31300f5410 2023-11-15T06:27:01.218 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:01.213+0000 7f3134d4d700 1 --2- 172.21.15.81:0/2479229466 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f31300f5410 0x7f3130108940 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f3124013510 tx=0x7f312400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:27:01.218 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:01.213+0000 7f312d7fa700 1 -- 172.21.15.81:0/2479229466 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3124009320 con 0x7f31300f5410 2023-11-15T06:27:01.219 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:01.213+0000 7f312d7fa700 1 -- 172.21.15.81:0/2479229466 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3124013f00 con 0x7f31300f5410 2023-11-15T06:27:01.219 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:01.213+0000 7f3136fb1700 1 -- 172.21.15.81:0/2479229466 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3130109030 con 0x7f31300f5410 2023-11-15T06:27:01.219 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:01.213+0000 7f312d7fa700 1 -- 172.21.15.81:0/2479229466 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3124014d90 con 0x7f31300f5410 2023-11-15T06:27:01.219 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:01.213+0000 7f3136fb1700 1 -- 172.21.15.81:0/2479229466 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31301094c0 con 0x7f31300f5410 2023-11-15T06:27:01.219 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:01.214+0000 7f312d7fa700 1 -- 172.21.15.81:0/2479229466 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f31240095f0 con 0x7f31300f5410 2023-11-15T06:27:01.220 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:01.214+0000 7f3136fb1700 1 -- 172.21.15.81:0/2479229466 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3130045ad0 con 0x7f31300f5410 2023-11-15T06:27:01.220 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:01.214+0000 7f312d7fa700 1 --2- 172.21.15.81:0/2479229466 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f312002ffb0 0x7f3120032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:27:01.220 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:01.214+0000 7f312d7fa700 1 -- 172.21.15.81:0/2479229466 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3124040f80 con 0x7f31300f5410 2023-11-15T06:27:01.221 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:01.214+0000 7f312ffff700 1 --2- 172.21.15.81:0/2479229466 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f312002ffb0 0x7f3120032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:27:01.221 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:01.215+0000 7f312ffff700 1 --2- 172.21.15.81:0/2479229466 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f312002ffb0 0x7f3120032470 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f311c0096a0 tx=0x7f311c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:27:01.221 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:01.219+0000 7f312d7fa700 1 -- 172.21.15.81:0/2479229466 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3124014420 con 0x7f31300f5410 2023-11-15T06:27:01.423 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:01.421+0000 7f3136fb1700 1 -- 172.21.15.81:0/2479229466 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f31300fae60 con 0x7f31300f5410 2023-11-15T06:27:01.424 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:01.422+0000 7f312d7fa700 1 -- 172.21.15.81:0/2479229466 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f312401d7f0 con 0x7f31300f5410 2023-11-15T06:27:01.424 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:27:01.425 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:27:01.425 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:01.424+0000 7f3136fb1700 1 -- 172.21.15.81:0/2479229466 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f312002ffb0 msgr2=0x7f3120032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:01.426 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:01.424+0000 7f3136fb1700 1 --2- 172.21.15.81:0/2479229466 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f312002ffb0 0x7f3120032470 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f311c0096a0 tx=0x7f311c006b40).stop 2023-11-15T06:27:01.426 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:01.424+0000 7f3136fb1700 1 -- 172.21.15.81:0/2479229466 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f31300f5410 msgr2=0x7f3130108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:01.426 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:01.424+0000 7f3136fb1700 1 --2- 172.21.15.81:0/2479229466 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f31300f5410 0x7f3130108940 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f3124013510 tx=0x7f312400b040).stop 2023-11-15T06:27:01.426 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:01.424+0000 7f3136fb1700 1 -- 172.21.15.81:0/2479229466 shutdown_connections 2023-11-15T06:27:01.427 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:01.424+0000 7f3136fb1700 1 --2- 172.21.15.81:0/2479229466 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f312002ffb0 0x7f3120032470 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:01.427 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:01.424+0000 7f3136fb1700 1 --2- 172.21.15.81:0/2479229466 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f31300f5410 0x7f3130108940 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:01.427 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:01.424+0000 7f3136fb1700 1 -- 172.21.15.81:0/2479229466 >> 172.21.15.81:0/2479229466 conn(0x7f31300f0b70 msgr2=0x7f31300f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:27:01.427 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:01.424+0000 7f3136fb1700 1 -- 172.21.15.81:0/2479229466 shutdown_connections 2023-11-15T06:27:01.427 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:01.424+0000 7f3136fb1700 1 -- 172.21.15.81:0/2479229466 wait complete. 2023-11-15T06:27:01.428 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:27:02.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:01 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:27:01.422787+0000 mon.smithi062 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.81:0/2479229466' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:27:02.803 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:27:02.804 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:27:03.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:27:01.787456+0000 mgr.smithi062.rwurtk (mgr.14164) 35 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:27:04.004 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:04.001+0000 7fee01262700 1 -- 172.21.15.81:0/3196093120 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fedfc0f51a0 msgr2=0x7fedfc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:04.005 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:04.001+0000 7fee01262700 1 --2- 172.21.15.81:0/3196093120 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fedfc0f51a0 0x7fedfc0f55c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fedf0009fb0 tx=0x7fedf0009b30).stop 2023-11-15T06:27:04.005 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:04.001+0000 7fee01262700 1 -- 172.21.15.81:0/3196093120 shutdown_connections 2023-11-15T06:27:04.005 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:04.001+0000 7fee01262700 1 --2- 172.21.15.81:0/3196093120 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fedfc0f51a0 0x7fedfc0f55c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:04.005 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:04.001+0000 7fee01262700 1 -- 172.21.15.81:0/3196093120 >> 172.21.15.81:0/3196093120 conn(0x7fedfc0f09a0 msgr2=0x7fedfc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:27:04.005 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:04.001+0000 7fee01262700 1 -- 172.21.15.81:0/3196093120 shutdown_connections 2023-11-15T06:27:04.006 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:04.001+0000 7fee01262700 1 -- 172.21.15.81:0/3196093120 wait complete. 2023-11-15T06:27:04.006 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:04.001+0000 7fee01262700 1 Processor -- start 2023-11-15T06:27:04.006 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:04.001+0000 7fee01262700 1 -- start start 2023-11-15T06:27:04.006 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:04.001+0000 7fee01262700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fedfc0f51a0 0x7fedfc1086c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:27:04.007 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:04.001+0000 7fee01262700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedfc108bb0 con 0x7fedfc0f51a0 2023-11-15T06:27:04.007 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:04.002+0000 7fedfbfff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fedfc0f51a0 0x7fedfc1086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:27:04.007 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:04.002+0000 7fedfbfff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fedfc0f51a0 0x7fedfc1086c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:41414/0 (socket says 172.21.15.81:41414) 2023-11-15T06:27:04.007 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:04.002+0000 7fedfbfff700 1 -- 172.21.15.81:0/3084029260 learned_addr learned my addr 172.21.15.81:0/3084029260 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:27:04.007 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:04.002+0000 7fedfbfff700 1 -- 172.21.15.81:0/3084029260 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf0005040 con 0x7fedfc0f51a0 2023-11-15T06:27:04.008 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:04.002+0000 7fedfbfff700 1 --2- 172.21.15.81:0/3084029260 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fedfc0f51a0 0x7fedfc1086c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fedf0013510 tx=0x7fedf000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:27:04.009 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:04.002+0000 7fedf8ff9700 1 -- 172.21.15.81:0/3084029260 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fedf0009320 con 0x7fedfc0f51a0 2023-11-15T06:27:04.009 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:04.002+0000 7fee01262700 1 -- 172.21.15.81:0/3084029260 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedfc108db0 con 0x7fedfc0f51a0 2023-11-15T06:27:04.009 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:04.002+0000 7fedf8ff9700 1 -- 172.21.15.81:0/3084029260 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fedf0013f00 con 0x7fedfc0f51a0 2023-11-15T06:27:04.009 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:04.002+0000 7fedf8ff9700 1 -- 172.21.15.81:0/3084029260 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fedf0014dc0 con 0x7fedfc0f51a0 2023-11-15T06:27:04.010 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:04.002+0000 7fee01262700 1 -- 172.21.15.81:0/3084029260 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedfc109240 con 0x7fedfc0f51a0 2023-11-15T06:27:04.010 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:04.003+0000 7fedf8ff9700 1 -- 172.21.15.81:0/3084029260 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fedf001d3d0 con 0x7fedfc0f51a0 2023-11-15T06:27:04.010 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:04.003+0000 7fee01262700 1 -- 172.21.15.81:0/3084029260 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedfc045ad0 con 0x7fedfc0f51a0 2023-11-15T06:27:04.010 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:04.003+0000 7fedf8ff9700 1 --2- 172.21.15.81:0/3084029260 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fede402ff60 0x7fede4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:27:04.010 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:04.003+0000 7fedf8ff9700 1 -- 172.21.15.81:0/3084029260 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fedf00412f0 con 0x7fedfc0f51a0 2023-11-15T06:27:04.011 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:04.004+0000 7fedfb7fe700 1 --2- 172.21.15.81:0/3084029260 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fede402ff60 0x7fede4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:27:04.011 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:04.004+0000 7fedfb7fe700 1 --2- 172.21.15.81:0/3084029260 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fede402ff60 0x7fede4032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fedec0096a0 tx=0x7fedec006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:27:04.011 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:04.008+0000 7fedf8ff9700 1 -- 172.21.15.81:0/3084029260 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fedf00126f0 con 0x7fedfc0f51a0 2023-11-15T06:27:04.217 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:04.214+0000 7fee01262700 1 -- 172.21.15.81:0/3084029260 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fedfc10a0f0 con 0x7fedfc0f51a0 2023-11-15T06:27:04.217 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:04.215+0000 7fedf8ff9700 1 -- 172.21.15.81:0/3084029260 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fedf0012330 con 0x7fedfc0f51a0 2023-11-15T06:27:04.218 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:27:04.218 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:27:04.220 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:04.217+0000 7fee01262700 1 -- 172.21.15.81:0/3084029260 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fede402ff60 msgr2=0x7fede4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:04.220 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:04.217+0000 7fee01262700 1 --2- 172.21.15.81:0/3084029260 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fede402ff60 0x7fede4032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fedec0096a0 tx=0x7fedec006b40).stop 2023-11-15T06:27:04.220 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:04.217+0000 7fee01262700 1 -- 172.21.15.81:0/3084029260 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fedfc0f51a0 msgr2=0x7fedfc1086c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:04.220 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:04.217+0000 7fee01262700 1 --2- 172.21.15.81:0/3084029260 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fedfc0f51a0 0x7fedfc1086c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fedf0013510 tx=0x7fedf000b040).stop 2023-11-15T06:27:04.221 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:04.217+0000 7fee01262700 1 -- 172.21.15.81:0/3084029260 shutdown_connections 2023-11-15T06:27:04.221 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:04.217+0000 7fee01262700 1 --2- 172.21.15.81:0/3084029260 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fede402ff60 0x7fede4032420 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:04.221 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:04.217+0000 7fee01262700 1 --2- 172.21.15.81:0/3084029260 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fedfc0f51a0 0x7fedfc1086c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:04.221 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:04.217+0000 7fee01262700 1 -- 172.21.15.81:0/3084029260 >> 172.21.15.81:0/3084029260 conn(0x7fedfc0f09a0 msgr2=0x7fedfc05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:27:04.222 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:04.217+0000 7fee01262700 1 -- 172.21.15.81:0/3084029260 shutdown_connections 2023-11-15T06:27:04.222 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:04.217+0000 7fee01262700 1 -- 172.21.15.81:0/3084029260 wait complete. 2023-11-15T06:27:04.222 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:27:05.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:04 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:27:03.787839+0000 mgr.smithi062.rwurtk (mgr.14164) 36 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:27:05.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:04 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:27:04.216837+0000 mon.smithi062 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.81:0/3084029260' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:27:05.615 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:27:05.615 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:27:06.816 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:06.813+0000 7f4255eb4700 1 -- 172.21.15.81:0/885994721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4250060220 msgr2=0x7f42500f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:06.816 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:06.813+0000 7f4255eb4700 1 --2- 172.21.15.81:0/885994721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4250060220 0x7f42500f9150 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f4238009fb0 tx=0x7f4238009b30).stop 2023-11-15T06:27:06.816 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:06.813+0000 7f4255eb4700 1 -- 172.21.15.81:0/885994721 shutdown_connections 2023-11-15T06:27:06.817 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:06.813+0000 7f4255eb4700 1 --2- 172.21.15.81:0/885994721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4250060220 0x7f42500f9150 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:06.817 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:06.813+0000 7f4255eb4700 1 -- 172.21.15.81:0/885994721 >> 172.21.15.81:0/885994721 conn(0x7f42500f0b70 msgr2=0x7f42500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:27:06.817 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:06.813+0000 7f4255eb4700 1 -- 172.21.15.81:0/885994721 shutdown_connections 2023-11-15T06:27:06.817 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:06.815+0000 7f4255eb4700 1 -- 172.21.15.81:0/885994721 wait complete. 2023-11-15T06:27:06.819 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:06.816+0000 7f4255eb4700 1 Processor -- start 2023-11-15T06:27:06.819 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:06.816+0000 7f4255eb4700 1 -- start start 2023-11-15T06:27:06.819 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:06.816+0000 7f4255eb4700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4250060220 0x7f42500f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:27:06.819 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:06.816+0000 7f4255eb4700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42500f5050 con 0x7f4250060220 2023-11-15T06:27:06.819 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:06.816+0000 7f424f7fe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4250060220 0x7f42500f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:27:06.820 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:06.816+0000 7f424f7fe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4250060220 0x7f42500f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:41418/0 (socket says 172.21.15.81:41418) 2023-11-15T06:27:06.820 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:06.816+0000 7f424f7fe700 1 -- 172.21.15.81:0/2915182449 learned_addr learned my addr 172.21.15.81:0/2915182449 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:27:06.821 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:06.817+0000 7f424f7fe700 1 -- 172.21.15.81:0/2915182449 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4238005040 con 0x7f4250060220 2023-11-15T06:27:06.821 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:06.817+0000 7f424f7fe700 1 --2- 172.21.15.81:0/2915182449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4250060220 0x7f42500f6bd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f4238013510 tx=0x7f423800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:27:06.822 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:06.817+0000 7f4247fff700 1 -- 172.21.15.81:0/2915182449 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4238009320 con 0x7f4250060220 2023-11-15T06:27:06.822 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:06.817+0000 7f4255eb4700 1 -- 172.21.15.81:0/2915182449 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42500f51f0 con 0x7f4250060220 2023-11-15T06:27:06.822 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:06.817+0000 7f4247fff700 1 -- 172.21.15.81:0/2915182449 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4238013f00 con 0x7f4250060220 2023-11-15T06:27:06.822 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:06.817+0000 7f4255eb4700 1 -- 172.21.15.81:0/2915182449 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42500f5680 con 0x7f4250060220 2023-11-15T06:27:06.823 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:06.817+0000 7f4247fff700 1 -- 172.21.15.81:0/2915182449 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4238014dc0 con 0x7f4250060220 2023-11-15T06:27:06.823 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:06.818+0000 7f4247fff700 1 -- 172.21.15.81:0/2915182449 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f423801d3d0 con 0x7f4250060220 2023-11-15T06:27:06.823 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:06.818+0000 7f4255eb4700 1 -- 172.21.15.81:0/2915182449 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4250059100 con 0x7f4250060220 2023-11-15T06:27:06.824 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:06.818+0000 7f4247fff700 1 --2- 172.21.15.81:0/2915182449 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f423c02ff60 0x7f423c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:27:06.824 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:06.818+0000 7f4247fff700 1 -- 172.21.15.81:0/2915182449 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4238041030 con 0x7f4250060220 2023-11-15T06:27:06.824 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:06.818+0000 7f424effd700 1 --2- 172.21.15.81:0/2915182449 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f423c02ff60 0x7f423c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:27:06.824 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:06.819+0000 7f424effd700 1 --2- 172.21.15.81:0/2915182449 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f423c02ff60 0x7f423c032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f42400096a0 tx=0x7f4240006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:27:06.826 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:06.823+0000 7f4247fff700 1 -- 172.21.15.81:0/2915182449 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f42380126f0 con 0x7f4250060220 2023-11-15T06:27:07.028 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:07.026+0000 7f4255eb4700 1 -- 172.21.15.81:0/2915182449 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4250045ad0 con 0x7f4250060220 2023-11-15T06:27:07.029 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:07.027+0000 7f4247fff700 1 -- 172.21.15.81:0/2915182449 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4238012330 con 0x7f4250060220 2023-11-15T06:27:07.030 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:27:07.030 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:27:07.031 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:07.028+0000 7f4255eb4700 1 -- 172.21.15.81:0/2915182449 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f423c02ff60 msgr2=0x7f423c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:07.031 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:07.028+0000 7f4255eb4700 1 --2- 172.21.15.81:0/2915182449 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f423c02ff60 0x7f423c032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f42400096a0 tx=0x7f4240006b40).stop 2023-11-15T06:27:07.031 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:07.028+0000 7f4255eb4700 1 -- 172.21.15.81:0/2915182449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4250060220 msgr2=0x7f42500f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:07.032 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:07.028+0000 7f4255eb4700 1 --2- 172.21.15.81:0/2915182449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4250060220 0x7f42500f6bd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f4238013510 tx=0x7f423800b040).stop 2023-11-15T06:27:07.032 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:07.028+0000 7f4255eb4700 1 -- 172.21.15.81:0/2915182449 shutdown_connections 2023-11-15T06:27:07.032 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:07.028+0000 7f4255eb4700 1 --2- 172.21.15.81:0/2915182449 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f423c02ff60 0x7f423c032420 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:07.032 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:07.028+0000 7f4255eb4700 1 --2- 172.21.15.81:0/2915182449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4250060220 0x7f42500f6bd0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:07.032 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:07.028+0000 7f4255eb4700 1 -- 172.21.15.81:0/2915182449 >> 172.21.15.81:0/2915182449 conn(0x7f42500f0b70 msgr2=0x7f42500f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:27:07.033 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:07.028+0000 7f4255eb4700 1 -- 172.21.15.81:0/2915182449 shutdown_connections 2023-11-15T06:27:07.033 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:07.028+0000 7f4255eb4700 1 -- 172.21.15.81:0/2915182449 wait complete. 2023-11-15T06:27:07.033 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:27:07.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:06 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:27:05.788181+0000 mgr.smithi062.rwurtk (mgr.14164) 37 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:27:08.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:07 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:27:07.028428+0000 mon.smithi062 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.81:0/2915182449' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:27:08.414 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:27:08.415 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:27:09.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:08 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:27:07.788595+0000 mgr.smithi062.rwurtk (mgr.14164) 38 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:27:09.548 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:09.544+0000 7f38b630f700 1 -- 172.21.15.81:0/2550735970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38b00f5410 msgr2=0x7f38b00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:09.548 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:09.544+0000 7f38b630f700 1 --2- 172.21.15.81:0/2550735970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38b00f5410 0x7f38b00f5830 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f38a0009fb0 tx=0x7f38a0009b30).stop 2023-11-15T06:27:09.549 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:09.544+0000 7f38b630f700 1 -- 172.21.15.81:0/2550735970 shutdown_connections 2023-11-15T06:27:09.549 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:09.544+0000 7f38b630f700 1 --2- 172.21.15.81:0/2550735970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38b00f5410 0x7f38b00f5830 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:09.549 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:09.544+0000 7f38b630f700 1 -- 172.21.15.81:0/2550735970 >> 172.21.15.81:0/2550735970 conn(0x7f38b00f0b70 msgr2=0x7f38b00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:27:09.549 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:09.545+0000 7f38b630f700 1 -- 172.21.15.81:0/2550735970 shutdown_connections 2023-11-15T06:27:09.549 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:09.545+0000 7f38b630f700 1 -- 172.21.15.81:0/2550735970 wait complete. 2023-11-15T06:27:09.550 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:09.545+0000 7f38b630f700 1 Processor -- start 2023-11-15T06:27:09.550 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:09.545+0000 7f38b630f700 1 -- start start 2023-11-15T06:27:09.550 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:09.545+0000 7f38b630f700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38b00f5410 0x7f38b0106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:27:09.550 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:09.545+0000 7f38b630f700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b0106c10 con 0x7f38b00f5410 2023-11-15T06:27:09.550 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:09.546+0000 7f38affff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38b00f5410 0x7f38b0106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:27:09.551 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:09.546+0000 7f38affff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38b00f5410 0x7f38b0106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:41432/0 (socket says 172.21.15.81:41432) 2023-11-15T06:27:09.551 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:09.546+0000 7f38affff700 1 -- 172.21.15.81:0/2436646759 learned_addr learned my addr 172.21.15.81:0/2436646759 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:27:09.551 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:09.546+0000 7f38affff700 1 -- 172.21.15.81:0/2436646759 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a0005040 con 0x7f38b00f5410 2023-11-15T06:27:09.551 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:09.546+0000 7f38affff700 1 --2- 172.21.15.81:0/2436646759 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38b00f5410 0x7f38b0106720 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f38a0013510 tx=0x7f38a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:27:09.553 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:09.546+0000 7f38acff9700 1 -- 172.21.15.81:0/2436646759 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38a0009320 con 0x7f38b00f5410 2023-11-15T06:27:09.553 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:09.546+0000 7f38acff9700 1 -- 172.21.15.81:0/2436646759 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f38a0013f00 con 0x7f38b00f5410 2023-11-15T06:27:09.553 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:09.546+0000 7f38b630f700 1 -- 172.21.15.81:0/2436646759 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38b0106e10 con 0x7f38b00f5410 2023-11-15T06:27:09.554 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:09.546+0000 7f38acff9700 1 -- 172.21.15.81:0/2436646759 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38a0014d90 con 0x7f38b00f5410 2023-11-15T06:27:09.554 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:09.546+0000 7f38b630f700 1 -- 172.21.15.81:0/2436646759 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38b01072a0 con 0x7f38b00f5410 2023-11-15T06:27:09.554 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:09.547+0000 7f38acff9700 1 -- 172.21.15.81:0/2436646759 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f38a00263a0 con 0x7f38b00f5410 2023-11-15T06:27:09.554 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:09.547+0000 7f38b630f700 1 -- 172.21.15.81:0/2436646759 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38b0045ad0 con 0x7f38b00f5410 2023-11-15T06:27:09.554 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:09.547+0000 7f38acff9700 1 --2- 172.21.15.81:0/2436646759 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f389802ff60 0x7f3898032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:27:09.555 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:09.547+0000 7f38acff9700 1 -- 172.21.15.81:0/2436646759 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f38a00412c0 con 0x7f38b00f5410 2023-11-15T06:27:09.555 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:09.547+0000 7f38af7fe700 1 --2- 172.21.15.81:0/2436646759 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f389802ff60 0x7f3898032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:27:09.555 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:09.548+0000 7f38af7fe700 1 --2- 172.21.15.81:0/2436646759 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f389802ff60 0x7f3898032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f38a40096a0 tx=0x7f38a4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:27:09.556 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:09.552+0000 7f38acff9700 1 -- 172.21.15.81:0/2436646759 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f38a00126f0 con 0x7f38b00f5410 2023-11-15T06:27:09.761 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:09.758+0000 7f38b630f700 1 -- 172.21.15.81:0/2436646759 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f38b0108150 con 0x7f38b00f5410 2023-11-15T06:27:09.761 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:09.758+0000 7f38acff9700 1 -- 172.21.15.81:0/2436646759 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f38a0012330 con 0x7f38b00f5410 2023-11-15T06:27:09.761 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:27:09.762 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:27:09.763 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:09.760+0000 7f38b630f700 1 -- 172.21.15.81:0/2436646759 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f389802ff60 msgr2=0x7f3898032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:09.763 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:09.760+0000 7f38b630f700 1 --2- 172.21.15.81:0/2436646759 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f389802ff60 0x7f3898032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f38a40096a0 tx=0x7f38a4006b40).stop 2023-11-15T06:27:09.763 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:09.760+0000 7f38b630f700 1 -- 172.21.15.81:0/2436646759 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38b00f5410 msgr2=0x7f38b0106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:09.763 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:09.760+0000 7f38b630f700 1 --2- 172.21.15.81:0/2436646759 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38b00f5410 0x7f38b0106720 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f38a0013510 tx=0x7f38a000b040).stop 2023-11-15T06:27:09.764 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:09.760+0000 7f38b630f700 1 -- 172.21.15.81:0/2436646759 shutdown_connections 2023-11-15T06:27:09.764 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:09.760+0000 7f38b630f700 1 --2- 172.21.15.81:0/2436646759 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f389802ff60 0x7f3898032420 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:09.764 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:09.760+0000 7f38b630f700 1 --2- 172.21.15.81:0/2436646759 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38b00f5410 0x7f38b0106720 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:09.764 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:09.760+0000 7f38b630f700 1 -- 172.21.15.81:0/2436646759 >> 172.21.15.81:0/2436646759 conn(0x7f38b00f0b70 msgr2=0x7f38b005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:27:09.764 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:09.760+0000 7f38b630f700 1 -- 172.21.15.81:0/2436646759 shutdown_connections 2023-11-15T06:27:09.765 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:09.760+0000 7f38b630f700 1 -- 172.21.15.81:0/2436646759 wait complete. 2023-11-15T06:27:09.765 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:27:10.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:09 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:27:09.759990+0000 mon.smithi062 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.81:0/2436646759' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:27:11.133 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:27:11.133 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:27:11.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:10 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:27:09.788824+0000 mgr.smithi062.rwurtk (mgr.14164) 39 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:27:12.352 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:12.349+0000 7fcad1818700 1 -- 172.21.15.81:0/3611496285 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcacc060180 msgr2=0x7fcacc0f8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:12.352 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:12.349+0000 7fcad1818700 1 --2- 172.21.15.81:0/3611496285 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcacc060180 0x7fcacc0f8f80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fcac0009fb0 tx=0x7fcac0009b30).stop 2023-11-15T06:27:12.353 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:12.349+0000 7fcad1818700 1 -- 172.21.15.81:0/3611496285 shutdown_connections 2023-11-15T06:27:12.353 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:12.349+0000 7fcad1818700 1 --2- 172.21.15.81:0/3611496285 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcacc060180 0x7fcacc0f8f80 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:12.353 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:12.349+0000 7fcad1818700 1 -- 172.21.15.81:0/3611496285 >> 172.21.15.81:0/3611496285 conn(0x7fcacc0f09a0 msgr2=0x7fcacc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:27:12.353 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:12.349+0000 7fcad1818700 1 -- 172.21.15.81:0/3611496285 shutdown_connections 2023-11-15T06:27:12.353 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:12.349+0000 7fcad1818700 1 -- 172.21.15.81:0/3611496285 wait complete. 2023-11-15T06:27:12.354 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:12.349+0000 7fcad1818700 1 Processor -- start 2023-11-15T06:27:12.354 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:12.349+0000 7fcad1818700 1 -- start start 2023-11-15T06:27:12.354 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:12.350+0000 7fcad1818700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcacc060180 0x7fcacc106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:27:12.354 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:12.350+0000 7fcad1818700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcacc1069f0 con 0x7fcacc060180 2023-11-15T06:27:12.356 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:12.350+0000 7fcad0816700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcacc060180 0x7fcacc106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:27:12.356 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:12.350+0000 7fcad0816700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcacc060180 0x7fcacc106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:44174/0 (socket says 172.21.15.81:44174) 2023-11-15T06:27:12.356 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:12.350+0000 7fcad0816700 1 -- 172.21.15.81:0/4222654395 learned_addr learned my addr 172.21.15.81:0/4222654395 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:27:12.356 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:12.350+0000 7fcad0816700 1 -- 172.21.15.81:0/4222654395 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac0005040 con 0x7fcacc060180 2023-11-15T06:27:12.357 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:12.350+0000 7fcad0816700 1 --2- 172.21.15.81:0/4222654395 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcacc060180 0x7fcacc106500 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fcac0013510 tx=0x7fcac000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:27:12.357 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:12.351+0000 7fcac97fa700 1 -- 172.21.15.81:0/4222654395 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcac0009320 con 0x7fcacc060180 2023-11-15T06:27:12.357 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:12.351+0000 7fcad1818700 1 -- 172.21.15.81:0/4222654395 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcacc106bf0 con 0x7fcacc060180 2023-11-15T06:27:12.357 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:12.351+0000 7fcac97fa700 1 -- 172.21.15.81:0/4222654395 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcac0013f00 con 0x7fcacc060180 2023-11-15T06:27:12.357 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:12.351+0000 7fcac97fa700 1 -- 172.21.15.81:0/4222654395 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcac0014d90 con 0x7fcacc060180 2023-11-15T06:27:12.358 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:12.351+0000 7fcad1818700 1 -- 172.21.15.81:0/4222654395 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcacc107080 con 0x7fcacc060180 2023-11-15T06:27:12.358 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:12.351+0000 7fcac97fa700 1 -- 172.21.15.81:0/4222654395 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcac00263a0 con 0x7fcacc060180 2023-11-15T06:27:12.358 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:12.351+0000 7fcad1818700 1 -- 172.21.15.81:0/4222654395 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcacc045ad0 con 0x7fcacc060180 2023-11-15T06:27:12.358 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:12.352+0000 7fcac97fa700 1 --2- 172.21.15.81:0/4222654395 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fcabc02ff60 0x7fcabc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:27:12.358 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:12.352+0000 7fcac97fa700 1 -- 172.21.15.81:0/4222654395 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcac00413c0 con 0x7fcacc060180 2023-11-15T06:27:12.359 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:12.352+0000 7fcacbfff700 1 --2- 172.21.15.81:0/4222654395 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fcabc02ff60 0x7fcabc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:27:12.359 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:12.352+0000 7fcacbfff700 1 --2- 172.21.15.81:0/4222654395 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fcabc02ff60 0x7fcabc032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fcab80096a0 tx=0x7fcab8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:27:12.359 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:12.356+0000 7fcac97fa700 1 -- 172.21.15.81:0/4222654395 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcac00126f0 con 0x7fcacc060180 2023-11-15T06:27:12.561 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:12.558+0000 7fcad1818700 1 -- 172.21.15.81:0/4222654395 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcacc107f30 con 0x7fcacc060180 2023-11-15T06:27:12.562 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:12.559+0000 7fcac97fa700 1 -- 172.21.15.81:0/4222654395 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcac0012330 con 0x7fcacc060180 2023-11-15T06:27:12.562 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:27:12.562 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:27:12.563 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:12.561+0000 7fcad1818700 1 -- 172.21.15.81:0/4222654395 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fcabc02ff60 msgr2=0x7fcabc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:12.563 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:12.561+0000 7fcad1818700 1 --2- 172.21.15.81:0/4222654395 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fcabc02ff60 0x7fcabc032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fcab80096a0 tx=0x7fcab8006b40).stop 2023-11-15T06:27:12.564 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:12.561+0000 7fcad1818700 1 -- 172.21.15.81:0/4222654395 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcacc060180 msgr2=0x7fcacc106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:12.564 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:12.561+0000 7fcad1818700 1 --2- 172.21.15.81:0/4222654395 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcacc060180 0x7fcacc106500 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fcac0013510 tx=0x7fcac000b040).stop 2023-11-15T06:27:12.564 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:12.561+0000 7fcad1818700 1 -- 172.21.15.81:0/4222654395 shutdown_connections 2023-11-15T06:27:12.564 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:12.561+0000 7fcad1818700 1 --2- 172.21.15.81:0/4222654395 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fcabc02ff60 0x7fcabc032420 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:12.564 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:12.561+0000 7fcad1818700 1 --2- 172.21.15.81:0/4222654395 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcacc060180 0x7fcacc106500 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:12.565 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:12.561+0000 7fcad1818700 1 -- 172.21.15.81:0/4222654395 >> 172.21.15.81:0/4222654395 conn(0x7fcacc0f09a0 msgr2=0x7fcacc0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:27:12.565 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:12.561+0000 7fcad1818700 1 -- 172.21.15.81:0/4222654395 shutdown_connections 2023-11-15T06:27:12.565 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:12.561+0000 7fcad1818700 1 -- 172.21.15.81:0/4222654395 wait complete. 2023-11-15T06:27:12.565 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:27:13.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:12 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:27:11.789110+0000 mgr.smithi062.rwurtk (mgr.14164) 40 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:27:13.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:12 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:27:12.560846+0000 mon.smithi062 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.81:0/4222654395' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:27:13.972 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:27:13.973 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:27:15.139 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:15.135+0000 7f35f51b7700 1 -- 172.21.15.81:0/2672730158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35f00f5330 msgr2=0x7f35f00f5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:15.140 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:15.135+0000 7f35f51b7700 1 --2- 172.21.15.81:0/2672730158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35f00f5330 0x7f35f00f5750 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f35e0009fb0 tx=0x7f35e0009b30).stop 2023-11-15T06:27:15.140 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:15.136+0000 7f35f51b7700 1 -- 172.21.15.81:0/2672730158 shutdown_connections 2023-11-15T06:27:15.140 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:15.136+0000 7f35f51b7700 1 --2- 172.21.15.81:0/2672730158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35f00f5330 0x7f35f00f5750 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:15.140 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:15.136+0000 7f35f51b7700 1 -- 172.21.15.81:0/2672730158 >> 172.21.15.81:0/2672730158 conn(0x7f35f00f0a90 msgr2=0x7f35f00f2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:27:15.141 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:15.136+0000 7f35f51b7700 1 -- 172.21.15.81:0/2672730158 shutdown_connections 2023-11-15T06:27:15.141 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:15.136+0000 7f35f51b7700 1 -- 172.21.15.81:0/2672730158 wait complete. 2023-11-15T06:27:15.141 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:15.136+0000 7f35f51b7700 1 Processor -- start 2023-11-15T06:27:15.141 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:15.136+0000 7f35f51b7700 1 -- start start 2023-11-15T06:27:15.141 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:15.136+0000 7f35f51b7700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35f00f5330 0x7f35f00fd330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:27:15.142 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:15.136+0000 7f35f51b7700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f00fb7b0 con 0x7f35f00f5330 2023-11-15T06:27:15.142 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:15.137+0000 7f35eed9d700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35f00f5330 0x7f35f00fd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:27:15.142 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:15.137+0000 7f35eed9d700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35f00f5330 0x7f35f00fd330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:44188/0 (socket says 172.21.15.81:44188) 2023-11-15T06:27:15.142 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:15.137+0000 7f35eed9d700 1 -- 172.21.15.81:0/3210019059 learned_addr learned my addr 172.21.15.81:0/3210019059 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:27:15.142 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:15.137+0000 7f35eed9d700 1 -- 172.21.15.81:0/3210019059 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e0005040 con 0x7f35f00f5330 2023-11-15T06:27:15.143 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:15.137+0000 7f35eed9d700 1 --2- 172.21.15.81:0/3210019059 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35f00f5330 0x7f35f00fd330 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f35e0013510 tx=0x7f35e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:27:15.143 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:15.137+0000 7f35d77fe700 1 -- 172.21.15.81:0/3210019059 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35e0009320 con 0x7f35f00f5330 2023-11-15T06:27:15.144 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:15.137+0000 7f35f51b7700 1 -- 172.21.15.81:0/3210019059 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35f00fb950 con 0x7f35f00f5330 2023-11-15T06:27:15.144 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:15.137+0000 7f35d77fe700 1 -- 172.21.15.81:0/3210019059 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f35e0013f00 con 0x7f35f00f5330 2023-11-15T06:27:15.144 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:15.137+0000 7f35d77fe700 1 -- 172.21.15.81:0/3210019059 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35e0014d90 con 0x7f35f00f5330 2023-11-15T06:27:15.145 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:15.137+0000 7f35f51b7700 1 -- 172.21.15.81:0/3210019059 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35f00fbde0 con 0x7f35f00f5330 2023-11-15T06:27:15.145 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:15.138+0000 7f35f51b7700 1 -- 172.21.15.81:0/3210019059 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35f0045ad0 con 0x7f35f00f5330 2023-11-15T06:27:15.145 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:15.138+0000 7f35d77fe700 1 -- 172.21.15.81:0/3210019059 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f35e00095f0 con 0x7f35f00f5330 2023-11-15T06:27:15.145 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:15.138+0000 7f35d77fe700 1 --2- 172.21.15.81:0/3210019059 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f35d802ff60 0x7f35d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:27:15.145 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:15.138+0000 7f35d77fe700 1 -- 172.21.15.81:0/3210019059 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f35e0040fd0 con 0x7f35f00f5330 2023-11-15T06:27:15.146 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:15.139+0000 7f35ee59c700 1 --2- 172.21.15.81:0/3210019059 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f35d802ff60 0x7f35d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:27:15.146 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:15.139+0000 7f35ee59c700 1 --2- 172.21.15.81:0/3210019059 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f35d802ff60 0x7f35d8032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f35e40096a0 tx=0x7f35e4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:27:15.147 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:15.143+0000 7f35d77fe700 1 -- 172.21.15.81:0/3210019059 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f35e0014420 con 0x7f35f00f5330 2023-11-15T06:27:15.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:14 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:27:13.789461+0000 mgr.smithi062.rwurtk (mgr.14164) 41 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:27:15.351 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:15.348+0000 7f35f51b7700 1 -- 172.21.15.81:0/3210019059 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f35f005ea50 con 0x7f35f00f5330 2023-11-15T06:27:15.352 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:15.349+0000 7f35d77fe700 1 -- 172.21.15.81:0/3210019059 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f35e001d7f0 con 0x7f35f00f5330 2023-11-15T06:27:15.352 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:27:15.352 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:27:15.354 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:15.351+0000 7f35f51b7700 1 -- 172.21.15.81:0/3210019059 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f35d802ff60 msgr2=0x7f35d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:15.354 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:15.351+0000 7f35f51b7700 1 --2- 172.21.15.81:0/3210019059 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f35d802ff60 0x7f35d8032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f35e40096a0 tx=0x7f35e4006b40).stop 2023-11-15T06:27:15.354 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:15.351+0000 7f35f51b7700 1 -- 172.21.15.81:0/3210019059 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35f00f5330 msgr2=0x7f35f00fd330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:15.354 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:15.351+0000 7f35f51b7700 1 --2- 172.21.15.81:0/3210019059 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35f00f5330 0x7f35f00fd330 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f35e0013510 tx=0x7f35e000b040).stop 2023-11-15T06:27:15.355 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:15.351+0000 7f35f51b7700 1 -- 172.21.15.81:0/3210019059 shutdown_connections 2023-11-15T06:27:15.355 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:15.351+0000 7f35f51b7700 1 --2- 172.21.15.81:0/3210019059 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f35d802ff60 0x7f35d8032420 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:15.355 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:15.351+0000 7f35f51b7700 1 --2- 172.21.15.81:0/3210019059 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35f00f5330 0x7f35f00fd330 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:15.355 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:15.351+0000 7f35f51b7700 1 -- 172.21.15.81:0/3210019059 >> 172.21.15.81:0/3210019059 conn(0x7f35f00f0a90 msgr2=0x7f35f00f18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:27:15.355 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:15.351+0000 7f35f51b7700 1 -- 172.21.15.81:0/3210019059 shutdown_connections 2023-11-15T06:27:15.356 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:15.351+0000 7f35f51b7700 1 -- 172.21.15.81:0/3210019059 wait complete. 2023-11-15T06:27:15.356 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:27:16.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:15 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:27:15.351041+0000 mon.smithi062 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.81:0/3210019059' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:27:16.768 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:27:16.768 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:27:17.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:16 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:27:15.789857+0000 mgr.smithi062.rwurtk (mgr.14164) 42 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:27:18.020 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:18.016+0000 7f3e13fff700 1 -- 172.21.15.81:0/230246237 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e140f51a0 msgr2=0x7f3e140f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:18.020 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:18.016+0000 7f3e13fff700 1 --2- 172.21.15.81:0/230246237 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e140f51a0 0x7f3e140f55c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f3e04009fb0 tx=0x7f3e04009b30).stop 2023-11-15T06:27:18.020 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:18.017+0000 7f3e13fff700 1 -- 172.21.15.81:0/230246237 shutdown_connections 2023-11-15T06:27:18.020 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:18.017+0000 7f3e13fff700 1 --2- 172.21.15.81:0/230246237 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e140f51a0 0x7f3e140f55c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:18.021 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:18.017+0000 7f3e13fff700 1 -- 172.21.15.81:0/230246237 >> 172.21.15.81:0/230246237 conn(0x7f3e140f09a0 msgr2=0x7f3e140f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:27:18.021 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:18.017+0000 7f3e13fff700 1 -- 172.21.15.81:0/230246237 shutdown_connections 2023-11-15T06:27:18.021 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:18.017+0000 7f3e13fff700 1 -- 172.21.15.81:0/230246237 wait complete. 2023-11-15T06:27:18.021 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:18.017+0000 7f3e13fff700 1 Processor -- start 2023-11-15T06:27:18.022 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:18.017+0000 7f3e13fff700 1 -- start start 2023-11-15T06:27:18.022 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:18.017+0000 7f3e13fff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e140f51a0 0x7f3e141091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:27:18.022 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:18.017+0000 7f3e13fff700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e14060220 con 0x7f3e140f51a0 2023-11-15T06:27:18.024 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:18.018+0000 7f3e12ffd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e140f51a0 0x7f3e141091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:27:18.024 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:18.018+0000 7f3e12ffd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e140f51a0 0x7f3e141091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:44206/0 (socket says 172.21.15.81:44206) 2023-11-15T06:27:18.024 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:18.018+0000 7f3e12ffd700 1 -- 172.21.15.81:0/1171459009 learned_addr learned my addr 172.21.15.81:0/1171459009 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:27:18.024 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:18.018+0000 7f3e12ffd700 1 -- 172.21.15.81:0/1171459009 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e04005040 con 0x7f3e140f51a0 2023-11-15T06:27:18.024 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:18.018+0000 7f3e12ffd700 1 --2- 172.21.15.81:0/1171459009 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e140f51a0 0x7f3e141091d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f3e04013510 tx=0x7f3e0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:27:18.025 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:18.018+0000 7f3dfb7fe700 1 -- 172.21.15.81:0/1171459009 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e04009320 con 0x7f3e140f51a0 2023-11-15T06:27:18.025 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:18.018+0000 7f3e13fff700 1 -- 172.21.15.81:0/1171459009 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e14060420 con 0x7f3e140f51a0 2023-11-15T06:27:18.025 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:18.018+0000 7f3dfb7fe700 1 -- 172.21.15.81:0/1171459009 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3e04013f00 con 0x7f3e140f51a0 2023-11-15T06:27:18.025 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:18.018+0000 7f3dfb7fe700 1 -- 172.21.15.81:0/1171459009 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e04014dc0 con 0x7f3e140f51a0 2023-11-15T06:27:18.025 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:18.018+0000 7f3e13fff700 1 -- 172.21.15.81:0/1171459009 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e14109880 con 0x7f3e140f51a0 2023-11-15T06:27:18.026 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:18.019+0000 7f3dfb7fe700 1 -- 172.21.15.81:0/1171459009 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3e04014420 con 0x7f3e140f51a0 2023-11-15T06:27:18.026 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:18.019+0000 7f3e13fff700 1 -- 172.21.15.81:0/1171459009 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e14059100 con 0x7f3e140f51a0 2023-11-15T06:27:18.026 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:18.019+0000 7f3dfb7fe700 1 --2- 172.21.15.81:0/1171459009 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f3dfc02ff60 0x7f3dfc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:27:18.026 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:18.019+0000 7f3dfb7fe700 1 -- 172.21.15.81:0/1171459009 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3e04041110 con 0x7f3e140f51a0 2023-11-15T06:27:18.027 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:18.019+0000 7f3e127fc700 1 --2- 172.21.15.81:0/1171459009 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f3dfc02ff60 0x7f3dfc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:27:18.027 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:18.020+0000 7f3e127fc700 1 --2- 172.21.15.81:0/1171459009 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f3dfc02ff60 0x7f3dfc032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f3e080096a0 tx=0x7f3e08006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:27:18.027 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:18.024+0000 7f3dfb7fe700 1 -- 172.21.15.81:0/1171459009 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3e040095f0 con 0x7f3e140f51a0 2023-11-15T06:27:18.229 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:18.226+0000 7f3e13fff700 1 -- 172.21.15.81:0/1171459009 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3e14045ad0 con 0x7f3e140f51a0 2023-11-15T06:27:18.229 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:18.227+0000 7f3dfb7fe700 1 -- 172.21.15.81:0/1171459009 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3e0401cb10 con 0x7f3e140f51a0 2023-11-15T06:27:18.230 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:27:18.230 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:27:18.231 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:18.228+0000 7f3e13fff700 1 -- 172.21.15.81:0/1171459009 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f3dfc02ff60 msgr2=0x7f3dfc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:18.231 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:18.228+0000 7f3e13fff700 1 --2- 172.21.15.81:0/1171459009 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f3dfc02ff60 0x7f3dfc032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f3e080096a0 tx=0x7f3e08006b40).stop 2023-11-15T06:27:18.231 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:18.228+0000 7f3e13fff700 1 -- 172.21.15.81:0/1171459009 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e140f51a0 msgr2=0x7f3e141091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:18.232 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:18.228+0000 7f3e13fff700 1 --2- 172.21.15.81:0/1171459009 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e140f51a0 0x7f3e141091d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f3e04013510 tx=0x7f3e0400b040).stop 2023-11-15T06:27:18.232 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:18.228+0000 7f3e13fff700 1 -- 172.21.15.81:0/1171459009 shutdown_connections 2023-11-15T06:27:18.232 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:18.228+0000 7f3e13fff700 1 --2- 172.21.15.81:0/1171459009 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f3dfc02ff60 0x7f3dfc032420 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:18.232 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:18.228+0000 7f3e13fff700 1 --2- 172.21.15.81:0/1171459009 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e140f51a0 0x7f3e141091d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:18.232 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:18.228+0000 7f3e13fff700 1 -- 172.21.15.81:0/1171459009 >> 172.21.15.81:0/1171459009 conn(0x7f3e140f09a0 msgr2=0x7f3e140f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:27:18.233 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:18.228+0000 7f3e13fff700 1 -- 172.21.15.81:0/1171459009 shutdown_connections 2023-11-15T06:27:18.233 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:18.228+0000 7f3e13fff700 1 -- 172.21.15.81:0/1171459009 wait complete. 2023-11-15T06:27:18.233 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:27:18.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:27:17.790221+0000 mgr.smithi062.rwurtk (mgr.14164) 43 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:27:18.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:27:18.228438+0000 mon.smithi062 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.81:0/1171459009' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:27:19.697 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:27:19.697 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:27:20.864 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:20.861+0000 7ff5d695f700 1 -- 172.21.15.81:0/3713805666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5d0060200 msgr2=0x7ff5d00f9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:20.864 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:20.861+0000 7ff5d695f700 1 --2- 172.21.15.81:0/3713805666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5d0060200 0x7ff5d00f9170 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7ff5c4009fb0 tx=0x7ff5c4009b30).stop 2023-11-15T06:27:20.864 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:20.861+0000 7ff5d695f700 1 -- 172.21.15.81:0/3713805666 shutdown_connections 2023-11-15T06:27:20.865 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:20.861+0000 7ff5d695f700 1 --2- 172.21.15.81:0/3713805666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5d0060200 0x7ff5d00f9170 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:20.865 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:20.861+0000 7ff5d695f700 1 -- 172.21.15.81:0/3713805666 >> 172.21.15.81:0/3713805666 conn(0x7ff5d00f0b70 msgr2=0x7ff5d00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:27:20.865 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:20.861+0000 7ff5d695f700 1 -- 172.21.15.81:0/3713805666 shutdown_connections 2023-11-15T06:27:20.865 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:20.861+0000 7ff5d695f700 1 -- 172.21.15.81:0/3713805666 wait complete. 2023-11-15T06:27:20.867 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:20.862+0000 7ff5d695f700 1 Processor -- start 2023-11-15T06:27:20.867 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:20.862+0000 7ff5d695f700 1 -- start start 2023-11-15T06:27:20.868 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:20.862+0000 7ff5d695f700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5d0060200 0x7ff5d01066a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:27:20.868 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:20.862+0000 7ff5d695f700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d0106b90 con 0x7ff5d0060200 2023-11-15T06:27:20.868 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:20.863+0000 7ff5cffff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5d0060200 0x7ff5d01066a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:27:20.868 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:20.863+0000 7ff5cffff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5d0060200 0x7ff5d01066a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:44216/0 (socket says 172.21.15.81:44216) 2023-11-15T06:27:20.868 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:20.863+0000 7ff5cffff700 1 -- 172.21.15.81:0/1204424774 learned_addr learned my addr 172.21.15.81:0/1204424774 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:27:20.869 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:20.863+0000 7ff5cffff700 1 -- 172.21.15.81:0/1204424774 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5c4005040 con 0x7ff5d0060200 2023-11-15T06:27:20.869 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:20.863+0000 7ff5cffff700 1 --2- 172.21.15.81:0/1204424774 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5d0060200 0x7ff5d01066a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7ff5c4013510 tx=0x7ff5c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:27:20.869 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:20.863+0000 7ff5ccff9700 1 -- 172.21.15.81:0/1204424774 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff5c4009320 con 0x7ff5d0060200 2023-11-15T06:27:20.869 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:20.863+0000 7ff5d695f700 1 -- 172.21.15.81:0/1204424774 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5d0106d90 con 0x7ff5d0060200 2023-11-15T06:27:20.869 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:20.863+0000 7ff5ccff9700 1 -- 172.21.15.81:0/1204424774 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff5c4013f00 con 0x7ff5d0060200 2023-11-15T06:27:20.870 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:20.863+0000 7ff5ccff9700 1 -- 172.21.15.81:0/1204424774 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff5c4014dc0 con 0x7ff5d0060200 2023-11-15T06:27:20.870 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:20.863+0000 7ff5d695f700 1 -- 172.21.15.81:0/1204424774 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5d0107220 con 0x7ff5d0060200 2023-11-15T06:27:20.870 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:20.864+0000 7ff5ccff9700 1 -- 172.21.15.81:0/1204424774 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff5c40095f0 con 0x7ff5d0060200 2023-11-15T06:27:20.870 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:20.864+0000 7ff5d695f700 1 -- 172.21.15.81:0/1204424774 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5d0045ad0 con 0x7ff5d0060200 2023-11-15T06:27:20.870 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:20.864+0000 7ff5ccff9700 1 --2- 172.21.15.81:0/1204424774 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff5b802ff60 0x7ff5b8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:27:20.871 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:20.864+0000 7ff5ccff9700 1 -- 172.21.15.81:0/1204424774 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff5c4041040 con 0x7ff5d0060200 2023-11-15T06:27:20.871 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:20.864+0000 7ff5cf7fe700 1 --2- 172.21.15.81:0/1204424774 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff5b802ff60 0x7ff5b8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:27:20.871 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:20.865+0000 7ff5cf7fe700 1 --2- 172.21.15.81:0/1204424774 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff5b802ff60 0x7ff5b8032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7ff5c00096a0 tx=0x7ff5c0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:27:20.871 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:20.869+0000 7ff5ccff9700 1 -- 172.21.15.81:0/1204424774 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff5c4014420 con 0x7ff5d0060200 2023-11-15T06:27:21.078 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:21.075+0000 7ff5d695f700 1 -- 172.21.15.81:0/1204424774 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff5d01080d0 con 0x7ff5d0060200 2023-11-15T06:27:21.079 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:21.077+0000 7ff5ccff9700 1 -- 172.21.15.81:0/1204424774 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff5c4010230 con 0x7ff5d0060200 2023-11-15T06:27:21.079 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:27:21.079 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:27:21.080 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:21.077+0000 7ff5d695f700 1 -- 172.21.15.81:0/1204424774 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff5b802ff60 msgr2=0x7ff5b8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:21.080 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:21.077+0000 7ff5d695f700 1 --2- 172.21.15.81:0/1204424774 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff5b802ff60 0x7ff5b8032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7ff5c00096a0 tx=0x7ff5c0006b40).stop 2023-11-15T06:27:21.081 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:21.078+0000 7ff5d695f700 1 -- 172.21.15.81:0/1204424774 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5d0060200 msgr2=0x7ff5d01066a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:21.081 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:21.078+0000 7ff5d695f700 1 --2- 172.21.15.81:0/1204424774 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5d0060200 0x7ff5d01066a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7ff5c4013510 tx=0x7ff5c400b040).stop 2023-11-15T06:27:21.081 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:21.078+0000 7ff5d695f700 1 -- 172.21.15.81:0/1204424774 shutdown_connections 2023-11-15T06:27:21.081 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:21.078+0000 7ff5d695f700 1 --2- 172.21.15.81:0/1204424774 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff5b802ff60 0x7ff5b8032420 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:21.081 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:21.078+0000 7ff5d695f700 1 --2- 172.21.15.81:0/1204424774 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5d0060200 0x7ff5d01066a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:21.082 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:21.078+0000 7ff5d695f700 1 -- 172.21.15.81:0/1204424774 >> 172.21.15.81:0/1204424774 conn(0x7ff5d00f0b70 msgr2=0x7ff5d00f1620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:27:21.082 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:21.078+0000 7ff5d695f700 1 -- 172.21.15.81:0/1204424774 shutdown_connections 2023-11-15T06:27:21.082 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:21.078+0000 7ff5d695f700 1 -- 172.21.15.81:0/1204424774 wait complete. 2023-11-15T06:27:21.082 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:27:21.169 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:20 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:27:19.790578+0000 mgr.smithi062.rwurtk (mgr.14164) 44 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:27:22.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:21 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:27:21.077677+0000 mon.smithi062 (mon.0) 139 : audit [DBG] from='client.? 172.21.15.81:0/1204424774' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:27:22.515 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:27:22.516 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:27:23.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:22 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:27:21.790873+0000 mgr.smithi062.rwurtk (mgr.14164) 45 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:27:23.766 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:23.763+0000 7f9e1a980700 1 -- 172.21.15.81:0/738966610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e140f6510 msgr2=0x7f9e140f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:23.766 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:23.763+0000 7f9e1a980700 1 --2- 172.21.15.81:0/738966610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e140f6510 0x7f9e140f6930 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f9e04009fb0 tx=0x7f9e04009b30).stop 2023-11-15T06:27:23.766 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:23.763+0000 7f9e1a980700 1 -- 172.21.15.81:0/738966610 shutdown_connections 2023-11-15T06:27:23.766 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:23.763+0000 7f9e1a980700 1 --2- 172.21.15.81:0/738966610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e140f6510 0x7f9e140f6930 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:23.767 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:23.763+0000 7f9e1a980700 1 -- 172.21.15.81:0/738966610 >> 172.21.15.81:0/738966610 conn(0x7f9e140f1c70 msgr2=0x7f9e140f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:27:23.767 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:23.763+0000 7f9e1a980700 1 -- 172.21.15.81:0/738966610 shutdown_connections 2023-11-15T06:27:23.767 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:23.763+0000 7f9e1a980700 1 -- 172.21.15.81:0/738966610 wait complete. 2023-11-15T06:27:23.767 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:23.764+0000 7f9e1a980700 1 Processor -- start 2023-11-15T06:27:23.767 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:23.764+0000 7f9e1a980700 1 -- start start 2023-11-15T06:27:23.768 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:23.764+0000 7f9e1a980700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e140f6510 0x7f9e14106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:27:23.768 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:23.764+0000 7f9e1a980700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e14106c20 con 0x7f9e140f6510 2023-11-15T06:27:23.768 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:23.764+0000 7f9e13fff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e140f6510 0x7f9e14106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:27:23.768 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:23.764+0000 7f9e13fff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e140f6510 0x7f9e14106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:49566/0 (socket says 172.21.15.81:49566) 2023-11-15T06:27:23.768 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:23.764+0000 7f9e13fff700 1 -- 172.21.15.81:0/4060450357 learned_addr learned my addr 172.21.15.81:0/4060450357 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:27:23.770 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:23.765+0000 7f9e13fff700 1 -- 172.21.15.81:0/4060450357 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e04005040 con 0x7f9e140f6510 2023-11-15T06:27:23.770 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:23.765+0000 7f9e13fff700 1 --2- 172.21.15.81:0/4060450357 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e140f6510 0x7f9e14106730 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f9e04013510 tx=0x7f9e0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:27:23.770 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:23.765+0000 7f9e10ff9700 1 -- 172.21.15.81:0/4060450357 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e04009320 con 0x7f9e140f6510 2023-11-15T06:27:23.770 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:23.765+0000 7f9e1a980700 1 -- 172.21.15.81:0/4060450357 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e14106e20 con 0x7f9e140f6510 2023-11-15T06:27:23.770 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:23.765+0000 7f9e10ff9700 1 -- 172.21.15.81:0/4060450357 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e04013f00 con 0x7f9e140f6510 2023-11-15T06:27:23.771 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:23.765+0000 7f9e10ff9700 1 -- 172.21.15.81:0/4060450357 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e04014d90 con 0x7f9e140f6510 2023-11-15T06:27:23.771 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:23.765+0000 7f9e1a980700 1 -- 172.21.15.81:0/4060450357 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e141072b0 con 0x7f9e140f6510 2023-11-15T06:27:23.771 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:23.766+0000 7f9e1a980700 1 -- 172.21.15.81:0/4060450357 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e14045ad0 con 0x7f9e140f6510 2023-11-15T06:27:23.771 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:23.766+0000 7f9e10ff9700 1 -- 172.21.15.81:0/4060450357 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9e04014420 con 0x7f9e140f6510 2023-11-15T06:27:23.771 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:23.766+0000 7f9e10ff9700 1 --2- 172.21.15.81:0/4060450357 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f9dfc02ff60 0x7f9dfc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:27:23.772 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:23.766+0000 7f9e10ff9700 1 -- 172.21.15.81:0/4060450357 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9e04041050 con 0x7f9e140f6510 2023-11-15T06:27:23.772 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:23.766+0000 7f9e137fe700 1 --2- 172.21.15.81:0/4060450357 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f9dfc02ff60 0x7f9dfc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:27:23.772 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:23.767+0000 7f9e137fe700 1 --2- 172.21.15.81:0/4060450357 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f9dfc02ff60 0x7f9dfc032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f9e080096a0 tx=0x7f9e08006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:27:23.773 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:23.771+0000 7f9e10ff9700 1 -- 172.21.15.81:0/4060450357 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9e040095f0 con 0x7f9e140f6510 2023-11-15T06:27:23.974 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:23.973+0000 7f9e1a980700 1 -- 172.21.15.81:0/4060450357 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9e140fbf60 con 0x7f9e140f6510 2023-11-15T06:27:23.976 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:23.974+0000 7f9e10ff9700 1 -- 172.21.15.81:0/4060450357 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9e0401db70 con 0x7f9e140f6510 2023-11-15T06:27:23.976 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:27:23.976 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:27:23.977 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:23.975+0000 7f9e1a980700 1 -- 172.21.15.81:0/4060450357 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f9dfc02ff60 msgr2=0x7f9dfc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:23.978 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:23.975+0000 7f9e1a980700 1 --2- 172.21.15.81:0/4060450357 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f9dfc02ff60 0x7f9dfc032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f9e080096a0 tx=0x7f9e08006b40).stop 2023-11-15T06:27:23.978 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:23.975+0000 7f9e1a980700 1 -- 172.21.15.81:0/4060450357 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e140f6510 msgr2=0x7f9e14106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:23.978 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:23.975+0000 7f9e1a980700 1 --2- 172.21.15.81:0/4060450357 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e140f6510 0x7f9e14106730 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f9e04013510 tx=0x7f9e0400b040).stop 2023-11-15T06:27:23.978 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:23.975+0000 7f9e1a980700 1 -- 172.21.15.81:0/4060450357 shutdown_connections 2023-11-15T06:27:23.978 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:23.975+0000 7f9e1a980700 1 --2- 172.21.15.81:0/4060450357 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f9dfc02ff60 0x7f9dfc032420 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:23.979 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:23.975+0000 7f9e1a980700 1 --2- 172.21.15.81:0/4060450357 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e140f6510 0x7f9e14106730 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:23.979 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:23.975+0000 7f9e1a980700 1 -- 172.21.15.81:0/4060450357 >> 172.21.15.81:0/4060450357 conn(0x7f9e140f1c70 msgr2=0x7f9e140f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:27:23.979 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:23.975+0000 7f9e1a980700 1 -- 172.21.15.81:0/4060450357 shutdown_connections 2023-11-15T06:27:23.980 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:23.975+0000 7f9e1a980700 1 -- 172.21.15.81:0/4060450357 wait complete. 2023-11-15T06:27:23.980 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:27:25.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:24 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:27:23.791163+0000 mgr.smithi062.rwurtk (mgr.14164) 46 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:27:25.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:24 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:27:23.974368+0000 mon.smithi062 (mon.0) 140 : audit [DBG] from='client.? 172.21.15.81:0/4060450357' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:27:25.366 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:27:25.366 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:27:26.818 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:26.814+0000 7f57d740f700 1 -- 172.21.15.81:0/3627673812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57d00f6510 msgr2=0x7f57d00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:26.818 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:26.814+0000 7f57d740f700 1 --2- 172.21.15.81:0/3627673812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57d00f6510 0x7f57d00f6930 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f57c0009fb0 tx=0x7f57c0009b30).stop 2023-11-15T06:27:26.818 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:26.814+0000 7f57d740f700 1 -- 172.21.15.81:0/3627673812 shutdown_connections 2023-11-15T06:27:26.819 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:26.814+0000 7f57d740f700 1 --2- 172.21.15.81:0/3627673812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57d00f6510 0x7f57d00f6930 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:26.819 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:26.814+0000 7f57d740f700 1 -- 172.21.15.81:0/3627673812 >> 172.21.15.81:0/3627673812 conn(0x7f57d00f1c70 msgr2=0x7f57d00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:27:26.819 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:26.815+0000 7f57d740f700 1 -- 172.21.15.81:0/3627673812 shutdown_connections 2023-11-15T06:27:26.819 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:26.815+0000 7f57d740f700 1 -- 172.21.15.81:0/3627673812 wait complete. 2023-11-15T06:27:26.819 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:26.815+0000 7f57d740f700 1 Processor -- start 2023-11-15T06:27:26.820 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:26.815+0000 7f57d740f700 1 -- start start 2023-11-15T06:27:26.820 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:26.815+0000 7f57d740f700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57d00f6510 0x7f57d0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:27:26.820 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:26.815+0000 7f57d740f700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d0106c20 con 0x7f57d00f6510 2023-11-15T06:27:26.820 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:26.816+0000 7f57d51ab700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57d00f6510 0x7f57d0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:27:26.820 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:26.816+0000 7f57d51ab700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57d00f6510 0x7f57d0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:49578/0 (socket says 172.21.15.81:49578) 2023-11-15T06:27:26.821 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:26.816+0000 7f57d51ab700 1 -- 172.21.15.81:0/2053573944 learned_addr learned my addr 172.21.15.81:0/2053573944 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:27:26.821 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:26.816+0000 7f57d51ab700 1 -- 172.21.15.81:0/2053573944 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c0005040 con 0x7f57d00f6510 2023-11-15T06:27:26.821 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:26.816+0000 7f57d51ab700 1 --2- 172.21.15.81:0/2053573944 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57d00f6510 0x7f57d0106730 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f57c0013510 tx=0x7f57c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:27:26.821 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:26.816+0000 7f57c5ffb700 1 -- 172.21.15.81:0/2053573944 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57c0009320 con 0x7f57d00f6510 2023-11-15T06:27:26.821 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:26.816+0000 7f57d740f700 1 -- 172.21.15.81:0/2053573944 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57d0106e20 con 0x7f57d00f6510 2023-11-15T06:27:26.822 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:26.816+0000 7f57c5ffb700 1 -- 172.21.15.81:0/2053573944 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f57c0013f00 con 0x7f57d00f6510 2023-11-15T06:27:26.822 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:26.816+0000 7f57c5ffb700 1 -- 172.21.15.81:0/2053573944 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57c0014d90 con 0x7f57d00f6510 2023-11-15T06:27:26.822 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:26.816+0000 7f57d740f700 1 -- 172.21.15.81:0/2053573944 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57d01072b0 con 0x7f57d00f6510 2023-11-15T06:27:26.823 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:26.817+0000 7f57c5ffb700 1 -- 172.21.15.81:0/2053573944 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f57c0014420 con 0x7f57d00f6510 2023-11-15T06:27:26.823 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:26.817+0000 7f57d740f700 1 -- 172.21.15.81:0/2053573944 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57d0045ad0 con 0x7f57d00f6510 2023-11-15T06:27:26.824 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:26.817+0000 7f57c5ffb700 1 --2- 172.21.15.81:0/2053573944 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f57bc02ff60 0x7f57bc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:27:26.825 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:26.817+0000 7f57c5ffb700 1 -- 172.21.15.81:0/2053573944 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f57c0041110 con 0x7f57d00f6510 2023-11-15T06:27:26.826 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:26.818+0000 7f57d49aa700 1 --2- 172.21.15.81:0/2053573944 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f57bc02ff60 0x7f57bc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:27:26.826 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:26.818+0000 7f57d49aa700 1 --2- 172.21.15.81:0/2053573944 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f57bc02ff60 0x7f57bc032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f57cc0096a0 tx=0x7f57cc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:27:26.826 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:26.822+0000 7f57c5ffb700 1 -- 172.21.15.81:0/2053573944 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f57c00095f0 con 0x7f57d00f6510 2023-11-15T06:27:27.026 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:27.024+0000 7f57d740f700 1 -- 172.21.15.81:0/2053573944 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f57d00fbf60 con 0x7f57d00f6510 2023-11-15T06:27:27.027 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:27.025+0000 7f57c5ffb700 1 -- 172.21.15.81:0/2053573944 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f57c001db70 con 0x7f57d00f6510 2023-11-15T06:27:27.027 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:27:27.028 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:27:27.029 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:27.027+0000 7f57d740f700 1 -- 172.21.15.81:0/2053573944 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f57bc02ff60 msgr2=0x7f57bc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:27.029 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:27.027+0000 7f57d740f700 1 --2- 172.21.15.81:0/2053573944 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f57bc02ff60 0x7f57bc032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f57cc0096a0 tx=0x7f57cc006b40).stop 2023-11-15T06:27:27.029 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:27.027+0000 7f57d740f700 1 -- 172.21.15.81:0/2053573944 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57d00f6510 msgr2=0x7f57d0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:27.029 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:27.027+0000 7f57d740f700 1 --2- 172.21.15.81:0/2053573944 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57d00f6510 0x7f57d0106730 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f57c0013510 tx=0x7f57c000b040).stop 2023-11-15T06:27:27.029 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:27.027+0000 7f57d740f700 1 -- 172.21.15.81:0/2053573944 shutdown_connections 2023-11-15T06:27:27.030 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:27.027+0000 7f57d740f700 1 --2- 172.21.15.81:0/2053573944 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f57bc02ff60 0x7f57bc032420 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:27.030 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:27.027+0000 7f57d740f700 1 --2- 172.21.15.81:0/2053573944 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57d00f6510 0x7f57d0106730 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:27.030 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:27.027+0000 7f57d740f700 1 -- 172.21.15.81:0/2053573944 >> 172.21.15.81:0/2053573944 conn(0x7f57d00f1c70 msgr2=0x7f57d00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:27:27.030 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:27.027+0000 7f57d740f700 1 -- 172.21.15.81:0/2053573944 shutdown_connections 2023-11-15T06:27:27.030 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:27.027+0000 7f57d740f700 1 -- 172.21.15.81:0/2053573944 wait complete. 2023-11-15T06:27:27.031 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:27:27.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:26 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:27:25.791536+0000 mgr.smithi062.rwurtk (mgr.14164) 47 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:27:28.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:27 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:27:27.025513+0000 mon.smithi062 (mon.0) 141 : audit [DBG] from='client.? 172.21.15.81:0/2053573944' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:27:28.416 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:27:28.417 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:27:29.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:28 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:27:27.791864+0000 mgr.smithi062.rwurtk (mgr.14164) 48 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:27:29.632 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:29.629+0000 7f99bf59e700 1 -- 172.21.15.81:0/1342489420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99c00f51a0 msgr2=0x7f99c00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:29.632 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:29.629+0000 7f99bf59e700 1 --2- 172.21.15.81:0/1342489420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99c00f51a0 0x7f99c00f55c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f99b0009fb0 tx=0x7f99b0009b30).stop 2023-11-15T06:27:29.633 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:29.629+0000 7f99bf59e700 1 -- 172.21.15.81:0/1342489420 shutdown_connections 2023-11-15T06:27:29.633 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:29.629+0000 7f99bf59e700 1 --2- 172.21.15.81:0/1342489420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99c00f51a0 0x7f99c00f55c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:29.633 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:29.629+0000 7f99bf59e700 1 -- 172.21.15.81:0/1342489420 >> 172.21.15.81:0/1342489420 conn(0x7f99c00f09a0 msgr2=0x7f99c00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:27:29.633 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:29.629+0000 7f99bf59e700 1 -- 172.21.15.81:0/1342489420 shutdown_connections 2023-11-15T06:27:29.633 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:29.629+0000 7f99bf59e700 1 -- 172.21.15.81:0/1342489420 wait complete. 2023-11-15T06:27:29.634 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:29.629+0000 7f99bf59e700 1 Processor -- start 2023-11-15T06:27:29.634 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:29.629+0000 7f99bf59e700 1 -- start start 2023-11-15T06:27:29.634 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:29.629+0000 7f99bf59e700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99c00f51a0 0x7f99c00fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:27:29.634 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:29.630+0000 7f99bf59e700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c00fb5a0 con 0x7f99c00f51a0 2023-11-15T06:27:29.634 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:29.630+0000 7f99be59c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99c00f51a0 0x7f99c00fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:27:29.636 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:29.630+0000 7f99be59c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99c00f51a0 0x7f99c00fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:49596/0 (socket says 172.21.15.81:49596) 2023-11-15T06:27:29.639 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:29.630+0000 7f99be59c700 1 -- 172.21.15.81:0/496740112 learned_addr learned my addr 172.21.15.81:0/496740112 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:27:29.640 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:29.630+0000 7f99be59c700 1 -- 172.21.15.81:0/496740112 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b0005040 con 0x7f99c00f51a0 2023-11-15T06:27:29.640 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:29.630+0000 7f99be59c700 1 --2- 172.21.15.81:0/496740112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99c00f51a0 0x7f99c00fd120 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f99b0013510 tx=0x7f99b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:27:29.640 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:29.630+0000 7f99aeffd700 1 -- 172.21.15.81:0/496740112 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99b0009320 con 0x7f99c00f51a0 2023-11-15T06:27:29.641 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:29.630+0000 7f99bf59e700 1 -- 172.21.15.81:0/496740112 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99c00fb740 con 0x7f99c00f51a0 2023-11-15T06:27:29.641 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:29.630+0000 7f99aeffd700 1 -- 172.21.15.81:0/496740112 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f99b0013f00 con 0x7f99c00f51a0 2023-11-15T06:27:29.641 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:29.630+0000 7f99aeffd700 1 -- 172.21.15.81:0/496740112 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99b0014dc0 con 0x7f99c00f51a0 2023-11-15T06:27:29.641 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:29.631+0000 7f99bf59e700 1 -- 172.21.15.81:0/496740112 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99c00fbbd0 con 0x7f99c00f51a0 2023-11-15T06:27:29.643 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:29.631+0000 7f99aeffd700 1 -- 172.21.15.81:0/496740112 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f99b0014420 con 0x7f99c00f51a0 2023-11-15T06:27:29.643 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:29.631+0000 7f99bf59e700 1 -- 172.21.15.81:0/496740112 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99c0059100 con 0x7f99c00f51a0 2023-11-15T06:27:29.644 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:29.631+0000 7f99aeffd700 1 --2- 172.21.15.81:0/496740112 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f99a802ff60 0x7f99a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:27:29.644 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:29.631+0000 7f99aeffd700 1 -- 172.21.15.81:0/496740112 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f99b0041030 con 0x7f99c00f51a0 2023-11-15T06:27:29.644 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:29.632+0000 7f99bdd9b700 1 --2- 172.21.15.81:0/496740112 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f99a802ff60 0x7f99a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:27:29.644 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:29.632+0000 7f99bdd9b700 1 --2- 172.21.15.81:0/496740112 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f99a802ff60 0x7f99a8032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f99b40096a0 tx=0x7f99b4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:27:29.644 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:29.636+0000 7f99aeffd700 1 -- 172.21.15.81:0/496740112 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f99b00095f0 con 0x7f99c00f51a0 2023-11-15T06:27:29.842 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:29.840+0000 7f99bf59e700 1 -- 172.21.15.81:0/496740112 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f99c00fabe0 con 0x7f99c00f51a0 2023-11-15T06:27:29.842 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:29.841+0000 7f99aeffd700 1 -- 172.21.15.81:0/496740112 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f99b001cb10 con 0x7f99c00f51a0 2023-11-15T06:27:29.843 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:27:29.843 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:27:29.844 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:29.842+0000 7f99bf59e700 1 -- 172.21.15.81:0/496740112 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f99a802ff60 msgr2=0x7f99a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:29.844 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:29.842+0000 7f99bf59e700 1 --2- 172.21.15.81:0/496740112 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f99a802ff60 0x7f99a8032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f99b40096a0 tx=0x7f99b4006b40).stop 2023-11-15T06:27:29.844 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:29.842+0000 7f99bf59e700 1 -- 172.21.15.81:0/496740112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99c00f51a0 msgr2=0x7f99c00fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:29.844 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:29.842+0000 7f99bf59e700 1 --2- 172.21.15.81:0/496740112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99c00f51a0 0x7f99c00fd120 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f99b0013510 tx=0x7f99b000b040).stop 2023-11-15T06:27:29.845 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:29.842+0000 7f99bf59e700 1 -- 172.21.15.81:0/496740112 shutdown_connections 2023-11-15T06:27:29.845 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:29.842+0000 7f99bf59e700 1 --2- 172.21.15.81:0/496740112 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f99a802ff60 0x7f99a8032420 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:29.845 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:29.842+0000 7f99bf59e700 1 --2- 172.21.15.81:0/496740112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99c00f51a0 0x7f99c00fd120 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:29.845 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:29.842+0000 7f99bf59e700 1 -- 172.21.15.81:0/496740112 >> 172.21.15.81:0/496740112 conn(0x7f99c00f09a0 msgr2=0x7f99c005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:27:29.845 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:29.842+0000 7f99bf59e700 1 -- 172.21.15.81:0/496740112 shutdown_connections 2023-11-15T06:27:29.846 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:29.842+0000 7f99bf59e700 1 -- 172.21.15.81:0/496740112 wait complete. 2023-11-15T06:27:29.846 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:27:30.169 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:29 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:27:29.841550+0000 mon.smithi062 (mon.0) 142 : audit [DBG] from='client.? 172.21.15.81:0/496740112' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:27:31.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:30 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:27:29.792222+0000 mgr.smithi062.rwurtk (mgr.14164) 49 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:27:31.262 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:27:31.263 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:27:32.579 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:32.576+0000 7f9987367700 1 -- 172.21.15.81:0/96378139 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99800f5410 msgr2=0x7f99800f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:32.580 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:32.576+0000 7f9987367700 1 --2- 172.21.15.81:0/96378139 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99800f5410 0x7f99800f5830 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f9970009fb0 tx=0x7f9970009b30).stop 2023-11-15T06:27:32.580 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:32.576+0000 7f9987367700 1 -- 172.21.15.81:0/96378139 shutdown_connections 2023-11-15T06:27:32.580 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:32.576+0000 7f9987367700 1 --2- 172.21.15.81:0/96378139 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99800f5410 0x7f99800f5830 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:32.580 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:32.576+0000 7f9987367700 1 -- 172.21.15.81:0/96378139 >> 172.21.15.81:0/96378139 conn(0x7f99800f0b70 msgr2=0x7f99800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:27:32.581 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:32.576+0000 7f9987367700 1 -- 172.21.15.81:0/96378139 shutdown_connections 2023-11-15T06:27:32.581 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:32.576+0000 7f9987367700 1 -- 172.21.15.81:0/96378139 wait complete. 2023-11-15T06:27:32.581 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:32.577+0000 7f9987367700 1 Processor -- start 2023-11-15T06:27:32.581 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:32.577+0000 7f9987367700 1 -- start start 2023-11-15T06:27:32.581 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:32.577+0000 7f9987367700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99800f5410 0x7f9980109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:27:32.582 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:32.577+0000 7f9987367700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9980060220 con 0x7f99800f5410 2023-11-15T06:27:32.582 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:32.577+0000 7f9985103700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99800f5410 0x7f9980109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:27:32.582 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:32.577+0000 7f9985103700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99800f5410 0x7f9980109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:53144/0 (socket says 172.21.15.81:53144) 2023-11-15T06:27:32.582 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:32.577+0000 7f9985103700 1 -- 172.21.15.81:0/597974781 learned_addr learned my addr 172.21.15.81:0/597974781 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:27:32.582 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:32.577+0000 7f9985103700 1 -- 172.21.15.81:0/597974781 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9970005040 con 0x7f99800f5410 2023-11-15T06:27:32.583 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:32.578+0000 7f9985103700 1 --2- 172.21.15.81:0/597974781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99800f5410 0x7f9980109450 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f9970013510 tx=0x7f997000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:27:32.583 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:32.578+0000 7f9975ffb700 1 -- 172.21.15.81:0/597974781 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9970009320 con 0x7f99800f5410 2023-11-15T06:27:32.583 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:32.578+0000 7f9987367700 1 -- 172.21.15.81:0/597974781 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9980060420 con 0x7f99800f5410 2023-11-15T06:27:32.583 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:32.578+0000 7f9975ffb700 1 -- 172.21.15.81:0/597974781 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9970013f00 con 0x7f99800f5410 2023-11-15T06:27:32.583 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:32.578+0000 7f9975ffb700 1 -- 172.21.15.81:0/597974781 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9970014d90 con 0x7f99800f5410 2023-11-15T06:27:32.584 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:32.578+0000 7f9987367700 1 -- 172.21.15.81:0/597974781 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9980109b00 con 0x7f99800f5410 2023-11-15T06:27:32.585 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:32.579+0000 7f9975ffb700 1 -- 172.21.15.81:0/597974781 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f99700095f0 con 0x7f99800f5410 2023-11-15T06:27:32.585 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:32.579+0000 7f9987367700 1 -- 172.21.15.81:0/597974781 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9980059100 con 0x7f99800f5410 2023-11-15T06:27:32.585 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:32.579+0000 7f9975ffb700 1 --2- 172.21.15.81:0/597974781 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f996c02ff60 0x7f996c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:27:32.585 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:32.579+0000 7f9975ffb700 1 -- 172.21.15.81:0/597974781 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9970040fc0 con 0x7f99800f5410 2023-11-15T06:27:32.586 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:32.579+0000 7f9984902700 1 --2- 172.21.15.81:0/597974781 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f996c02ff60 0x7f996c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:27:32.586 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:32.579+0000 7f9984902700 1 --2- 172.21.15.81:0/597974781 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f996c02ff60 0x7f996c032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f997c0096a0 tx=0x7f997c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:27:32.587 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:32.584+0000 7f9975ffb700 1 -- 172.21.15.81:0/597974781 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9970014420 con 0x7f99800f5410 2023-11-15T06:27:32.790 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:32.788+0000 7f9987367700 1 -- 172.21.15.81:0/597974781 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9980045ad0 con 0x7f99800f5410 2023-11-15T06:27:32.791 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:32.789+0000 7f9975ffb700 1 -- 172.21.15.81:0/597974781 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9970010230 con 0x7f99800f5410 2023-11-15T06:27:32.791 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:27:32.791 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:27:32.793 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:32.791+0000 7f9987367700 1 -- 172.21.15.81:0/597974781 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f996c02ff60 msgr2=0x7f996c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:32.793 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:32.791+0000 7f9987367700 1 --2- 172.21.15.81:0/597974781 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f996c02ff60 0x7f996c032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f997c0096a0 tx=0x7f997c006b40).stop 2023-11-15T06:27:32.793 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:32.791+0000 7f9987367700 1 -- 172.21.15.81:0/597974781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99800f5410 msgr2=0x7f9980109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:32.793 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:32.791+0000 7f9987367700 1 --2- 172.21.15.81:0/597974781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99800f5410 0x7f9980109450 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f9970013510 tx=0x7f997000b040).stop 2023-11-15T06:27:32.794 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:32.791+0000 7f9987367700 1 -- 172.21.15.81:0/597974781 shutdown_connections 2023-11-15T06:27:32.794 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:32.791+0000 7f9987367700 1 --2- 172.21.15.81:0/597974781 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f996c02ff60 0x7f996c032420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:32.794 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:32.791+0000 7f9987367700 1 --2- 172.21.15.81:0/597974781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99800f5410 0x7f9980109450 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:32.794 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:32.791+0000 7f9987367700 1 -- 172.21.15.81:0/597974781 >> 172.21.15.81:0/597974781 conn(0x7f99800f0b70 msgr2=0x7f99800f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:27:32.795 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:32.791+0000 7f9987367700 1 -- 172.21.15.81:0/597974781 shutdown_connections 2023-11-15T06:27:32.795 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:32.791+0000 7f9987367700 1 -- 172.21.15.81:0/597974781 wait complete. 2023-11-15T06:27:32.795 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:27:33.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:32 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:27:31.792623+0000 mgr.smithi062.rwurtk (mgr.14164) 50 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:27:33.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:32 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:27:32.789784+0000 mon.smithi062 (mon.0) 143 : audit [DBG] from='client.? 172.21.15.81:0/597974781' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:27:34.147 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:27:34.148 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:27:35.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:34 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:27:33.792925+0000 mgr.smithi062.rwurtk (mgr.14164) 51 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:27:35.297 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:35.294+0000 7fd47bda7700 1 -- 172.21.15.81:0/511202438 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4740f5410 msgr2=0x7fd4740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:35.298 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:35.294+0000 7fd47bda7700 1 --2- 172.21.15.81:0/511202438 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4740f5410 0x7fd4740f5830 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7fd464009fb0 tx=0x7fd464009b30).stop 2023-11-15T06:27:35.298 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:35.294+0000 7fd47bda7700 1 -- 172.21.15.81:0/511202438 shutdown_connections 2023-11-15T06:27:35.298 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:35.294+0000 7fd47bda7700 1 --2- 172.21.15.81:0/511202438 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4740f5410 0x7fd4740f5830 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:35.299 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:35.294+0000 7fd47bda7700 1 -- 172.21.15.81:0/511202438 >> 172.21.15.81:0/511202438 conn(0x7fd4740f0b70 msgr2=0x7fd4740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:27:35.299 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:35.294+0000 7fd47bda7700 1 -- 172.21.15.81:0/511202438 shutdown_connections 2023-11-15T06:27:35.299 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:35.294+0000 7fd47bda7700 1 -- 172.21.15.81:0/511202438 wait complete. 2023-11-15T06:27:35.299 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:35.295+0000 7fd47bda7700 1 Processor -- start 2023-11-15T06:27:35.299 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:35.295+0000 7fd47bda7700 1 -- start start 2023-11-15T06:27:35.300 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:35.295+0000 7fd47bda7700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4740f5410 0x7fd474109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:27:35.300 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:35.295+0000 7fd47bda7700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd474060240 con 0x7fd4740f5410 2023-11-15T06:27:35.300 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:35.295+0000 7fd479b43700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4740f5410 0x7fd474109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:27:35.300 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:35.295+0000 7fd479b43700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4740f5410 0x7fd474109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:53166/0 (socket says 172.21.15.81:53166) 2023-11-15T06:27:35.300 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:35.295+0000 7fd479b43700 1 -- 172.21.15.81:0/2873858320 learned_addr learned my addr 172.21.15.81:0/2873858320 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:27:35.301 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:35.296+0000 7fd479b43700 1 -- 172.21.15.81:0/2873858320 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd464005040 con 0x7fd4740f5410 2023-11-15T06:27:35.301 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:35.296+0000 7fd479b43700 1 --2- 172.21.15.81:0/2873858320 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4740f5410 0x7fd474109430 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fd464013510 tx=0x7fd46400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:27:35.301 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:35.296+0000 7fd46a7fc700 1 -- 172.21.15.81:0/2873858320 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd464009320 con 0x7fd4740f5410 2023-11-15T06:27:35.301 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:35.296+0000 7fd47bda7700 1 -- 172.21.15.81:0/2873858320 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd474060440 con 0x7fd4740f5410 2023-11-15T06:27:35.301 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:35.296+0000 7fd46a7fc700 1 -- 172.21.15.81:0/2873858320 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd464013f00 con 0x7fd4740f5410 2023-11-15T06:27:35.302 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:35.296+0000 7fd46a7fc700 1 -- 172.21.15.81:0/2873858320 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd464014d90 con 0x7fd4740f5410 2023-11-15T06:27:35.302 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:35.296+0000 7fd47bda7700 1 -- 172.21.15.81:0/2873858320 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd474109ae0 con 0x7fd4740f5410 2023-11-15T06:27:35.305 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:35.297+0000 7fd46a7fc700 1 -- 172.21.15.81:0/2873858320 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd4640095f0 con 0x7fd4740f5410 2023-11-15T06:27:35.305 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:35.297+0000 7fd47bda7700 1 -- 172.21.15.81:0/2873858320 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd474045ad0 con 0x7fd4740f5410 2023-11-15T06:27:35.305 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:35.297+0000 7fd46a7fc700 1 --2- 172.21.15.81:0/2873858320 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fd46002ff60 0x7fd460032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:27:35.306 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:35.297+0000 7fd46a7fc700 1 -- 172.21.15.81:0/2873858320 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd4640410a0 con 0x7fd4740f5410 2023-11-15T06:27:35.307 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:35.297+0000 7fd479342700 1 --2- 172.21.15.81:0/2873858320 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fd46002ff60 0x7fd460032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:27:35.307 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:35.297+0000 7fd479342700 1 --2- 172.21.15.81:0/2873858320 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fd46002ff60 0x7fd460032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fd4700096a0 tx=0x7fd470006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:27:35.307 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:35.302+0000 7fd46a7fc700 1 -- 172.21.15.81:0/2873858320 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd46401d7f0 con 0x7fd4740f5410 2023-11-15T06:27:35.510 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:35.507+0000 7fd47bda7700 1 -- 172.21.15.81:0/2873858320 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd4740fae60 con 0x7fd4740f5410 2023-11-15T06:27:35.510 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:35.507+0000 7fd46a7fc700 1 -- 172.21.15.81:0/2873858320 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd464010230 con 0x7fd4740f5410 2023-11-15T06:27:35.511 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:27:35.511 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:27:35.512 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:35.509+0000 7fd47bda7700 1 -- 172.21.15.81:0/2873858320 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fd46002ff60 msgr2=0x7fd460032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:35.512 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:35.509+0000 7fd47bda7700 1 --2- 172.21.15.81:0/2873858320 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fd46002ff60 0x7fd460032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fd4700096a0 tx=0x7fd470006b40).stop 2023-11-15T06:27:35.512 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:35.509+0000 7fd47bda7700 1 -- 172.21.15.81:0/2873858320 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4740f5410 msgr2=0x7fd474109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:35.512 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:35.509+0000 7fd47bda7700 1 --2- 172.21.15.81:0/2873858320 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4740f5410 0x7fd474109430 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fd464013510 tx=0x7fd46400b040).stop 2023-11-15T06:27:35.513 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:35.509+0000 7fd47bda7700 1 -- 172.21.15.81:0/2873858320 shutdown_connections 2023-11-15T06:27:35.513 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:35.509+0000 7fd47bda7700 1 --2- 172.21.15.81:0/2873858320 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fd46002ff60 0x7fd460032420 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:35.513 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:35.509+0000 7fd47bda7700 1 --2- 172.21.15.81:0/2873858320 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4740f5410 0x7fd474109430 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:35.513 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:35.509+0000 7fd47bda7700 1 -- 172.21.15.81:0/2873858320 >> 172.21.15.81:0/2873858320 conn(0x7fd4740f0b70 msgr2=0x7fd4740f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:27:35.513 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:35.509+0000 7fd47bda7700 1 -- 172.21.15.81:0/2873858320 shutdown_connections 2023-11-15T06:27:35.514 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:35.509+0000 7fd47bda7700 1 -- 172.21.15.81:0/2873858320 wait complete. 2023-11-15T06:27:35.514 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:27:36.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:35 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:27:35.508083+0000 mon.smithi062 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.81:0/2873858320' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:27:36.897 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:27:36.898 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:27:37.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:36 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:27:35.793172+0000 mgr.smithi062.rwurtk (mgr.14164) 52 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:27:38.097 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:38.095+0000 7f301762e700 1 -- 172.21.15.81:0/366090559 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30100f5410 msgr2=0x7f30100f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:38.097 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:38.095+0000 7f301762e700 1 --2- 172.21.15.81:0/366090559 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30100f5410 0x7f30100f5830 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f300c009fb0 tx=0x7f300c009b30).stop 2023-11-15T06:27:38.098 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:38.095+0000 7f301762e700 1 -- 172.21.15.81:0/366090559 shutdown_connections 2023-11-15T06:27:38.098 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:38.095+0000 7f301762e700 1 --2- 172.21.15.81:0/366090559 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30100f5410 0x7f30100f5830 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:38.098 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:38.095+0000 7f301762e700 1 -- 172.21.15.81:0/366090559 >> 172.21.15.81:0/366090559 conn(0x7f30100f0b70 msgr2=0x7f30100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:27:38.098 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:38.095+0000 7f301762e700 1 -- 172.21.15.81:0/366090559 shutdown_connections 2023-11-15T06:27:38.099 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:38.095+0000 7f301762e700 1 -- 172.21.15.81:0/366090559 wait complete. 2023-11-15T06:27:38.099 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:38.095+0000 7f301762e700 1 Processor -- start 2023-11-15T06:27:38.099 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:38.095+0000 7f301762e700 1 -- start start 2023-11-15T06:27:38.099 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:38.096+0000 7f301762e700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30100f5410 0x7f301010b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:27:38.099 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:38.096+0000 7f301762e700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3010107240 con 0x7f30100f5410 2023-11-15T06:27:38.099 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:38.096+0000 7f30153ca700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30100f5410 0x7f301010b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:27:38.100 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:38.096+0000 7f30153ca700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30100f5410 0x7f301010b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:53196/0 (socket says 172.21.15.81:53196) 2023-11-15T06:27:38.100 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:38.096+0000 7f30153ca700 1 -- 172.21.15.81:0/541429794 learned_addr learned my addr 172.21.15.81:0/541429794 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:27:38.101 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:38.096+0000 7f30153ca700 1 -- 172.21.15.81:0/541429794 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f300c005040 con 0x7f30100f5410 2023-11-15T06:27:38.102 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:38.096+0000 7f30153ca700 1 --2- 172.21.15.81:0/541429794 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30100f5410 0x7f301010b850 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f300c013510 tx=0x7f300c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:27:38.102 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:38.096+0000 7f3005ffb700 1 -- 172.21.15.81:0/541429794 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f300c009320 con 0x7f30100f5410 2023-11-15T06:27:38.102 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:38.096+0000 7f301762e700 1 -- 172.21.15.81:0/541429794 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3010060220 con 0x7f30100f5410 2023-11-15T06:27:38.102 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:38.096+0000 7f3005ffb700 1 -- 172.21.15.81:0/541429794 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f300c013f00 con 0x7f30100f5410 2023-11-15T06:27:38.102 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:38.096+0000 7f3005ffb700 1 -- 172.21.15.81:0/541429794 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f300c014d90 con 0x7f30100f5410 2023-11-15T06:27:38.103 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:38.097+0000 7f301762e700 1 -- 172.21.15.81:0/541429794 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f301010bd40 con 0x7f30100f5410 2023-11-15T06:27:38.103 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:38.097+0000 7f301762e700 1 -- 172.21.15.81:0/541429794 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3010045ad0 con 0x7f30100f5410 2023-11-15T06:27:38.103 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:38.098+0000 7f3005ffb700 1 -- 172.21.15.81:0/541429794 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f300c014420 con 0x7f30100f5410 2023-11-15T06:27:38.103 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:38.098+0000 7f3005ffb700 1 --2- 172.21.15.81:0/541429794 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f2ffc02ff60 0x7f2ffc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:27:38.103 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:38.098+0000 7f3005ffb700 1 -- 172.21.15.81:0/541429794 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f300c041050 con 0x7f30100f5410 2023-11-15T06:27:38.104 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:38.098+0000 7f3014bc9700 1 --2- 172.21.15.81:0/541429794 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f2ffc02ff60 0x7f2ffc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:27:38.104 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:38.098+0000 7f3014bc9700 1 --2- 172.21.15.81:0/541429794 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f2ffc02ff60 0x7f2ffc032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f30000096a0 tx=0x7f3000006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:27:38.104 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:38.103+0000 7f3005ffb700 1 -- 172.21.15.81:0/541429794 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f300c0095f0 con 0x7f30100f5410 2023-11-15T06:27:38.309 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:38.307+0000 7f301762e700 1 -- 172.21.15.81:0/541429794 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f30100fae60 con 0x7f30100f5410 2023-11-15T06:27:38.310 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:38.308+0000 7f3005ffb700 1 -- 172.21.15.81:0/541429794 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f300c01db70 con 0x7f30100f5410 2023-11-15T06:27:38.310 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:27:38.310 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:27:38.311 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:38.310+0000 7f301762e700 1 -- 172.21.15.81:0/541429794 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f2ffc02ff60 msgr2=0x7f2ffc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:38.312 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:38.310+0000 7f301762e700 1 --2- 172.21.15.81:0/541429794 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f2ffc02ff60 0x7f2ffc032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f30000096a0 tx=0x7f3000006b40).stop 2023-11-15T06:27:38.312 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:38.310+0000 7f301762e700 1 -- 172.21.15.81:0/541429794 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30100f5410 msgr2=0x7f301010b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:38.312 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:38.310+0000 7f301762e700 1 --2- 172.21.15.81:0/541429794 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30100f5410 0x7f301010b850 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f300c013510 tx=0x7f300c00b040).stop 2023-11-15T06:27:38.312 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:38.310+0000 7f301762e700 1 -- 172.21.15.81:0/541429794 shutdown_connections 2023-11-15T06:27:38.312 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:38.310+0000 7f301762e700 1 --2- 172.21.15.81:0/541429794 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f2ffc02ff60 0x7f2ffc032420 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:38.313 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:38.310+0000 7f301762e700 1 --2- 172.21.15.81:0/541429794 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30100f5410 0x7f301010b850 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:38.313 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:38.310+0000 7f301762e700 1 -- 172.21.15.81:0/541429794 >> 172.21.15.81:0/541429794 conn(0x7f30100f0b70 msgr2=0x7f30100f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:27:38.314 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:38.310+0000 7f301762e700 1 -- 172.21.15.81:0/541429794 shutdown_connections 2023-11-15T06:27:38.314 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:38.310+0000 7f301762e700 1 -- 172.21.15.81:0/541429794 wait complete. 2023-11-15T06:27:38.315 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:27:38.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:38 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:27:37.793555+0000 mgr.smithi062.rwurtk (mgr.14164) 53 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:27:38.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:38 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:27:38.308852+0000 mon.smithi062 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.81:0/541429794' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:27:39.738 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:27:39.739 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:27:41.009 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:41.006+0000 7fc7570fd700 1 -- 172.21.15.81:0/542584370 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7500f47c0 msgr2=0x7fc7500f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:41.010 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:41.006+0000 7fc7570fd700 1 --2- 172.21.15.81:0/542584370 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7500f47c0 0x7fc7500f4be0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fc744009fb0 tx=0x7fc744009b30).stop 2023-11-15T06:27:41.010 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:41.006+0000 7fc7570fd700 1 -- 172.21.15.81:0/542584370 shutdown_connections 2023-11-15T06:27:41.010 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:41.006+0000 7fc7570fd700 1 --2- 172.21.15.81:0/542584370 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7500f47c0 0x7fc7500f4be0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:41.010 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:41.006+0000 7fc7570fd700 1 -- 172.21.15.81:0/542584370 >> 172.21.15.81:0/542584370 conn(0x7fc7500f02e0 msgr2=0x7fc7500f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:27:41.010 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:41.006+0000 7fc7570fd700 1 -- 172.21.15.81:0/542584370 shutdown_connections 2023-11-15T06:27:41.011 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:41.007+0000 7fc7570fd700 1 -- 172.21.15.81:0/542584370 wait complete. 2023-11-15T06:27:41.011 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:41.007+0000 7fc7570fd700 1 Processor -- start 2023-11-15T06:27:41.011 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:41.007+0000 7fc7570fd700 1 -- start start 2023-11-15T06:27:41.011 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:41.008+0000 7fc7570fd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7500f47c0 0x7fc750106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:27:41.012 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:41.008+0000 7fc7570fd700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc750106bf0 con 0x7fc7500f47c0 2023-11-15T06:27:41.014 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:41.008+0000 7fc754e99700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7500f47c0 0x7fc750106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:27:41.014 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:41.008+0000 7fc754e99700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7500f47c0 0x7fc750106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:53208/0 (socket says 172.21.15.81:53208) 2023-11-15T06:27:41.014 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:41.008+0000 7fc754e99700 1 -- 172.21.15.81:0/2578330022 learned_addr learned my addr 172.21.15.81:0/2578330022 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:27:41.015 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:41.008+0000 7fc754e99700 1 -- 172.21.15.81:0/2578330022 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc744005040 con 0x7fc7500f47c0 2023-11-15T06:27:41.016 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:41.008+0000 7fc754e99700 1 --2- 172.21.15.81:0/2578330022 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7500f47c0 0x7fc750106700 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fc744014000 tx=0x7fc74400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:27:41.016 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:41.008+0000 7fc74d7fa700 1 -- 172.21.15.81:0/2578330022 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7440095f0 con 0x7fc7500f47c0 2023-11-15T06:27:41.016 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:41.008+0000 7fc7570fd700 1 -- 172.21.15.81:0/2578330022 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc750106df0 con 0x7fc7500f47c0 2023-11-15T06:27:41.016 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:41.009+0000 7fc74d7fa700 1 -- 172.21.15.81:0/2578330022 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc744007e70 con 0x7fc7500f47c0 2023-11-15T06:27:41.017 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:41.009+0000 7fc74d7fa700 1 -- 172.21.15.81:0/2578330022 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc744014ea0 con 0x7fc7500f47c0 2023-11-15T06:27:41.017 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:41.009+0000 7fc7570fd700 1 -- 172.21.15.81:0/2578330022 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc750107280 con 0x7fc7500f47c0 2023-11-15T06:27:41.017 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:41.009+0000 7fc74d7fa700 1 -- 172.21.15.81:0/2578330022 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc744025070 con 0x7fc7500f47c0 2023-11-15T06:27:41.017 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:41.009+0000 7fc7570fd700 1 -- 172.21.15.81:0/2578330022 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc750045ad0 con 0x7fc7500f47c0 2023-11-15T06:27:41.018 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:41.009+0000 7fc74d7fa700 1 --2- 172.21.15.81:0/2578330022 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fc74002ffb0 0x7fc740032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:27:41.018 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:41.009+0000 7fc74d7fa700 1 -- 172.21.15.81:0/2578330022 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc7440409e0 con 0x7fc7500f47c0 2023-11-15T06:27:41.018 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:41.010+0000 7fc74ffff700 1 --2- 172.21.15.81:0/2578330022 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fc74002ffb0 0x7fc740032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:27:41.018 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:41.010+0000 7fc74ffff700 1 --2- 172.21.15.81:0/2578330022 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fc74002ffb0 0x7fc740032470 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fc73c0096a0 tx=0x7fc73c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:27:41.019 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:41.014+0000 7fc74d7fa700 1 -- 172.21.15.81:0/2578330022 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc74400f4e0 con 0x7fc7500f47c0 2023-11-15T06:27:41.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:40 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:27:39.793854+0000 mgr.smithi062.rwurtk (mgr.14164) 54 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:27:41.220 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:41.218+0000 7fc7570fd700 1 -- 172.21.15.81:0/2578330022 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc750108130 con 0x7fc7500f47c0 2023-11-15T06:27:41.221 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:41.219+0000 7fc74d7fa700 1 -- 172.21.15.81:0/2578330022 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc74400f300 con 0x7fc7500f47c0 2023-11-15T06:27:41.221 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:27:41.221 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:27:41.222 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:41.220+0000 7fc7570fd700 1 -- 172.21.15.81:0/2578330022 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fc74002ffb0 msgr2=0x7fc740032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:41.222 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:41.220+0000 7fc7570fd700 1 --2- 172.21.15.81:0/2578330022 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fc74002ffb0 0x7fc740032470 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fc73c0096a0 tx=0x7fc73c006b40).stop 2023-11-15T06:27:41.223 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:41.220+0000 7fc7570fd700 1 -- 172.21.15.81:0/2578330022 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7500f47c0 msgr2=0x7fc750106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:41.223 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:41.220+0000 7fc7570fd700 1 --2- 172.21.15.81:0/2578330022 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7500f47c0 0x7fc750106700 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fc744014000 tx=0x7fc74400b040).stop 2023-11-15T06:27:41.223 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:41.220+0000 7fc7570fd700 1 -- 172.21.15.81:0/2578330022 shutdown_connections 2023-11-15T06:27:41.223 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:41.220+0000 7fc7570fd700 1 --2- 172.21.15.81:0/2578330022 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fc74002ffb0 0x7fc740032470 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:41.223 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:41.220+0000 7fc7570fd700 1 --2- 172.21.15.81:0/2578330022 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7500f47c0 0x7fc750106700 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:41.224 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:41.220+0000 7fc7570fd700 1 -- 172.21.15.81:0/2578330022 >> 172.21.15.81:0/2578330022 conn(0x7fc7500f02e0 msgr2=0x7fc7500f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:27:41.224 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:41.220+0000 7fc7570fd700 1 -- 172.21.15.81:0/2578330022 shutdown_connections 2023-11-15T06:27:41.224 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:41.220+0000 7fc7570fd700 1 -- 172.21.15.81:0/2578330022 wait complete. 2023-11-15T06:27:41.224 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:27:42.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:41 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:27:41.219577+0000 mon.smithi062 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.81:0/2578330022' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:27:42.602 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:27:42.603 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:27:43.171 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:42 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:27:41.794141+0000 mgr.smithi062.rwurtk (mgr.14164) 55 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:27:43.847 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:43.844+0000 7fed37fc3700 1 -- 172.21.15.81:0/2574142991 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed300f6510 msgr2=0x7fed300f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:43.848 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:43.844+0000 7fed37fc3700 1 --2- 172.21.15.81:0/2574142991 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed300f6510 0x7fed300f6930 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fed24009fb0 tx=0x7fed24009b30).stop 2023-11-15T06:27:43.848 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:43.844+0000 7fed37fc3700 1 -- 172.21.15.81:0/2574142991 shutdown_connections 2023-11-15T06:27:43.848 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:43.844+0000 7fed37fc3700 1 --2- 172.21.15.81:0/2574142991 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed300f6510 0x7fed300f6930 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:43.848 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:43.844+0000 7fed37fc3700 1 -- 172.21.15.81:0/2574142991 >> 172.21.15.81:0/2574142991 conn(0x7fed300f1c70 msgr2=0x7fed300f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:27:43.849 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:43.844+0000 7fed37fc3700 1 -- 172.21.15.81:0/2574142991 shutdown_connections 2023-11-15T06:27:43.849 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:43.844+0000 7fed37fc3700 1 -- 172.21.15.81:0/2574142991 wait complete. 2023-11-15T06:27:43.849 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:43.845+0000 7fed37fc3700 1 Processor -- start 2023-11-15T06:27:43.849 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:43.845+0000 7fed37fc3700 1 -- start start 2023-11-15T06:27:43.849 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:43.845+0000 7fed37fc3700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed300f6510 0x7fed30106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:27:43.850 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:43.845+0000 7fed37fc3700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed30106c20 con 0x7fed300f6510 2023-11-15T06:27:43.850 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:43.845+0000 7fed35d5f700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed300f6510 0x7fed30106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:27:43.850 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:43.845+0000 7fed35d5f700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed300f6510 0x7fed30106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:49038/0 (socket says 172.21.15.81:49038) 2023-11-15T06:27:43.852 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:43.845+0000 7fed35d5f700 1 -- 172.21.15.81:0/2487921582 learned_addr learned my addr 172.21.15.81:0/2487921582 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:27:43.852 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:43.845+0000 7fed35d5f700 1 -- 172.21.15.81:0/2487921582 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed24005040 con 0x7fed300f6510 2023-11-15T06:27:43.852 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:43.845+0000 7fed35d5f700 1 --2- 172.21.15.81:0/2487921582 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed300f6510 0x7fed30106730 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fed24013510 tx=0x7fed2400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:27:43.852 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:43.846+0000 7fed227fc700 1 -- 172.21.15.81:0/2487921582 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed24009320 con 0x7fed300f6510 2023-11-15T06:27:43.852 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:43.846+0000 7fed37fc3700 1 -- 172.21.15.81:0/2487921582 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed30106e20 con 0x7fed300f6510 2023-11-15T06:27:43.853 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:43.846+0000 7fed227fc700 1 -- 172.21.15.81:0/2487921582 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fed24013f00 con 0x7fed300f6510 2023-11-15T06:27:43.853 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:43.846+0000 7fed227fc700 1 -- 172.21.15.81:0/2487921582 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed24014d90 con 0x7fed300f6510 2023-11-15T06:27:43.853 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:43.846+0000 7fed37fc3700 1 -- 172.21.15.81:0/2487921582 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed301072b0 con 0x7fed300f6510 2023-11-15T06:27:43.854 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:43.846+0000 7fed227fc700 1 -- 172.21.15.81:0/2487921582 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fed24014420 con 0x7fed300f6510 2023-11-15T06:27:43.854 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:43.846+0000 7fed37fc3700 1 -- 172.21.15.81:0/2487921582 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed30045ad0 con 0x7fed300f6510 2023-11-15T06:27:43.855 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:43.847+0000 7fed227fc700 1 --2- 172.21.15.81:0/2487921582 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fed1c02ff60 0x7fed1c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:27:43.855 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:43.847+0000 7fed227fc700 1 -- 172.21.15.81:0/2487921582 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fed240410c0 con 0x7fed300f6510 2023-11-15T06:27:43.855 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:43.847+0000 7fed3555e700 1 --2- 172.21.15.81:0/2487921582 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fed1c02ff60 0x7fed1c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:27:43.855 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:43.847+0000 7fed3555e700 1 --2- 172.21.15.81:0/2487921582 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fed1c02ff60 0x7fed1c032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fed2c0096a0 tx=0x7fed2c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:27:43.855 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:43.851+0000 7fed227fc700 1 -- 172.21.15.81:0/2487921582 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fed240095f0 con 0x7fed300f6510 2023-11-15T06:27:44.056 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:44.055+0000 7fed37fc3700 1 -- 172.21.15.81:0/2487921582 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fed300fbf60 con 0x7fed300f6510 2023-11-15T06:27:44.057 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:44.055+0000 7fed227fc700 1 -- 172.21.15.81:0/2487921582 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fed2401db70 con 0x7fed300f6510 2023-11-15T06:27:44.057 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:27:44.058 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:27:44.058 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:44.057+0000 7fed37fc3700 1 -- 172.21.15.81:0/2487921582 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fed1c02ff60 msgr2=0x7fed1c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:44.059 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:44.057+0000 7fed37fc3700 1 --2- 172.21.15.81:0/2487921582 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fed1c02ff60 0x7fed1c032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fed2c0096a0 tx=0x7fed2c006b40).stop 2023-11-15T06:27:44.059 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:44.057+0000 7fed37fc3700 1 -- 172.21.15.81:0/2487921582 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed300f6510 msgr2=0x7fed30106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:44.059 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:44.057+0000 7fed37fc3700 1 --2- 172.21.15.81:0/2487921582 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed300f6510 0x7fed30106730 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fed24013510 tx=0x7fed2400b040).stop 2023-11-15T06:27:44.059 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:44.057+0000 7fed37fc3700 1 -- 172.21.15.81:0/2487921582 shutdown_connections 2023-11-15T06:27:44.059 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:44.057+0000 7fed37fc3700 1 --2- 172.21.15.81:0/2487921582 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fed1c02ff60 0x7fed1c032420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:44.060 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:44.057+0000 7fed37fc3700 1 --2- 172.21.15.81:0/2487921582 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed300f6510 0x7fed30106730 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:44.060 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:44.057+0000 7fed37fc3700 1 -- 172.21.15.81:0/2487921582 >> 172.21.15.81:0/2487921582 conn(0x7fed300f1c70 msgr2=0x7fed300f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:27:44.060 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:44.057+0000 7fed37fc3700 1 -- 172.21.15.81:0/2487921582 shutdown_connections 2023-11-15T06:27:44.060 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:44.057+0000 7fed37fc3700 1 -- 172.21.15.81:0/2487921582 wait complete. 2023-11-15T06:27:44.061 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:27:45.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:44 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:27:43.794461+0000 mgr.smithi062.rwurtk (mgr.14164) 56 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:27:45.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:44 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:27:44.056147+0000 mon.smithi062 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.81:0/2487921582' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:27:45.493 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:27:45.494 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:27:46.761 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:46.755+0000 7fa1f5faf700 1 -- 172.21.15.81:0/749204436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f00601e0 msgr2=0x7fa1f00f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:46.761 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:46.755+0000 7fa1f5faf700 1 --2- 172.21.15.81:0/749204436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f00601e0 0x7fa1f00f9190 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fa1e0009fb0 tx=0x7fa1e0009b30).stop 2023-11-15T06:27:46.762 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:46.755+0000 7fa1f5faf700 1 -- 172.21.15.81:0/749204436 shutdown_connections 2023-11-15T06:27:46.762 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:46.755+0000 7fa1f5faf700 1 --2- 172.21.15.81:0/749204436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f00601e0 0x7fa1f00f9190 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:46.762 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:46.755+0000 7fa1f5faf700 1 -- 172.21.15.81:0/749204436 >> 172.21.15.81:0/749204436 conn(0x7fa1f00f0b70 msgr2=0x7fa1f00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:27:46.763 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:46.755+0000 7fa1f5faf700 1 -- 172.21.15.81:0/749204436 shutdown_connections 2023-11-15T06:27:46.763 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:46.755+0000 7fa1f5faf700 1 -- 172.21.15.81:0/749204436 wait complete. 2023-11-15T06:27:46.764 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:46.756+0000 7fa1f5faf700 1 Processor -- start 2023-11-15T06:27:46.764 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:46.756+0000 7fa1f5faf700 1 -- start start 2023-11-15T06:27:46.764 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:46.756+0000 7fa1f5faf700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f00601e0 0x7fa1f01066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:27:46.765 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:46.756+0000 7fa1f5faf700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f0106bd0 con 0x7fa1f00601e0 2023-11-15T06:27:46.765 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:46.757+0000 7fa1ef7fe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f00601e0 0x7fa1f01066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:27:46.766 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:46.757+0000 7fa1ef7fe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f00601e0 0x7fa1f01066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:49062/0 (socket says 172.21.15.81:49062) 2023-11-15T06:27:46.766 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:46.757+0000 7fa1ef7fe700 1 -- 172.21.15.81:0/3597579805 learned_addr learned my addr 172.21.15.81:0/3597579805 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:27:46.766 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:46.757+0000 7fa1ef7fe700 1 -- 172.21.15.81:0/3597579805 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e0005040 con 0x7fa1f00601e0 2023-11-15T06:27:46.767 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:46.757+0000 7fa1ef7fe700 1 --2- 172.21.15.81:0/3597579805 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f00601e0 0x7fa1f01066e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fa1e0004dd0 tx=0x7fa1e0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:27:46.767 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:46.757+0000 7fa1d7fff700 1 -- 172.21.15.81:0/3597579805 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1e0018070 con 0x7fa1f00601e0 2023-11-15T06:27:46.767 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:46.757+0000 7fa1f5faf700 1 -- 172.21.15.81:0/3597579805 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1f0106dd0 con 0x7fa1f00601e0 2023-11-15T06:27:46.768 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:46.757+0000 7fa1f5faf700 1 -- 172.21.15.81:0/3597579805 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1f0107260 con 0x7fa1f00601e0 2023-11-15T06:27:46.768 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:46.757+0000 7fa1d7fff700 1 -- 172.21.15.81:0/3597579805 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa1e0013410 con 0x7fa1f00601e0 2023-11-15T06:27:46.768 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:46.758+0000 7fa1d7fff700 1 -- 172.21.15.81:0/3597579805 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1e001c3c0 con 0x7fa1f00601e0 2023-11-15T06:27:46.769 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:46.758+0000 7fa1d7fff700 1 -- 172.21.15.81:0/3597579805 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa1e001d420 con 0x7fa1f00601e0 2023-11-15T06:27:46.769 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:46.758+0000 7fa1f5faf700 1 -- 172.21.15.81:0/3597579805 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1f0045ad0 con 0x7fa1f00601e0 2023-11-15T06:27:46.769 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:46.759+0000 7fa1d7fff700 1 --2- 172.21.15.81:0/3597579805 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fa1d802ff60 0x7fa1d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:27:46.770 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:46.759+0000 7fa1d7fff700 1 -- 172.21.15.81:0/3597579805 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa1e00416e0 con 0x7fa1f00601e0 2023-11-15T06:27:46.770 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:46.759+0000 7fa1eeffd700 1 --2- 172.21.15.81:0/3597579805 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fa1d802ff60 0x7fa1d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:27:46.770 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:46.759+0000 7fa1eeffd700 1 --2- 172.21.15.81:0/3597579805 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fa1d802ff60 0x7fa1d8032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fa1e40096a0 tx=0x7fa1e4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:27:46.771 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:46.763+0000 7fa1d7fff700 1 -- 172.21.15.81:0/3597579805 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa1e001b370 con 0x7fa1f00601e0 2023-11-15T06:27:46.970 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:46.968+0000 7fa1f5faf700 1 -- 172.21.15.81:0/3597579805 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa1f0108110 con 0x7fa1f00601e0 2023-11-15T06:27:46.971 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:46.969+0000 7fa1d7fff700 1 -- 172.21.15.81:0/3597579805 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa1e0004490 con 0x7fa1f00601e0 2023-11-15T06:27:46.971 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:27:46.972 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:27:46.973 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:46.970+0000 7fa1f5faf700 1 -- 172.21.15.81:0/3597579805 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fa1d802ff60 msgr2=0x7fa1d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:46.973 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:46.970+0000 7fa1f5faf700 1 --2- 172.21.15.81:0/3597579805 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fa1d802ff60 0x7fa1d8032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fa1e40096a0 tx=0x7fa1e4006b40).stop 2023-11-15T06:27:46.974 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:46.970+0000 7fa1f5faf700 1 -- 172.21.15.81:0/3597579805 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f00601e0 msgr2=0x7fa1f01066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:46.974 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:46.970+0000 7fa1f5faf700 1 --2- 172.21.15.81:0/3597579805 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f00601e0 0x7fa1f01066e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fa1e0004dd0 tx=0x7fa1e0009b30).stop 2023-11-15T06:27:46.974 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:46.970+0000 7fa1f5faf700 1 -- 172.21.15.81:0/3597579805 shutdown_connections 2023-11-15T06:27:46.975 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:46.970+0000 7fa1f5faf700 1 --2- 172.21.15.81:0/3597579805 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fa1d802ff60 0x7fa1d8032420 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:46.975 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:46.970+0000 7fa1f5faf700 1 --2- 172.21.15.81:0/3597579805 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f00601e0 0x7fa1f01066e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:46.975 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:46.970+0000 7fa1f5faf700 1 -- 172.21.15.81:0/3597579805 >> 172.21.15.81:0/3597579805 conn(0x7fa1f00f0b70 msgr2=0x7fa1f00f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:27:46.976 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:46.970+0000 7fa1f5faf700 1 -- 172.21.15.81:0/3597579805 shutdown_connections 2023-11-15T06:27:46.976 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:46.970+0000 7fa1f5faf700 1 -- 172.21.15.81:0/3597579805 wait complete. 2023-11-15T06:27:46.976 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:27:47.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:46 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:27:45.794715+0000 mgr.smithi062.rwurtk (mgr.14164) 57 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:27:48.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:47 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:27:46.969443+0000 mon.smithi062 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.81:0/3597579805' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:27:48.361 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:27:48.361 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:27:49.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:48 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:27:47.795063+0000 mgr.smithi062.rwurtk (mgr.14164) 58 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:27:49.607 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:49.602+0000 7fabec140700 1 -- 172.21.15.81:0/1433129173 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabe40f5410 msgr2=0x7fabe40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:49.607 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:49.602+0000 7fabec140700 1 --2- 172.21.15.81:0/1433129173 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabe40f5410 0x7fabe40f5830 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fabd4009fb0 tx=0x7fabd4009b30).stop 2023-11-15T06:27:49.607 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:49.602+0000 7fabec140700 1 -- 172.21.15.81:0/1433129173 shutdown_connections 2023-11-15T06:27:49.608 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:49.602+0000 7fabec140700 1 --2- 172.21.15.81:0/1433129173 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabe40f5410 0x7fabe40f5830 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:49.608 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:49.602+0000 7fabec140700 1 -- 172.21.15.81:0/1433129173 >> 172.21.15.81:0/1433129173 conn(0x7fabe40f0b70 msgr2=0x7fabe40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:27:49.608 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:49.602+0000 7fabec140700 1 -- 172.21.15.81:0/1433129173 shutdown_connections 2023-11-15T06:27:49.609 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:49.602+0000 7fabec140700 1 -- 172.21.15.81:0/1433129173 wait complete. 2023-11-15T06:27:49.609 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:49.602+0000 7fabec140700 1 Processor -- start 2023-11-15T06:27:49.609 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:49.602+0000 7fabec140700 1 -- start start 2023-11-15T06:27:49.610 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:49.603+0000 7fabec140700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabe40f5410 0x7fabe410b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:27:49.610 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:49.603+0000 7fabec140700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe4107220 con 0x7fabe40f5410 2023-11-15T06:27:49.610 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:49.603+0000 7fabe9edc700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabe40f5410 0x7fabe410b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:27:49.610 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:49.603+0000 7fabe9edc700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabe40f5410 0x7fabe410b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:49072/0 (socket says 172.21.15.81:49072) 2023-11-15T06:27:49.611 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:49.603+0000 7fabe9edc700 1 -- 172.21.15.81:0/3771327443 learned_addr learned my addr 172.21.15.81:0/3771327443 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:27:49.611 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:49.603+0000 7fabe9edc700 1 -- 172.21.15.81:0/3771327443 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabd4005040 con 0x7fabe40f5410 2023-11-15T06:27:49.611 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:49.603+0000 7fabe9edc700 1 --2- 172.21.15.81:0/3771327443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabe40f5410 0x7fabe410b830 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fabd4013510 tx=0x7fabd400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:27:49.611 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:49.603+0000 7fabda7fc700 1 -- 172.21.15.81:0/3771327443 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fabd4009320 con 0x7fabe40f5410 2023-11-15T06:27:49.611 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:49.603+0000 7fabec140700 1 -- 172.21.15.81:0/3771327443 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabe4060240 con 0x7fabe40f5410 2023-11-15T06:27:49.612 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:49.603+0000 7fabda7fc700 1 -- 172.21.15.81:0/3771327443 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fabd4013f00 con 0x7fabe40f5410 2023-11-15T06:27:49.612 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:49.603+0000 7fabda7fc700 1 -- 172.21.15.81:0/3771327443 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fabd4014d90 con 0x7fabe40f5410 2023-11-15T06:27:49.612 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:49.603+0000 7fabec140700 1 -- 172.21.15.81:0/3771327443 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabe410bd20 con 0x7fabe40f5410 2023-11-15T06:27:49.613 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:49.604+0000 7fabda7fc700 1 -- 172.21.15.81:0/3771327443 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fabd40095f0 con 0x7fabe40f5410 2023-11-15T06:27:49.613 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:49.604+0000 7fabec140700 1 -- 172.21.15.81:0/3771327443 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabe4045ad0 con 0x7fabe40f5410 2023-11-15T06:27:49.613 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:49.604+0000 7fabda7fc700 1 --2- 172.21.15.81:0/3771327443 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fabd002ff60 0x7fabd0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:27:49.613 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:49.604+0000 7fabda7fc700 1 -- 172.21.15.81:0/3771327443 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fabd40410c0 con 0x7fabe40f5410 2023-11-15T06:27:49.613 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:49.605+0000 7fabe96db700 1 --2- 172.21.15.81:0/3771327443 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fabd002ff60 0x7fabd0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:27:49.614 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:49.605+0000 7fabe96db700 1 --2- 172.21.15.81:0/3771327443 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fabd002ff60 0x7fabd0032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fabe00096a0 tx=0x7fabe0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:27:49.614 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:49.609+0000 7fabda7fc700 1 -- 172.21.15.81:0/3771327443 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fabd4014420 con 0x7fabe40f5410 2023-11-15T06:27:49.814 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:49.813+0000 7fabec140700 1 -- 172.21.15.81:0/3771327443 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fabe40fae60 con 0x7fabe40f5410 2023-11-15T06:27:49.815 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:49.813+0000 7fabda7fc700 1 -- 172.21.15.81:0/3771327443 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fabd4010230 con 0x7fabe40f5410 2023-11-15T06:27:49.816 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:27:49.816 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:27:49.817 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:49.815+0000 7fabec140700 1 -- 172.21.15.81:0/3771327443 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fabd002ff60 msgr2=0x7fabd0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:49.817 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:49.815+0000 7fabec140700 1 --2- 172.21.15.81:0/3771327443 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fabd002ff60 0x7fabd0032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fabe00096a0 tx=0x7fabe0006b40).stop 2023-11-15T06:27:49.817 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:49.815+0000 7fabec140700 1 -- 172.21.15.81:0/3771327443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabe40f5410 msgr2=0x7fabe410b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:49.817 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:49.815+0000 7fabec140700 1 --2- 172.21.15.81:0/3771327443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabe40f5410 0x7fabe410b830 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fabd4013510 tx=0x7fabd400b040).stop 2023-11-15T06:27:49.818 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:49.815+0000 7fabec140700 1 -- 172.21.15.81:0/3771327443 shutdown_connections 2023-11-15T06:27:49.818 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:49.815+0000 7fabec140700 1 --2- 172.21.15.81:0/3771327443 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fabd002ff60 0x7fabd0032420 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:49.818 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:49.815+0000 7fabec140700 1 --2- 172.21.15.81:0/3771327443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabe40f5410 0x7fabe410b830 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:49.819 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:49.815+0000 7fabec140700 1 -- 172.21.15.81:0/3771327443 >> 172.21.15.81:0/3771327443 conn(0x7fabe40f0b70 msgr2=0x7fabe40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:27:49.819 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:49.815+0000 7fabec140700 1 -- 172.21.15.81:0/3771327443 shutdown_connections 2023-11-15T06:27:49.819 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:49.815+0000 7fabec140700 1 -- 172.21.15.81:0/3771327443 wait complete. 2023-11-15T06:27:49.819 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:27:50.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:49 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:27:49.814182+0000 mon.smithi062 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.81:0/3771327443' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:27:51.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:50 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:27:49.795434+0000 mgr.smithi062.rwurtk (mgr.14164) 59 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:27:51.244 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:27:51.245 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:27:52.526 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:52.524+0000 7f3e1a30a700 1 -- 172.21.15.81:0/1312722310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e140f51a0 msgr2=0x7f3e140f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:52.526 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:52.524+0000 7f3e1a30a700 1 --2- 172.21.15.81:0/1312722310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e140f51a0 0x7f3e140f55c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f3e10009fb0 tx=0x7f3e10009b30).stop 2023-11-15T06:27:52.526 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:52.524+0000 7f3e1a30a700 1 -- 172.21.15.81:0/1312722310 shutdown_connections 2023-11-15T06:27:52.527 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:52.524+0000 7f3e1a30a700 1 --2- 172.21.15.81:0/1312722310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e140f51a0 0x7f3e140f55c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:52.527 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:52.524+0000 7f3e1a30a700 1 -- 172.21.15.81:0/1312722310 >> 172.21.15.81:0/1312722310 conn(0x7f3e140f09a0 msgr2=0x7f3e140f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:27:52.527 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:52.524+0000 7f3e1a30a700 1 -- 172.21.15.81:0/1312722310 shutdown_connections 2023-11-15T06:27:52.527 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:52.524+0000 7f3e1a30a700 1 -- 172.21.15.81:0/1312722310 wait complete. 2023-11-15T06:27:52.529 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:52.524+0000 7f3e1a30a700 1 Processor -- start 2023-11-15T06:27:52.529 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:52.525+0000 7f3e1a30a700 1 -- start start 2023-11-15T06:27:52.529 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:52.525+0000 7f3e1a30a700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e140f51a0 0x7f3e141042a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:27:52.530 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:52.525+0000 7f3e1a30a700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e14104790 con 0x7f3e140f51a0 2023-11-15T06:27:52.532 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:52.525+0000 7f3e19308700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e140f51a0 0x7f3e141042a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:27:52.532 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:52.525+0000 7f3e19308700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e140f51a0 0x7f3e141042a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:56676/0 (socket says 172.21.15.81:56676) 2023-11-15T06:27:52.532 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:52.525+0000 7f3e19308700 1 -- 172.21.15.81:0/3053042667 learned_addr learned my addr 172.21.15.81:0/3053042667 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:27:52.532 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:52.525+0000 7f3e19308700 1 -- 172.21.15.81:0/3053042667 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e10005040 con 0x7f3e140f51a0 2023-11-15T06:27:52.532 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:52.525+0000 7f3e19308700 1 --2- 172.21.15.81:0/3053042667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e140f51a0 0x7f3e141042a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f3e10013510 tx=0x7f3e1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:27:52.533 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:52.525+0000 7f3e09ffb700 1 -- 172.21.15.81:0/3053042667 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e10009320 con 0x7f3e140f51a0 2023-11-15T06:27:52.533 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:52.526+0000 7f3e1a30a700 1 -- 172.21.15.81:0/3053042667 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e14104990 con 0x7f3e140f51a0 2023-11-15T06:27:52.533 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:52.526+0000 7f3e09ffb700 1 -- 172.21.15.81:0/3053042667 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3e10013f00 con 0x7f3e140f51a0 2023-11-15T06:27:52.533 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:52.526+0000 7f3e09ffb700 1 -- 172.21.15.81:0/3053042667 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e10014dc0 con 0x7f3e140f51a0 2023-11-15T06:27:52.533 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:52.526+0000 7f3e1a30a700 1 -- 172.21.15.81:0/3053042667 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e14104e20 con 0x7f3e140f51a0 2023-11-15T06:27:52.534 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:52.526+0000 7f3e09ffb700 1 -- 172.21.15.81:0/3053042667 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3e10014420 con 0x7f3e140f51a0 2023-11-15T06:27:52.534 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:52.527+0000 7f3e1a30a700 1 -- 172.21.15.81:0/3053042667 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e14045ad0 con 0x7f3e140f51a0 2023-11-15T06:27:52.534 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:52.527+0000 7f3e09ffb700 1 --2- 172.21.15.81:0/3053042667 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f3e00032200 0x7f3e000346c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:27:52.534 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:52.527+0000 7f3e09ffb700 1 -- 172.21.15.81:0/3053042667 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3e10041060 con 0x7f3e140f51a0 2023-11-15T06:27:52.534 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:52.527+0000 7f3e18b07700 1 --2- 172.21.15.81:0/3053042667 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f3e00032200 0x7f3e000346c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:27:52.535 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:52.527+0000 7f3e18b07700 1 --2- 172.21.15.81:0/3053042667 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f3e00032200 0x7f3e000346c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f3e040096a0 tx=0x7f3e04006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:27:52.535 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:52.531+0000 7f3e09ffb700 1 -- 172.21.15.81:0/3053042667 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3e1006d2b0 con 0x7f3e140f51a0 2023-11-15T06:27:52.739 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:52.737+0000 7f3e1a30a700 1 -- 172.21.15.81:0/3053042667 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3e14105cd0 con 0x7f3e140f51a0 2023-11-15T06:27:52.740 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:52.737+0000 7f3e09ffb700 1 -- 172.21.15.81:0/3053042667 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3e100095f0 con 0x7f3e140f51a0 2023-11-15T06:27:52.741 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:27:52.741 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:27:52.742 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:52.739+0000 7f3e1a30a700 1 -- 172.21.15.81:0/3053042667 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f3e00032200 msgr2=0x7f3e000346c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:52.743 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:52.739+0000 7f3e1a30a700 1 --2- 172.21.15.81:0/3053042667 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f3e00032200 0x7f3e000346c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f3e040096a0 tx=0x7f3e04006b40).stop 2023-11-15T06:27:52.743 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:52.739+0000 7f3e1a30a700 1 -- 172.21.15.81:0/3053042667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e140f51a0 msgr2=0x7f3e141042a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:52.744 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:52.739+0000 7f3e1a30a700 1 --2- 172.21.15.81:0/3053042667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e140f51a0 0x7f3e141042a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f3e10013510 tx=0x7f3e1000b040).stop 2023-11-15T06:27:52.744 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:52.739+0000 7f3e1a30a700 1 -- 172.21.15.81:0/3053042667 shutdown_connections 2023-11-15T06:27:52.744 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:52.739+0000 7f3e1a30a700 1 --2- 172.21.15.81:0/3053042667 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f3e00032200 0x7f3e000346c0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:52.745 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:52.739+0000 7f3e1a30a700 1 --2- 172.21.15.81:0/3053042667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e140f51a0 0x7f3e141042a0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:52.745 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:52.739+0000 7f3e1a30a700 1 -- 172.21.15.81:0/3053042667 >> 172.21.15.81:0/3053042667 conn(0x7f3e140f09a0 msgr2=0x7f3e140f1400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:27:52.745 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:52.739+0000 7f3e1a30a700 1 -- 172.21.15.81:0/3053042667 shutdown_connections 2023-11-15T06:27:52.746 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:52.739+0000 7f3e1a30a700 1 -- 172.21.15.81:0/3053042667 wait complete. 2023-11-15T06:27:52.746 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:27:53.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:52 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:27:51.795688+0000 mgr.smithi062.rwurtk (mgr.14164) 60 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:27:53.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:52 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:27:52.738228+0000 mon.smithi062 (mon.0) 150 : audit [DBG] from='client.? 172.21.15.81:0/3053042667' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:27:54.137 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:27:54.138 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:27:55.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:54 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:27:53.795987+0000 mgr.smithi062.rwurtk (mgr.14164) 61 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:27:55.169 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:54 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:27:53.915181+0000 mon.smithi062 (mon.0) 151 : audit [INF] from='mgr.14164 172.21.15.62:0/147479783' entity='mgr.smithi062.rwurtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi062.rwurtk/mirror_snapshot_schedule"}]: dispatch 2023-11-15T06:27:55.169 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:54 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:27:53.917380+0000 mon.smithi062 (mon.0) 152 : audit [INF] from='mgr.14164 172.21.15.62:0/147479783' entity='mgr.smithi062.rwurtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi062.rwurtk/trash_purge_schedule"}]: dispatch 2023-11-15T06:27:55.386 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:55.376+0000 7f868d5f0700 1 -- 172.21.15.81:0/2454016266 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86880f5410 msgr2=0x7f86880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:55.387 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:55.376+0000 7f868d5f0700 1 --2- 172.21.15.81:0/2454016266 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86880f5410 0x7f86880f5830 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f867c009fb0 tx=0x7f867c009b30).stop 2023-11-15T06:27:55.387 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:55.376+0000 7f868d5f0700 1 -- 172.21.15.81:0/2454016266 shutdown_connections 2023-11-15T06:27:55.388 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:55.376+0000 7f868d5f0700 1 --2- 172.21.15.81:0/2454016266 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86880f5410 0x7f86880f5830 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:55.388 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:55.376+0000 7f868d5f0700 1 -- 172.21.15.81:0/2454016266 >> 172.21.15.81:0/2454016266 conn(0x7f86880f0b70 msgr2=0x7f86880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:27:55.388 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:55.376+0000 7f868d5f0700 1 -- 172.21.15.81:0/2454016266 shutdown_connections 2023-11-15T06:27:55.389 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:55.376+0000 7f868d5f0700 1 -- 172.21.15.81:0/2454016266 wait complete. 2023-11-15T06:27:55.389 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:55.377+0000 7f868d5f0700 1 Processor -- start 2023-11-15T06:27:55.389 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:55.377+0000 7f868d5f0700 1 -- start start 2023-11-15T06:27:55.390 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:55.377+0000 7f868d5f0700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86880f5410 0x7f86881093c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:27:55.390 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:55.377+0000 7f868d5f0700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8688060280 con 0x7f86880f5410 2023-11-15T06:27:55.390 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:55.378+0000 7f8686ffd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86880f5410 0x7f86881093c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:27:55.391 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:55.378+0000 7f8686ffd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86880f5410 0x7f86881093c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:56702/0 (socket says 172.21.15.81:56702) 2023-11-15T06:27:55.391 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:55.378+0000 7f8686ffd700 1 -- 172.21.15.81:0/1737637332 learned_addr learned my addr 172.21.15.81:0/1737637332 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:27:55.392 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:55.378+0000 7f8686ffd700 1 -- 172.21.15.81:0/1737637332 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f867c005040 con 0x7f86880f5410 2023-11-15T06:27:55.392 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:55.378+0000 7f8686ffd700 1 --2- 172.21.15.81:0/1737637332 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86880f5410 0x7f86881093c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f867c013510 tx=0x7f867c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:27:55.392 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:55.378+0000 7f866f7fe700 1 -- 172.21.15.81:0/1737637332 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f867c009320 con 0x7f86880f5410 2023-11-15T06:27:55.392 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:55.378+0000 7f866f7fe700 1 -- 172.21.15.81:0/1737637332 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f867c013f00 con 0x7f86880f5410 2023-11-15T06:27:55.393 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:55.378+0000 7f868d5f0700 1 -- 172.21.15.81:0/1737637332 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8688060480 con 0x7f86880f5410 2023-11-15T06:27:55.393 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:55.378+0000 7f866f7fe700 1 -- 172.21.15.81:0/1737637332 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f867c014d90 con 0x7f86880f5410 2023-11-15T06:27:55.393 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:55.378+0000 7f868d5f0700 1 -- 172.21.15.81:0/1737637332 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8688109af0 con 0x7f86880f5410 2023-11-15T06:27:55.394 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:55.379+0000 7f866f7fe700 1 -- 172.21.15.81:0/1737637332 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f867c0095f0 con 0x7f86880f5410 2023-11-15T06:27:55.394 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:55.379+0000 7f868d5f0700 1 -- 172.21.15.81:0/1737637332 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8688045ad0 con 0x7f86880f5410 2023-11-15T06:27:55.395 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:55.379+0000 7f866f7fe700 1 --2- 172.21.15.81:0/1737637332 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f867002ffb0 0x7f8670032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:27:55.396 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:55.379+0000 7f866f7fe700 1 -- 172.21.15.81:0/1737637332 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f867c041040 con 0x7f86880f5410 2023-11-15T06:27:55.398 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:55.379+0000 7f86867fc700 1 --2- 172.21.15.81:0/1737637332 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f867002ffb0 0x7f8670032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:27:55.399 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:55.380+0000 7f86867fc700 1 --2- 172.21.15.81:0/1737637332 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f867002ffb0 0x7f8670032470 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f86780096a0 tx=0x7f8678006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:27:55.399 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:55.384+0000 7f866f7fe700 1 -- 172.21.15.81:0/1737637332 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f867c014420 con 0x7f86880f5410 2023-11-15T06:27:55.590 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:55.587+0000 7f868d5f0700 1 -- 172.21.15.81:0/1737637332 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f868810a310 con 0x7f86880f5410 2023-11-15T06:27:55.591 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:55.588+0000 7f866f7fe700 1 -- 172.21.15.81:0/1737637332 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f867c01d7f0 con 0x7f86880f5410 2023-11-15T06:27:55.591 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:27:55.591 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:27:55.592 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:55.589+0000 7f868d5f0700 1 -- 172.21.15.81:0/1737637332 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f867002ffb0 msgr2=0x7f8670032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:55.592 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:55.589+0000 7f868d5f0700 1 --2- 172.21.15.81:0/1737637332 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f867002ffb0 0x7f8670032470 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f86780096a0 tx=0x7f8678006b40).stop 2023-11-15T06:27:55.593 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:55.589+0000 7f868d5f0700 1 -- 172.21.15.81:0/1737637332 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86880f5410 msgr2=0x7f86881093c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:55.593 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:55.589+0000 7f868d5f0700 1 --2- 172.21.15.81:0/1737637332 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86880f5410 0x7f86881093c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f867c013510 tx=0x7f867c00b040).stop 2023-11-15T06:27:55.593 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:55.589+0000 7f868d5f0700 1 -- 172.21.15.81:0/1737637332 shutdown_connections 2023-11-15T06:27:55.593 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:55.589+0000 7f868d5f0700 1 --2- 172.21.15.81:0/1737637332 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f867002ffb0 0x7f8670032470 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:55.593 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:55.589+0000 7f868d5f0700 1 --2- 172.21.15.81:0/1737637332 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86880f5410 0x7f86881093c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:55.594 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:55.589+0000 7f868d5f0700 1 -- 172.21.15.81:0/1737637332 >> 172.21.15.81:0/1737637332 conn(0x7f86880f0b70 msgr2=0x7f86880f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:27:55.594 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:55.589+0000 7f868d5f0700 1 -- 172.21.15.81:0/1737637332 shutdown_connections 2023-11-15T06:27:55.594 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:55.589+0000 7f868d5f0700 1 -- 172.21.15.81:0/1737637332 wait complete. 2023-11-15T06:27:55.595 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:27:56.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:55 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:27:55.589516+0000 mon.smithi062 (mon.0) 153 : audit [DBG] from='client.? 172.21.15.81:0/1737637332' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:27:57.012 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:27:57.013 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:27:57.171 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:56 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:27:55.796317+0000 mgr.smithi062.rwurtk (mgr.14164) 62 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:27:58.283 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:58.277+0000 7f61c9f85700 1 -- 172.21.15.81:0/1454846521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61c4060220 msgr2=0x7f61c40f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:58.283 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:58.277+0000 7f61c9f85700 1 --2- 172.21.15.81:0/1454846521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61c4060220 0x7f61c40f9150 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f61b4009fb0 tx=0x7f61b4009b30).stop 2023-11-15T06:27:58.283 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:58.277+0000 7f61c9f85700 1 -- 172.21.15.81:0/1454846521 shutdown_connections 2023-11-15T06:27:58.283 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:58.277+0000 7f61c9f85700 1 --2- 172.21.15.81:0/1454846521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61c4060220 0x7f61c40f9150 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:58.284 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:58.277+0000 7f61c9f85700 1 -- 172.21.15.81:0/1454846521 >> 172.21.15.81:0/1454846521 conn(0x7f61c40f0b70 msgr2=0x7f61c40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:27:58.284 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:58.277+0000 7f61c9f85700 1 -- 172.21.15.81:0/1454846521 shutdown_connections 2023-11-15T06:27:58.284 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:58.278+0000 7f61c9f85700 1 -- 172.21.15.81:0/1454846521 wait complete. 2023-11-15T06:27:58.284 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:58.278+0000 7f61c9f85700 1 Processor -- start 2023-11-15T06:27:58.284 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:58.278+0000 7f61c9f85700 1 -- start start 2023-11-15T06:27:58.285 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:58.278+0000 7f61c9f85700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61c4060220 0x7f61c41066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:27:58.285 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:58.278+0000 7f61c9f85700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c4106bd0 con 0x7f61c4060220 2023-11-15T06:27:58.285 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:58.278+0000 7f61c37fe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61c4060220 0x7f61c41066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:27:58.285 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:58.279+0000 7f61c37fe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61c4060220 0x7f61c41066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:56718/0 (socket says 172.21.15.81:56718) 2023-11-15T06:27:58.285 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:58.279+0000 7f61c37fe700 1 -- 172.21.15.81:0/1478488708 learned_addr learned my addr 172.21.15.81:0/1478488708 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:27:58.286 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:58.279+0000 7f61c37fe700 1 -- 172.21.15.81:0/1478488708 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b4005040 con 0x7f61c4060220 2023-11-15T06:27:58.286 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:58.279+0000 7f61c37fe700 1 --2- 172.21.15.81:0/1478488708 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61c4060220 0x7f61c41066e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f61b4013510 tx=0x7f61b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:27:58.286 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:58.279+0000 7f61abfff700 1 -- 172.21.15.81:0/1478488708 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61b4009320 con 0x7f61c4060220 2023-11-15T06:27:58.286 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:58.279+0000 7f61c9f85700 1 -- 172.21.15.81:0/1478488708 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61c4106dd0 con 0x7f61c4060220 2023-11-15T06:27:58.287 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:58.279+0000 7f61abfff700 1 -- 172.21.15.81:0/1478488708 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f61b4013f00 con 0x7f61c4060220 2023-11-15T06:27:58.287 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:58.279+0000 7f61abfff700 1 -- 172.21.15.81:0/1478488708 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61b4014dc0 con 0x7f61c4060220 2023-11-15T06:27:58.287 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:58.279+0000 7f61c9f85700 1 -- 172.21.15.81:0/1478488708 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61c4107260 con 0x7f61c4060220 2023-11-15T06:27:58.289 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:58.280+0000 7f61abfff700 1 -- 172.21.15.81:0/1478488708 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f61b40095f0 con 0x7f61c4060220 2023-11-15T06:27:58.289 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:58.280+0000 7f61c9f85700 1 -- 172.21.15.81:0/1478488708 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61c4045ad0 con 0x7f61c4060220 2023-11-15T06:27:58.290 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:58.280+0000 7f61abfff700 1 --2- 172.21.15.81:0/1478488708 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f61ac02ffb0 0x7f61ac032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:27:58.290 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:58.280+0000 7f61abfff700 1 -- 172.21.15.81:0/1478488708 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f61b4041100 con 0x7f61c4060220 2023-11-15T06:27:58.291 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:58.280+0000 7f61c2ffd700 1 --2- 172.21.15.81:0/1478488708 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f61ac02ffb0 0x7f61ac032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:27:58.291 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:58.281+0000 7f61c2ffd700 1 --2- 172.21.15.81:0/1478488708 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f61ac02ffb0 0x7f61ac032470 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f61b80096a0 tx=0x7f61b8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:27:58.291 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:58.285+0000 7f61abfff700 1 -- 172.21.15.81:0/1478488708 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f61b4014420 con 0x7f61c4060220 2023-11-15T06:27:58.494 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:58.492+0000 7f61c9f85700 1 -- 172.21.15.81:0/1478488708 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f61c4108110 con 0x7f61c4060220 2023-11-15T06:27:58.495 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:58.492+0000 7f61abfff700 1 -- 172.21.15.81:0/1478488708 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f61b4010230 con 0x7f61c4060220 2023-11-15T06:27:58.496 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:27:58.496 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:27:58.497 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:58.494+0000 7f61c9f85700 1 -- 172.21.15.81:0/1478488708 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f61ac02ffb0 msgr2=0x7f61ac032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:58.497 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:58.494+0000 7f61c9f85700 1 --2- 172.21.15.81:0/1478488708 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f61ac02ffb0 0x7f61ac032470 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f61b80096a0 tx=0x7f61b8006b40).stop 2023-11-15T06:27:58.497 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:58.494+0000 7f61c9f85700 1 -- 172.21.15.81:0/1478488708 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61c4060220 msgr2=0x7f61c41066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:27:58.497 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:58.494+0000 7f61c9f85700 1 --2- 172.21.15.81:0/1478488708 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61c4060220 0x7f61c41066e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f61b4013510 tx=0x7f61b400b040).stop 2023-11-15T06:27:58.498 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:58.494+0000 7f61c9f85700 1 -- 172.21.15.81:0/1478488708 shutdown_connections 2023-11-15T06:27:58.498 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:58.494+0000 7f61c9f85700 1 --2- 172.21.15.81:0/1478488708 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f61ac02ffb0 0x7f61ac032470 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:58.498 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:58.494+0000 7f61c9f85700 1 --2- 172.21.15.81:0/1478488708 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61c4060220 0x7f61c41066e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:27:58.498 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:58.494+0000 7f61c9f85700 1 -- 172.21.15.81:0/1478488708 >> 172.21.15.81:0/1478488708 conn(0x7f61c40f0b70 msgr2=0x7f61c40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:27:58.499 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:58.494+0000 7f61c9f85700 1 -- 172.21.15.81:0/1478488708 shutdown_connections 2023-11-15T06:27:58.499 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:27:58.494+0000 7f61c9f85700 1 -- 172.21.15.81:0/1478488708 wait complete. 2023-11-15T06:27:58.499 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:27:58.920 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:58 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:27:57.796636+0000 mgr.smithi062.rwurtk (mgr.14164) 63 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:27:58.921 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:27:58 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:27:58.494240+0000 mon.smithi062 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.81:0/1478488708' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:27:59.886 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:27:59.887 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:28:01.103 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:01.097+0000 7f5070c9c700 1 -- 172.21.15.81:0/3488492352 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f506c0f51a0 msgr2=0x7f506c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:01.104 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:01.097+0000 7f5070c9c700 1 --2- 172.21.15.81:0/3488492352 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f506c0f51a0 0x7f506c0f55c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f505c009fb0 tx=0x7f505c009b30).stop 2023-11-15T06:28:01.104 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:01.097+0000 7f5070c9c700 1 -- 172.21.15.81:0/3488492352 shutdown_connections 2023-11-15T06:28:01.105 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:01.097+0000 7f5070c9c700 1 --2- 172.21.15.81:0/3488492352 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f506c0f51a0 0x7f506c0f55c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:01.105 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:01.097+0000 7f5070c9c700 1 -- 172.21.15.81:0/3488492352 >> 172.21.15.81:0/3488492352 conn(0x7f506c0f09a0 msgr2=0x7f506c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:28:01.106 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:01.097+0000 7f5070c9c700 1 -- 172.21.15.81:0/3488492352 shutdown_connections 2023-11-15T06:28:01.106 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:01.097+0000 7f5070c9c700 1 -- 172.21.15.81:0/3488492352 wait complete. 2023-11-15T06:28:01.109 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:01.098+0000 7f5070c9c700 1 Processor -- start 2023-11-15T06:28:01.110 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:01.098+0000 7f5070c9c700 1 -- start start 2023-11-15T06:28:01.110 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:01.098+0000 7f5070c9c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f506c0f51a0 0x7f506c1064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:28:01.111 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:01.098+0000 7f5070c9c700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f506c1069d0 con 0x7f506c0f51a0 2023-11-15T06:28:01.111 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:01.098+0000 7f506b7fe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f506c0f51a0 0x7f506c1064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:28:01.111 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:01.098+0000 7f506b7fe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f506c0f51a0 0x7f506c1064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:56742/0 (socket says 172.21.15.81:56742) 2023-11-15T06:28:01.112 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:01.098+0000 7f506b7fe700 1 -- 172.21.15.81:0/2797043021 learned_addr learned my addr 172.21.15.81:0/2797043021 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:28:01.112 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:01.099+0000 7f506b7fe700 1 -- 172.21.15.81:0/2797043021 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f505c005040 con 0x7f506c0f51a0 2023-11-15T06:28:01.112 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:01.099+0000 7f506b7fe700 1 --2- 172.21.15.81:0/2797043021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f506c0f51a0 0x7f506c1064e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f505c013510 tx=0x7f505c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:28:01.113 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:01.099+0000 7f5053fff700 1 -- 172.21.15.81:0/2797043021 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f505c009320 con 0x7f506c0f51a0 2023-11-15T06:28:01.113 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:01.099+0000 7f5070c9c700 1 -- 172.21.15.81:0/2797043021 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f506c106bd0 con 0x7f506c0f51a0 2023-11-15T06:28:01.113 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:01.099+0000 7f5053fff700 1 -- 172.21.15.81:0/2797043021 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f505c013f00 con 0x7f506c0f51a0 2023-11-15T06:28:01.114 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:01.099+0000 7f5053fff700 1 -- 172.21.15.81:0/2797043021 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f505c014dc0 con 0x7f506c0f51a0 2023-11-15T06:28:01.114 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:01.099+0000 7f5070c9c700 1 -- 172.21.15.81:0/2797043021 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f506c107060 con 0x7f506c0f51a0 2023-11-15T06:28:01.114 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:01.100+0000 7f5053fff700 1 -- 172.21.15.81:0/2797043021 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f505c01d3d0 con 0x7f506c0f51a0 2023-11-15T06:28:01.115 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:01.100+0000 7f5053fff700 1 --2- 172.21.15.81:0/2797043021 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f505402ff60 0x7f5054032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:28:01.115 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:01.100+0000 7f5070c9c700 1 -- 172.21.15.81:0/2797043021 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f506c045ad0 con 0x7f506c0f51a0 2023-11-15T06:28:01.115 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:01.100+0000 7f5053fff700 1 -- 172.21.15.81:0/2797043021 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f505c0413c0 con 0x7f506c0f51a0 2023-11-15T06:28:01.116 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:01.100+0000 7f506affd700 1 --2- 172.21.15.81:0/2797043021 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f505402ff60 0x7f5054032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:28:01.116 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:01.101+0000 7f506affd700 1 --2- 172.21.15.81:0/2797043021 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f505402ff60 0x7f5054032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f50600096a0 tx=0x7f5060006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:28:01.117 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:01.105+0000 7f5053fff700 1 -- 172.21.15.81:0/2797043021 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f505c01cad0 con 0x7f506c0f51a0 2023-11-15T06:28:01.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:00 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:27:59.797040+0000 mgr.smithi062.rwurtk (mgr.14164) 64 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:28:01.313 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:01.310+0000 7f5070c9c700 1 -- 172.21.15.81:0/2797043021 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f506c107f10 con 0x7f506c0f51a0 2023-11-15T06:28:01.314 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:01.311+0000 7f5053fff700 1 -- 172.21.15.81:0/2797043021 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f505c01cad0 con 0x7f506c0f51a0 2023-11-15T06:28:01.314 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:28:01.315 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:28:01.316 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:01.313+0000 7f5070c9c700 1 -- 172.21.15.81:0/2797043021 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f505402ff60 msgr2=0x7f5054032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:01.316 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:01.313+0000 7f5070c9c700 1 --2- 172.21.15.81:0/2797043021 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f505402ff60 0x7f5054032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f50600096a0 tx=0x7f5060006b40).stop 2023-11-15T06:28:01.317 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:01.313+0000 7f5070c9c700 1 -- 172.21.15.81:0/2797043021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f506c0f51a0 msgr2=0x7f506c1064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:01.317 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:01.313+0000 7f5070c9c700 1 --2- 172.21.15.81:0/2797043021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f506c0f51a0 0x7f506c1064e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f505c013510 tx=0x7f505c00b040).stop 2023-11-15T06:28:01.317 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:01.313+0000 7f5070c9c700 1 -- 172.21.15.81:0/2797043021 shutdown_connections 2023-11-15T06:28:01.318 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:01.313+0000 7f5070c9c700 1 --2- 172.21.15.81:0/2797043021 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f505402ff60 0x7f5054032420 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:01.318 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:01.313+0000 7f5070c9c700 1 --2- 172.21.15.81:0/2797043021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f506c0f51a0 0x7f506c1064e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:01.318 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:01.313+0000 7f5070c9c700 1 -- 172.21.15.81:0/2797043021 >> 172.21.15.81:0/2797043021 conn(0x7f506c0f09a0 msgr2=0x7f506c05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:28:01.318 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:01.313+0000 7f5070c9c700 1 -- 172.21.15.81:0/2797043021 shutdown_connections 2023-11-15T06:28:01.320 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:01.313+0000 7f5070c9c700 1 -- 172.21.15.81:0/2797043021 wait complete. 2023-11-15T06:28:01.320 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:28:02.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:01 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:28:01.312978+0000 mon.smithi062 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.81:0/2797043021' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:28:02.688 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:28:02.689 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:28:03.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:28:01.797414+0000 mgr.smithi062.rwurtk (mgr.14164) 65 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:28:03.850 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:03.846+0000 7fcee2491700 1 -- 172.21.15.81:0/487521810 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcedc0f6510 msgr2=0x7fcedc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:03.851 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:03.846+0000 7fcee2491700 1 --2- 172.21.15.81:0/487521810 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcedc0f6510 0x7fcedc0f6930 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fcecc009fb0 tx=0x7fcecc009b30).stop 2023-11-15T06:28:03.851 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:03.846+0000 7fcee2491700 1 -- 172.21.15.81:0/487521810 shutdown_connections 2023-11-15T06:28:03.851 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:03.846+0000 7fcee2491700 1 --2- 172.21.15.81:0/487521810 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcedc0f6510 0x7fcedc0f6930 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:03.852 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:03.846+0000 7fcee2491700 1 -- 172.21.15.81:0/487521810 >> 172.21.15.81:0/487521810 conn(0x7fcedc0f1c70 msgr2=0x7fcedc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:28:03.852 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:03.846+0000 7fcee2491700 1 -- 172.21.15.81:0/487521810 shutdown_connections 2023-11-15T06:28:03.852 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:03.846+0000 7fcee2491700 1 -- 172.21.15.81:0/487521810 wait complete. 2023-11-15T06:28:03.852 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:03.847+0000 7fcee2491700 1 Processor -- start 2023-11-15T06:28:03.852 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:03.847+0000 7fcee2491700 1 -- start start 2023-11-15T06:28:03.853 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:03.847+0000 7fcee2491700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcedc0f6510 0x7fcedc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:28:03.853 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:03.847+0000 7fcee2491700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcedc106c20 con 0x7fcedc0f6510 2023-11-15T06:28:03.853 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:03.847+0000 7fcedbfff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcedc0f6510 0x7fcedc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:28:03.853 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:03.847+0000 7fcedbfff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcedc0f6510 0x7fcedc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:52096/0 (socket says 172.21.15.81:52096) 2023-11-15T06:28:03.853 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:03.847+0000 7fcedbfff700 1 -- 172.21.15.81:0/3792676361 learned_addr learned my addr 172.21.15.81:0/3792676361 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:28:03.854 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:03.848+0000 7fcedbfff700 1 -- 172.21.15.81:0/3792676361 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcecc005040 con 0x7fcedc0f6510 2023-11-15T06:28:03.854 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:03.848+0000 7fcedbfff700 1 --2- 172.21.15.81:0/3792676361 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcedc0f6510 0x7fcedc106730 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fcecc013510 tx=0x7fcecc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:28:03.854 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:03.848+0000 7fced8ff9700 1 -- 172.21.15.81:0/3792676361 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcecc009320 con 0x7fcedc0f6510 2023-11-15T06:28:03.854 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:03.848+0000 7fcee2491700 1 -- 172.21.15.81:0/3792676361 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcedc106e20 con 0x7fcedc0f6510 2023-11-15T06:28:03.854 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:03.848+0000 7fced8ff9700 1 -- 172.21.15.81:0/3792676361 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcecc013f00 con 0x7fcedc0f6510 2023-11-15T06:28:03.855 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:03.848+0000 7fced8ff9700 1 -- 172.21.15.81:0/3792676361 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcecc014d90 con 0x7fcedc0f6510 2023-11-15T06:28:03.855 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:03.848+0000 7fcee2491700 1 -- 172.21.15.81:0/3792676361 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcedc1072b0 con 0x7fcedc0f6510 2023-11-15T06:28:03.856 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:03.849+0000 7fced8ff9700 1 -- 172.21.15.81:0/3792676361 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcecc0095f0 con 0x7fcedc0f6510 2023-11-15T06:28:03.856 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:03.849+0000 7fcee2491700 1 -- 172.21.15.81:0/3792676361 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcedc045ad0 con 0x7fcedc0f6510 2023-11-15T06:28:03.856 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:03.849+0000 7fced8ff9700 1 --2- 172.21.15.81:0/3792676361 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fcec402ff60 0x7fcec4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:28:03.856 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:03.849+0000 7fced8ff9700 1 -- 172.21.15.81:0/3792676361 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcecc040fd0 con 0x7fcedc0f6510 2023-11-15T06:28:03.857 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:03.849+0000 7fcedb7fe700 1 --2- 172.21.15.81:0/3792676361 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fcec402ff60 0x7fcec4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:28:03.857 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:03.850+0000 7fcedb7fe700 1 --2- 172.21.15.81:0/3792676361 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fcec402ff60 0x7fcec4032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fced00096a0 tx=0x7fced0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:28:03.857 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:03.854+0000 7fced8ff9700 1 -- 172.21.15.81:0/3792676361 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcecc014420 con 0x7fcedc0f6510 2023-11-15T06:28:04.064 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:04.060+0000 7fcee2491700 1 -- 172.21.15.81:0/3792676361 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcedc0fbf60 con 0x7fcedc0f6510 2023-11-15T06:28:04.064 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:04.061+0000 7fced8ff9700 1 -- 172.21.15.81:0/3792676361 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcecc01d7f0 con 0x7fcedc0f6510 2023-11-15T06:28:04.065 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:28:04.065 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:28:04.066 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:04.063+0000 7fcee2491700 1 -- 172.21.15.81:0/3792676361 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fcec402ff60 msgr2=0x7fcec4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:04.066 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:04.063+0000 7fcee2491700 1 --2- 172.21.15.81:0/3792676361 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fcec402ff60 0x7fcec4032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fced00096a0 tx=0x7fced0006b40).stop 2023-11-15T06:28:04.066 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:04.063+0000 7fcee2491700 1 -- 172.21.15.81:0/3792676361 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcedc0f6510 msgr2=0x7fcedc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:04.067 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:04.063+0000 7fcee2491700 1 --2- 172.21.15.81:0/3792676361 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcedc0f6510 0x7fcedc106730 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fcecc013510 tx=0x7fcecc00b040).stop 2023-11-15T06:28:04.067 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:04.063+0000 7fcee2491700 1 -- 172.21.15.81:0/3792676361 shutdown_connections 2023-11-15T06:28:04.067 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:04.063+0000 7fcee2491700 1 --2- 172.21.15.81:0/3792676361 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fcec402ff60 0x7fcec4032420 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:04.067 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:04.063+0000 7fcee2491700 1 --2- 172.21.15.81:0/3792676361 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcedc0f6510 0x7fcedc106730 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:04.067 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:04.063+0000 7fcee2491700 1 -- 172.21.15.81:0/3792676361 >> 172.21.15.81:0/3792676361 conn(0x7fcedc0f1c70 msgr2=0x7fcedc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:28:04.068 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:04.063+0000 7fcee2491700 1 -- 172.21.15.81:0/3792676361 shutdown_connections 2023-11-15T06:28:04.068 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:04.063+0000 7fcee2491700 1 -- 172.21.15.81:0/3792676361 wait complete. 2023-11-15T06:28:04.068 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:28:05.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:04 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:28:03.797713+0000 mgr.smithi062.rwurtk (mgr.14164) 66 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:28:05.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:04 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:28:04.062721+0000 mon.smithi062 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.81:0/3792676361' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:28:05.512 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:28:05.513 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:28:06.759 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:06.756+0000 7f0754dd9700 1 -- 172.21.15.81:0/2882476941 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07500f51a0 msgr2=0x7f07500f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:06.760 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:06.756+0000 7f0754dd9700 1 --2- 172.21.15.81:0/2882476941 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07500f51a0 0x7f07500f55c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f0740009fb0 tx=0x7f0740009b30).stop 2023-11-15T06:28:06.760 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:06.756+0000 7f0754dd9700 1 -- 172.21.15.81:0/2882476941 shutdown_connections 2023-11-15T06:28:06.760 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:06.756+0000 7f0754dd9700 1 --2- 172.21.15.81:0/2882476941 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07500f51a0 0x7f07500f55c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:06.760 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:06.756+0000 7f0754dd9700 1 -- 172.21.15.81:0/2882476941 >> 172.21.15.81:0/2882476941 conn(0x7f07500f09a0 msgr2=0x7f07500f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:28:06.760 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:06.756+0000 7f0754dd9700 1 -- 172.21.15.81:0/2882476941 shutdown_connections 2023-11-15T06:28:06.761 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:06.756+0000 7f0754dd9700 1 -- 172.21.15.81:0/2882476941 wait complete. 2023-11-15T06:28:06.761 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:06.757+0000 7f0754dd9700 1 Processor -- start 2023-11-15T06:28:06.763 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:06.757+0000 7f0754dd9700 1 -- start start 2023-11-15T06:28:06.763 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:06.757+0000 7f0754dd9700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07500f51a0 0x7f07501091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:28:06.763 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:06.757+0000 7f0754dd9700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0750060240 con 0x7f07500f51a0 2023-11-15T06:28:06.763 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:06.757+0000 7f074f7fe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07500f51a0 0x7f07501091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:28:06.764 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:06.757+0000 7f074f7fe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07500f51a0 0x7f07501091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:52106/0 (socket says 172.21.15.81:52106) 2023-11-15T06:28:06.764 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:06.757+0000 7f074f7fe700 1 -- 172.21.15.81:0/457087908 learned_addr learned my addr 172.21.15.81:0/457087908 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:28:06.764 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:06.758+0000 7f074f7fe700 1 -- 172.21.15.81:0/457087908 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0740005040 con 0x7f07500f51a0 2023-11-15T06:28:06.764 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:06.758+0000 7f074f7fe700 1 --2- 172.21.15.81:0/457087908 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07500f51a0 0x7f07501091b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f0740013510 tx=0x7f074000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:28:06.765 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:06.758+0000 7f0737fff700 1 -- 172.21.15.81:0/457087908 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0740009320 con 0x7f07500f51a0 2023-11-15T06:28:06.765 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:06.758+0000 7f0754dd9700 1 -- 172.21.15.81:0/457087908 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0750060440 con 0x7f07500f51a0 2023-11-15T06:28:06.765 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:06.758+0000 7f0737fff700 1 -- 172.21.15.81:0/457087908 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0740013f00 con 0x7f07500f51a0 2023-11-15T06:28:06.765 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:06.758+0000 7f0737fff700 1 -- 172.21.15.81:0/457087908 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0740014dc0 con 0x7f07500f51a0 2023-11-15T06:28:06.765 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:06.758+0000 7f0754dd9700 1 -- 172.21.15.81:0/457087908 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0750109860 con 0x7f07500f51a0 2023-11-15T06:28:06.766 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:06.759+0000 7f0737fff700 1 -- 172.21.15.81:0/457087908 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0740014420 con 0x7f07500f51a0 2023-11-15T06:28:06.766 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:06.759+0000 7f0754dd9700 1 -- 172.21.15.81:0/457087908 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0750045ad0 con 0x7f07500f51a0 2023-11-15T06:28:06.766 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:06.759+0000 7f0737fff700 1 --2- 172.21.15.81:0/457087908 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f073802ff60 0x7f0738032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:28:06.766 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:06.759+0000 7f0737fff700 1 -- 172.21.15.81:0/457087908 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0740041050 con 0x7f07500f51a0 2023-11-15T06:28:06.766 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:06.759+0000 7f074effd700 1 --2- 172.21.15.81:0/457087908 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f073802ff60 0x7f0738032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:28:06.767 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:06.760+0000 7f074effd700 1 --2- 172.21.15.81:0/457087908 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f073802ff60 0x7f0738032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f07440096a0 tx=0x7f0744006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:28:06.767 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:06.764+0000 7f0737fff700 1 -- 172.21.15.81:0/457087908 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f07400095f0 con 0x7f07500f51a0 2023-11-15T06:28:06.970 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:06.967+0000 7f0754dd9700 1 -- 172.21.15.81:0/457087908 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f075010a080 con 0x7f07500f51a0 2023-11-15T06:28:06.971 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:06.968+0000 7f0737fff700 1 -- 172.21.15.81:0/457087908 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f074001cb10 con 0x7f07500f51a0 2023-11-15T06:28:06.971 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:28:06.971 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:28:06.973 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:06.970+0000 7f0754dd9700 1 -- 172.21.15.81:0/457087908 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f073802ff60 msgr2=0x7f0738032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:06.973 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:06.970+0000 7f0754dd9700 1 --2- 172.21.15.81:0/457087908 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f073802ff60 0x7f0738032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f07440096a0 tx=0x7f0744006b40).stop 2023-11-15T06:28:06.973 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:06.970+0000 7f0754dd9700 1 -- 172.21.15.81:0/457087908 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07500f51a0 msgr2=0x7f07501091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:06.973 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:06.970+0000 7f0754dd9700 1 --2- 172.21.15.81:0/457087908 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07500f51a0 0x7f07501091b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f0740013510 tx=0x7f074000b040).stop 2023-11-15T06:28:06.974 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:06.970+0000 7f0754dd9700 1 -- 172.21.15.81:0/457087908 shutdown_connections 2023-11-15T06:28:06.974 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:06.970+0000 7f0754dd9700 1 --2- 172.21.15.81:0/457087908 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f073802ff60 0x7f0738032420 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:06.974 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:06.970+0000 7f0754dd9700 1 --2- 172.21.15.81:0/457087908 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07500f51a0 0x7f07501091b0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:06.975 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:06.970+0000 7f0754dd9700 1 -- 172.21.15.81:0/457087908 >> 172.21.15.81:0/457087908 conn(0x7f07500f09a0 msgr2=0x7f07500f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:28:06.975 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:06.970+0000 7f0754dd9700 1 -- 172.21.15.81:0/457087908 shutdown_connections 2023-11-15T06:28:06.976 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:06.970+0000 7f0754dd9700 1 -- 172.21.15.81:0/457087908 wait complete. 2023-11-15T06:28:06.977 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:28:07.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:06 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:28:05.798050+0000 mgr.smithi062.rwurtk (mgr.14164) 67 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:28:08.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:07 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:28:06.970050+0000 mon.smithi062 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.81:0/457087908' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:28:08.352 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:28:08.353 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:28:09.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:08 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:28:07.798406+0000 mgr.smithi062.rwurtk (mgr.14164) 68 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:28:09.537 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:09.533+0000 7f74a2756700 1 -- 172.21.15.81:0/2914869552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f749c060280 msgr2=0x7f749c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:09.537 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:09.533+0000 7f74a2756700 1 --2- 172.21.15.81:0/2914869552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f749c060280 0x7f749c0f90f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f748c009fb0 tx=0x7f748c009b30).stop 2023-11-15T06:28:09.538 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:09.533+0000 7f74a2756700 1 -- 172.21.15.81:0/2914869552 shutdown_connections 2023-11-15T06:28:09.538 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:09.533+0000 7f74a2756700 1 --2- 172.21.15.81:0/2914869552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f749c060280 0x7f749c0f90f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:09.539 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:09.533+0000 7f74a2756700 1 -- 172.21.15.81:0/2914869552 >> 172.21.15.81:0/2914869552 conn(0x7f749c0f0b70 msgr2=0x7f749c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:28:09.539 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:09.533+0000 7f74a2756700 1 -- 172.21.15.81:0/2914869552 shutdown_connections 2023-11-15T06:28:09.539 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:09.533+0000 7f74a2756700 1 -- 172.21.15.81:0/2914869552 wait complete. 2023-11-15T06:28:09.539 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:09.534+0000 7f74a2756700 1 Processor -- start 2023-11-15T06:28:09.539 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:09.534+0000 7f74a2756700 1 -- start start 2023-11-15T06:28:09.541 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:09.534+0000 7f74a2756700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f749c060280 0x7f749c0f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:28:09.541 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:09.534+0000 7f74a2756700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f749c0f4ff0 con 0x7f749c060280 2023-11-15T06:28:09.542 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:09.534+0000 7f749bfff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f749c060280 0x7f749c0f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:28:09.542 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:09.534+0000 7f749bfff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f749c060280 0x7f749c0f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:52114/0 (socket says 172.21.15.81:52114) 2023-11-15T06:28:09.542 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:09.534+0000 7f749bfff700 1 -- 172.21.15.81:0/895452695 learned_addr learned my addr 172.21.15.81:0/895452695 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:28:09.543 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:09.534+0000 7f749bfff700 1 -- 172.21.15.81:0/895452695 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f748c005040 con 0x7f749c060280 2023-11-15T06:28:09.543 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:09.535+0000 7f749bfff700 1 --2- 172.21.15.81:0/895452695 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f749c060280 0x7f749c0f6b70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f748c013510 tx=0x7f748c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:28:09.543 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:09.535+0000 7f7498ff9700 1 -- 172.21.15.81:0/895452695 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f748c009320 con 0x7f749c060280 2023-11-15T06:28:09.545 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:09.535+0000 7f74a2756700 1 -- 172.21.15.81:0/895452695 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f749c0f5190 con 0x7f749c060280 2023-11-15T06:28:09.545 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:09.535+0000 7f7498ff9700 1 -- 172.21.15.81:0/895452695 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f748c013f00 con 0x7f749c060280 2023-11-15T06:28:09.545 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:09.535+0000 7f7498ff9700 1 -- 172.21.15.81:0/895452695 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f748c014dc0 con 0x7f749c060280 2023-11-15T06:28:09.545 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:09.535+0000 7f74a2756700 1 -- 172.21.15.81:0/895452695 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f749c0f5620 con 0x7f749c060280 2023-11-15T06:28:09.545 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:09.536+0000 7f7498ff9700 1 -- 172.21.15.81:0/895452695 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f748c014420 con 0x7f749c060280 2023-11-15T06:28:09.546 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:09.536+0000 7f74a2756700 1 -- 172.21.15.81:0/895452695 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f749c059100 con 0x7f749c060280 2023-11-15T06:28:09.546 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:09.536+0000 7f7498ff9700 1 --2- 172.21.15.81:0/895452695 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f748402ff60 0x7f7484032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:28:09.546 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:09.536+0000 7f7498ff9700 1 -- 172.21.15.81:0/895452695 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f748c041110 con 0x7f749c060280 2023-11-15T06:28:09.546 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:09.536+0000 7f749b7fe700 1 --2- 172.21.15.81:0/895452695 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f748402ff60 0x7f7484032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:28:09.546 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:09.536+0000 7f749b7fe700 1 --2- 172.21.15.81:0/895452695 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f748402ff60 0x7f7484032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f74900096a0 tx=0x7f7490006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:28:09.547 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:09.541+0000 7f7498ff9700 1 -- 172.21.15.81:0/895452695 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f748c0095f0 con 0x7f749c060280 2023-11-15T06:28:09.748 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:09.746+0000 7f74a2756700 1 -- 172.21.15.81:0/895452695 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f749c0fbf50 con 0x7f749c060280 2023-11-15T06:28:09.750 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:09.747+0000 7f7498ff9700 1 -- 172.21.15.81:0/895452695 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f748c01cb10 con 0x7f749c060280 2023-11-15T06:28:09.751 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:28:09.751 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:28:09.752 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:09.748+0000 7f74a2756700 1 -- 172.21.15.81:0/895452695 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f748402ff60 msgr2=0x7f7484032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:09.752 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:09.748+0000 7f74a2756700 1 --2- 172.21.15.81:0/895452695 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f748402ff60 0x7f7484032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f74900096a0 tx=0x7f7490006b40).stop 2023-11-15T06:28:09.752 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:09.748+0000 7f74a2756700 1 -- 172.21.15.81:0/895452695 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f749c060280 msgr2=0x7f749c0f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:09.752 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:09.748+0000 7f74a2756700 1 --2- 172.21.15.81:0/895452695 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f749c060280 0x7f749c0f6b70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f748c013510 tx=0x7f748c00b040).stop 2023-11-15T06:28:09.753 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:09.748+0000 7f74a2756700 1 -- 172.21.15.81:0/895452695 shutdown_connections 2023-11-15T06:28:09.753 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:09.748+0000 7f74a2756700 1 --2- 172.21.15.81:0/895452695 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f748402ff60 0x7f7484032420 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:09.753 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:09.748+0000 7f74a2756700 1 --2- 172.21.15.81:0/895452695 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f749c060280 0x7f749c0f6b70 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:09.753 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:09.748+0000 7f74a2756700 1 -- 172.21.15.81:0/895452695 >> 172.21.15.81:0/895452695 conn(0x7f749c0f0b70 msgr2=0x7f749c0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:28:09.753 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:09.748+0000 7f74a2756700 1 -- 172.21.15.81:0/895452695 shutdown_connections 2023-11-15T06:28:09.753 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:09.748+0000 7f74a2756700 1 -- 172.21.15.81:0/895452695 wait complete. 2023-11-15T06:28:09.754 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:28:10.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:09 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:28:09.748259+0000 mon.smithi062 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.81:0/895452695' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:28:11.143 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:28:11.143 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:28:11.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:10 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:28:09.798836+0000 mgr.smithi062.rwurtk (mgr.14164) 69 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:28:12.299 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:12.296+0000 7f191077a700 1 -- 172.21.15.81:0/942986415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19080f6510 msgr2=0x7f19080f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:12.300 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:12.296+0000 7f191077a700 1 --2- 172.21.15.81:0/942986415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19080f6510 0x7f19080f6930 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f1904009fb0 tx=0x7f1904009b30).stop 2023-11-15T06:28:12.300 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:12.296+0000 7f191077a700 1 -- 172.21.15.81:0/942986415 shutdown_connections 2023-11-15T06:28:12.300 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:12.296+0000 7f191077a700 1 --2- 172.21.15.81:0/942986415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19080f6510 0x7f19080f6930 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:12.300 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:12.296+0000 7f191077a700 1 -- 172.21.15.81:0/942986415 >> 172.21.15.81:0/942986415 conn(0x7f19080f1c70 msgr2=0x7f19080f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:28:12.301 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:12.296+0000 7f191077a700 1 -- 172.21.15.81:0/942986415 shutdown_connections 2023-11-15T06:28:12.301 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:12.296+0000 7f191077a700 1 -- 172.21.15.81:0/942986415 wait complete. 2023-11-15T06:28:12.301 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:12.296+0000 7f191077a700 1 Processor -- start 2023-11-15T06:28:12.301 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:12.296+0000 7f191077a700 1 -- start start 2023-11-15T06:28:12.301 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:12.296+0000 7f191077a700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19080f6510 0x7f190806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:28:12.302 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:12.296+0000 7f191077a700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1908069e20 con 0x7f19080f6510 2023-11-15T06:28:12.302 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:12.297+0000 7f190e516700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19080f6510 0x7f190806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:28:12.302 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:12.297+0000 7f190e516700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19080f6510 0x7f190806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:51370/0 (socket says 172.21.15.81:51370) 2023-11-15T06:28:12.302 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:12.297+0000 7f190e516700 1 -- 172.21.15.81:0/2953994957 learned_addr learned my addr 172.21.15.81:0/2953994957 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:28:12.302 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:12.297+0000 7f190e516700 1 -- 172.21.15.81:0/2953994957 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1904005040 con 0x7f19080f6510 2023-11-15T06:28:12.303 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:12.297+0000 7f190e516700 1 --2- 172.21.15.81:0/2953994957 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19080f6510 0x7f190806b9a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f1904013510 tx=0x7f190400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:28:12.304 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:12.297+0000 7f18feffd700 1 -- 172.21.15.81:0/2953994957 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1904009320 con 0x7f19080f6510 2023-11-15T06:28:12.304 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:12.297+0000 7f191077a700 1 -- 172.21.15.81:0/2953994957 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1908069fc0 con 0x7f19080f6510 2023-11-15T06:28:12.304 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:12.297+0000 7f191077a700 1 -- 172.21.15.81:0/2953994957 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f190806a450 con 0x7f19080f6510 2023-11-15T06:28:12.305 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:12.297+0000 7f18feffd700 1 -- 172.21.15.81:0/2953994957 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1904013f00 con 0x7f19080f6510 2023-11-15T06:28:12.305 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:12.297+0000 7f18feffd700 1 -- 172.21.15.81:0/2953994957 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1904014d60 con 0x7f19080f6510 2023-11-15T06:28:12.305 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:12.298+0000 7f18feffd700 1 -- 172.21.15.81:0/2953994957 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1904014420 con 0x7f19080f6510 2023-11-15T06:28:12.305 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:12.298+0000 7f191077a700 1 -- 172.21.15.81:0/2953994957 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1908059100 con 0x7f19080f6510 2023-11-15T06:28:12.305 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:12.298+0000 7f18feffd700 1 --2- 172.21.15.81:0/2953994957 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f18f402ffb0 0x7f18f4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:28:12.306 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:12.298+0000 7f18feffd700 1 -- 172.21.15.81:0/2953994957 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1904041350 con 0x7f19080f6510 2023-11-15T06:28:12.306 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:12.298+0000 7f190dd15700 1 --2- 172.21.15.81:0/2953994957 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f18f402ffb0 0x7f18f4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:28:12.306 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:12.299+0000 7f190dd15700 1 --2- 172.21.15.81:0/2953994957 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f18f402ffb0 0x7f18f4032470 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f18f80096a0 tx=0x7f18f8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:28:12.306 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:12.303+0000 7f18feffd700 1 -- 172.21.15.81:0/2953994957 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1904012b20 con 0x7f19080f6510 2023-11-15T06:28:12.515 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:12.512+0000 7f191077a700 1 -- 172.21.15.81:0/2953994957 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f19080fbf60 con 0x7f19080f6510 2023-11-15T06:28:12.515 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:12.513+0000 7f18feffd700 1 -- 172.21.15.81:0/2953994957 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1904012b20 con 0x7f19080f6510 2023-11-15T06:28:12.516 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:28:12.516 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:28:12.517 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:12.514+0000 7f191077a700 1 -- 172.21.15.81:0/2953994957 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f18f402ffb0 msgr2=0x7f18f4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:12.517 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:12.514+0000 7f191077a700 1 --2- 172.21.15.81:0/2953994957 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f18f402ffb0 0x7f18f4032470 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f18f80096a0 tx=0x7f18f8006b40).stop 2023-11-15T06:28:12.517 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:12.514+0000 7f191077a700 1 -- 172.21.15.81:0/2953994957 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19080f6510 msgr2=0x7f190806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:12.518 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:12.514+0000 7f191077a700 1 --2- 172.21.15.81:0/2953994957 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19080f6510 0x7f190806b9a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f1904013510 tx=0x7f190400b040).stop 2023-11-15T06:28:12.518 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:12.514+0000 7f191077a700 1 -- 172.21.15.81:0/2953994957 shutdown_connections 2023-11-15T06:28:12.518 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:12.514+0000 7f191077a700 1 --2- 172.21.15.81:0/2953994957 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f18f402ffb0 0x7f18f4032470 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:12.518 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:12.514+0000 7f191077a700 1 --2- 172.21.15.81:0/2953994957 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19080f6510 0x7f190806b9a0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:12.518 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:12.514+0000 7f191077a700 1 -- 172.21.15.81:0/2953994957 >> 172.21.15.81:0/2953994957 conn(0x7f19080f1c70 msgr2=0x7f19080f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:28:12.519 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:12.514+0000 7f191077a700 1 -- 172.21.15.81:0/2953994957 shutdown_connections 2023-11-15T06:28:12.519 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:12.514+0000 7f191077a700 1 -- 172.21.15.81:0/2953994957 wait complete. 2023-11-15T06:28:12.519 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:28:13.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:12 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:28:11.799207+0000 mgr.smithi062.rwurtk (mgr.14164) 70 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:28:13.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:12 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:28:12.514486+0000 mon.smithi062 (mon.0) 159 : audit [DBG] from='client.? 172.21.15.81:0/2953994957' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:28:13.910 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:28:13.910 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:28:15.125 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:15.123+0000 7f4d4abae700 1 -- 172.21.15.81:0/3703227256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d440f6510 msgr2=0x7f4d440f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:15.125 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:15.123+0000 7f4d4abae700 1 --2- 172.21.15.81:0/3703227256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d440f6510 0x7f4d440f6930 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f4d38009fb0 tx=0x7f4d38009b30).stop 2023-11-15T06:28:15.126 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:15.123+0000 7f4d4abae700 1 -- 172.21.15.81:0/3703227256 shutdown_connections 2023-11-15T06:28:15.126 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:15.123+0000 7f4d4abae700 1 --2- 172.21.15.81:0/3703227256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d440f6510 0x7f4d440f6930 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:15.126 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:15.123+0000 7f4d4abae700 1 -- 172.21.15.81:0/3703227256 >> 172.21.15.81:0/3703227256 conn(0x7f4d440f1c70 msgr2=0x7f4d440f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:28:15.126 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:15.123+0000 7f4d4abae700 1 -- 172.21.15.81:0/3703227256 shutdown_connections 2023-11-15T06:28:15.127 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:15.123+0000 7f4d4abae700 1 -- 172.21.15.81:0/3703227256 wait complete. 2023-11-15T06:28:15.127 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:15.124+0000 7f4d4abae700 1 Processor -- start 2023-11-15T06:28:15.127 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:15.124+0000 7f4d4abae700 1 -- start start 2023-11-15T06:28:15.129 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:15.124+0000 7f4d4abae700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d440f6510 0x7f4d44106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:28:15.129 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:15.124+0000 7f4d4abae700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d44106c20 con 0x7f4d440f6510 2023-11-15T06:28:15.129 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:15.124+0000 7f4d4894a700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d440f6510 0x7f4d44106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:28:15.129 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:15.124+0000 7f4d4894a700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d440f6510 0x7f4d44106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:51386/0 (socket says 172.21.15.81:51386) 2023-11-15T06:28:15.129 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:15.124+0000 7f4d4894a700 1 -- 172.21.15.81:0/3430023092 learned_addr learned my addr 172.21.15.81:0/3430023092 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:28:15.130 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:15.125+0000 7f4d4894a700 1 -- 172.21.15.81:0/3430023092 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d38005040 con 0x7f4d440f6510 2023-11-15T06:28:15.132 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:15.125+0000 7f4d4894a700 1 --2- 172.21.15.81:0/3430023092 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d440f6510 0x7f4d44106730 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f4d38013510 tx=0x7f4d3800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:28:15.132 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:15.125+0000 7f4d417fa700 1 -- 172.21.15.81:0/3430023092 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d38009320 con 0x7f4d440f6510 2023-11-15T06:28:15.132 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:15.125+0000 7f4d4abae700 1 -- 172.21.15.81:0/3430023092 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d44106e20 con 0x7f4d440f6510 2023-11-15T06:28:15.133 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:15.125+0000 7f4d417fa700 1 -- 172.21.15.81:0/3430023092 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4d38013f00 con 0x7f4d440f6510 2023-11-15T06:28:15.133 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:15.125+0000 7f4d417fa700 1 -- 172.21.15.81:0/3430023092 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d38014d90 con 0x7f4d440f6510 2023-11-15T06:28:15.134 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:15.125+0000 7f4d4abae700 1 -- 172.21.15.81:0/3430023092 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d441072b0 con 0x7f4d440f6510 2023-11-15T06:28:15.134 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:15.126+0000 7f4d417fa700 1 -- 172.21.15.81:0/3430023092 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4d380095f0 con 0x7f4d440f6510 2023-11-15T06:28:15.135 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:15.126+0000 7f4d4abae700 1 -- 172.21.15.81:0/3430023092 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d44045ad0 con 0x7f4d440f6510 2023-11-15T06:28:15.135 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:15.126+0000 7f4d417fa700 1 --2- 172.21.15.81:0/3430023092 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f4d3402ff60 0x7f4d34032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:28:15.135 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:15.126+0000 7f4d417fa700 1 -- 172.21.15.81:0/3430023092 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4d38040fc0 con 0x7f4d440f6510 2023-11-15T06:28:15.135 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:15.126+0000 7f4d43fff700 1 --2- 172.21.15.81:0/3430023092 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f4d3402ff60 0x7f4d34032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:28:15.136 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:15.127+0000 7f4d43fff700 1 --2- 172.21.15.81:0/3430023092 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f4d3402ff60 0x7f4d34032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f4d300096a0 tx=0x7f4d30006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:28:15.137 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:15.131+0000 7f4d417fa700 1 -- 172.21.15.81:0/3430023092 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4d38014420 con 0x7f4d440f6510 2023-11-15T06:28:15.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:14 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:28:13.799621+0000 mgr.smithi062.rwurtk (mgr.14164) 71 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:28:15.337 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:15.335+0000 7f4d4abae700 1 -- 172.21.15.81:0/3430023092 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4d440fbf60 con 0x7f4d440f6510 2023-11-15T06:28:15.337 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:15.335+0000 7f4d417fa700 1 -- 172.21.15.81:0/3430023092 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4d38010230 con 0x7f4d440f6510 2023-11-15T06:28:15.338 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:28:15.338 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:28:15.339 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:15.337+0000 7f4d4abae700 1 -- 172.21.15.81:0/3430023092 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f4d3402ff60 msgr2=0x7f4d34032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:15.339 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:15.337+0000 7f4d4abae700 1 --2- 172.21.15.81:0/3430023092 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f4d3402ff60 0x7f4d34032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f4d300096a0 tx=0x7f4d30006b40).stop 2023-11-15T06:28:15.339 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:15.337+0000 7f4d4abae700 1 -- 172.21.15.81:0/3430023092 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d440f6510 msgr2=0x7f4d44106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:15.339 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:15.337+0000 7f4d4abae700 1 --2- 172.21.15.81:0/3430023092 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d440f6510 0x7f4d44106730 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f4d38013510 tx=0x7f4d3800b040).stop 2023-11-15T06:28:15.339 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:15.337+0000 7f4d4abae700 1 -- 172.21.15.81:0/3430023092 shutdown_connections 2023-11-15T06:28:15.340 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:15.337+0000 7f4d4abae700 1 --2- 172.21.15.81:0/3430023092 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f4d3402ff60 0x7f4d34032420 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:15.340 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:15.337+0000 7f4d4abae700 1 --2- 172.21.15.81:0/3430023092 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d440f6510 0x7f4d44106730 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:15.340 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:15.337+0000 7f4d4abae700 1 -- 172.21.15.81:0/3430023092 >> 172.21.15.81:0/3430023092 conn(0x7f4d440f1c70 msgr2=0x7f4d440f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:28:15.340 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:15.337+0000 7f4d4abae700 1 -- 172.21.15.81:0/3430023092 shutdown_connections 2023-11-15T06:28:15.340 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:15.337+0000 7f4d4abae700 1 -- 172.21.15.81:0/3430023092 wait complete. 2023-11-15T06:28:15.342 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:28:16.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:15 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:28:15.336242+0000 mon.smithi062 (mon.0) 160 : audit [DBG] from='client.? 172.21.15.81:0/3430023092' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:28:16.776 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:28:16.777 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:28:17.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:16 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:28:15.799997+0000 mgr.smithi062.rwurtk (mgr.14164) 72 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:28:18.003 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:18.000+0000 7f63477f2700 1 -- 172.21.15.81:0/1122700479 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63400f6510 msgr2=0x7f63400f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:18.003 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:18.000+0000 7f63477f2700 1 --2- 172.21.15.81:0/1122700479 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63400f6510 0x7f63400f6930 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f6334009fb0 tx=0x7f6334009b30).stop 2023-11-15T06:28:18.004 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:18.000+0000 7f63477f2700 1 -- 172.21.15.81:0/1122700479 shutdown_connections 2023-11-15T06:28:18.004 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:18.000+0000 7f63477f2700 1 --2- 172.21.15.81:0/1122700479 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63400f6510 0x7f63400f6930 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:18.004 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:18.000+0000 7f63477f2700 1 -- 172.21.15.81:0/1122700479 >> 172.21.15.81:0/1122700479 conn(0x7f63400f1c70 msgr2=0x7f63400f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:28:18.004 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:18.000+0000 7f63477f2700 1 -- 172.21.15.81:0/1122700479 shutdown_connections 2023-11-15T06:28:18.004 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:18.000+0000 7f63477f2700 1 -- 172.21.15.81:0/1122700479 wait complete. 2023-11-15T06:28:18.005 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:18.000+0000 7f63477f2700 1 Processor -- start 2023-11-15T06:28:18.005 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:18.000+0000 7f63477f2700 1 -- start start 2023-11-15T06:28:18.005 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:18.000+0000 7f63477f2700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63400f6510 0x7f6340106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:28:18.005 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:18.000+0000 7f63477f2700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6340106c20 con 0x7f63400f6510 2023-11-15T06:28:18.006 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:18.000+0000 7f634558e700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63400f6510 0x7f6340106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:28:18.006 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:18.000+0000 7f634558e700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63400f6510 0x7f6340106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:51404/0 (socket says 172.21.15.81:51404) 2023-11-15T06:28:18.006 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:18.000+0000 7f634558e700 1 -- 172.21.15.81:0/2926730811 learned_addr learned my addr 172.21.15.81:0/2926730811 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:28:18.006 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:18.001+0000 7f634558e700 1 -- 172.21.15.81:0/2926730811 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6334005040 con 0x7f63400f6510 2023-11-15T06:28:18.006 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:18.001+0000 7f634558e700 1 --2- 172.21.15.81:0/2926730811 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63400f6510 0x7f6340106730 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f6334013510 tx=0x7f633400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:28:18.007 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:18.001+0000 7f6331ffb700 1 -- 172.21.15.81:0/2926730811 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6334009320 con 0x7f63400f6510 2023-11-15T06:28:18.007 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:18.001+0000 7f63477f2700 1 -- 172.21.15.81:0/2926730811 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6340106e20 con 0x7f63400f6510 2023-11-15T06:28:18.007 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:18.001+0000 7f6331ffb700 1 -- 172.21.15.81:0/2926730811 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6334013f00 con 0x7f63400f6510 2023-11-15T06:28:18.007 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:18.001+0000 7f6331ffb700 1 -- 172.21.15.81:0/2926730811 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6334014d90 con 0x7f63400f6510 2023-11-15T06:28:18.007 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:18.001+0000 7f63477f2700 1 -- 172.21.15.81:0/2926730811 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63401072b0 con 0x7f63400f6510 2023-11-15T06:28:18.008 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:18.002+0000 7f6331ffb700 1 -- 172.21.15.81:0/2926730811 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f63340263a0 con 0x7f63400f6510 2023-11-15T06:28:18.008 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:18.002+0000 7f6331ffb700 1 --2- 172.21.15.81:0/2926730811 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f632c02ffb0 0x7f632c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:28:18.009 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:18.002+0000 7f6331ffb700 1 -- 172.21.15.81:0/2926730811 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f63340410f0 con 0x7f63400f6510 2023-11-15T06:28:18.009 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:18.002+0000 7f63477f2700 1 -- 172.21.15.81:0/2926730811 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6340045ad0 con 0x7f63400f6510 2023-11-15T06:28:18.009 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:18.002+0000 7f6344d8d700 1 --2- 172.21.15.81:0/2926730811 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f632c02ffb0 0x7f632c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:28:18.009 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:18.003+0000 7f6344d8d700 1 --2- 172.21.15.81:0/2926730811 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f632c02ffb0 0x7f632c032470 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f633c0096a0 tx=0x7f633c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:28:18.010 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:18.007+0000 7f6331ffb700 1 -- 172.21.15.81:0/2926730811 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6334012920 con 0x7f63400f6510 2023-11-15T06:28:18.213 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:18.211+0000 7f63477f2700 1 -- 172.21.15.81:0/2926730811 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f63400fc050 con 0x7f63400f6510 2023-11-15T06:28:18.213 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:18.212+0000 7f6331ffb700 1 -- 172.21.15.81:0/2926730811 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f633402b020 con 0x7f63400f6510 2023-11-15T06:28:18.214 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:28:18.214 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:28:18.215 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:18.213+0000 7f63477f2700 1 -- 172.21.15.81:0/2926730811 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f632c02ffb0 msgr2=0x7f632c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:18.215 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:18.213+0000 7f63477f2700 1 --2- 172.21.15.81:0/2926730811 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f632c02ffb0 0x7f632c032470 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f633c0096a0 tx=0x7f633c006b40).stop 2023-11-15T06:28:18.216 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:18.213+0000 7f63477f2700 1 -- 172.21.15.81:0/2926730811 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63400f6510 msgr2=0x7f6340106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:18.216 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:18.213+0000 7f63477f2700 1 --2- 172.21.15.81:0/2926730811 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63400f6510 0x7f6340106730 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f6334013510 tx=0x7f633400b040).stop 2023-11-15T06:28:18.216 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:18.213+0000 7f63477f2700 1 -- 172.21.15.81:0/2926730811 shutdown_connections 2023-11-15T06:28:18.216 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:18.213+0000 7f63477f2700 1 --2- 172.21.15.81:0/2926730811 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f632c02ffb0 0x7f632c032470 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:18.216 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:18.213+0000 7f63477f2700 1 --2- 172.21.15.81:0/2926730811 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63400f6510 0x7f6340106730 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:18.217 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:18.213+0000 7f63477f2700 1 -- 172.21.15.81:0/2926730811 >> 172.21.15.81:0/2926730811 conn(0x7f63400f1c70 msgr2=0x7f63400f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:28:18.217 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:18.213+0000 7f63477f2700 1 -- 172.21.15.81:0/2926730811 shutdown_connections 2023-11-15T06:28:18.217 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:18.213+0000 7f63477f2700 1 -- 172.21.15.81:0/2926730811 wait complete. 2023-11-15T06:28:18.218 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:28:18.919 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:28:17.800317+0000 mgr.smithi062.rwurtk (mgr.14164) 73 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:28:18.919 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:28:18.212573+0000 mon.smithi062 (mon.0) 161 : audit [DBG] from='client.? 172.21.15.81:0/2926730811' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:28:19.608 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:28:19.609 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:28:20.862 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:20.857+0000 7f8a745a1700 1 -- 172.21.15.81:0/3465595905 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a6c0f5410 msgr2=0x7f8a6c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:20.862 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:20.857+0000 7f8a745a1700 1 --2- 172.21.15.81:0/3465595905 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a6c0f5410 0x7f8a6c0f5830 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f8a5c009fb0 tx=0x7f8a5c009b30).stop 2023-11-15T06:28:20.862 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:20.857+0000 7f8a745a1700 1 -- 172.21.15.81:0/3465595905 shutdown_connections 2023-11-15T06:28:20.862 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:20.857+0000 7f8a745a1700 1 --2- 172.21.15.81:0/3465595905 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a6c0f5410 0x7f8a6c0f5830 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:20.863 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:20.857+0000 7f8a745a1700 1 -- 172.21.15.81:0/3465595905 >> 172.21.15.81:0/3465595905 conn(0x7f8a6c0f0b70 msgr2=0x7f8a6c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:28:20.863 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:20.857+0000 7f8a745a1700 1 -- 172.21.15.81:0/3465595905 shutdown_connections 2023-11-15T06:28:20.863 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:20.857+0000 7f8a745a1700 1 -- 172.21.15.81:0/3465595905 wait complete. 2023-11-15T06:28:20.863 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:20.857+0000 7f8a745a1700 1 Processor -- start 2023-11-15T06:28:20.863 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:20.857+0000 7f8a745a1700 1 -- start start 2023-11-15T06:28:20.864 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:20.858+0000 7f8a745a1700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a6c0f5410 0x7f8a6c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:28:20.864 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:20.858+0000 7f8a745a1700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a6c106c20 con 0x7f8a6c0f5410 2023-11-15T06:28:20.864 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:20.858+0000 7f8a7233d700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a6c0f5410 0x7f8a6c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:28:20.864 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:20.858+0000 7f8a7233d700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a6c0f5410 0x7f8a6c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:51418/0 (socket says 172.21.15.81:51418) 2023-11-15T06:28:20.864 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:20.858+0000 7f8a7233d700 1 -- 172.21.15.81:0/4046078497 learned_addr learned my addr 172.21.15.81:0/4046078497 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:28:20.865 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:20.858+0000 7f8a7233d700 1 -- 172.21.15.81:0/4046078497 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a5c005040 con 0x7f8a6c0f5410 2023-11-15T06:28:20.865 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:20.858+0000 7f8a7233d700 1 --2- 172.21.15.81:0/4046078497 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a6c0f5410 0x7f8a6c106730 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f8a5c013510 tx=0x7f8a5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:28:20.865 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:20.858+0000 7f8a62ffd700 1 -- 172.21.15.81:0/4046078497 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a5c009320 con 0x7f8a6c0f5410 2023-11-15T06:28:20.865 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:20.858+0000 7f8a745a1700 1 -- 172.21.15.81:0/4046078497 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a6c106e20 con 0x7f8a6c0f5410 2023-11-15T06:28:20.865 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:20.858+0000 7f8a62ffd700 1 -- 172.21.15.81:0/4046078497 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8a5c013f00 con 0x7f8a6c0f5410 2023-11-15T06:28:20.866 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:20.858+0000 7f8a62ffd700 1 -- 172.21.15.81:0/4046078497 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a5c014d90 con 0x7f8a6c0f5410 2023-11-15T06:28:20.866 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:20.858+0000 7f8a745a1700 1 -- 172.21.15.81:0/4046078497 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a6c1072b0 con 0x7f8a6c0f5410 2023-11-15T06:28:20.868 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:20.859+0000 7f8a62ffd700 1 -- 172.21.15.81:0/4046078497 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8a5c014420 con 0x7f8a6c0f5410 2023-11-15T06:28:20.868 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:20.859+0000 7f8a62ffd700 1 --2- 172.21.15.81:0/4046078497 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f8a5802ff60 0x7f8a58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:28:20.868 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:20.859+0000 7f8a745a1700 1 -- 172.21.15.81:0/4046078497 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a6c045ad0 con 0x7f8a6c0f5410 2023-11-15T06:28:20.868 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:20.859+0000 7f8a62ffd700 1 -- 172.21.15.81:0/4046078497 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8a5c041070 con 0x7f8a6c0f5410 2023-11-15T06:28:20.869 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:20.860+0000 7f8a71b3c700 1 --2- 172.21.15.81:0/4046078497 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f8a5802ff60 0x7f8a58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:28:20.869 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:20.860+0000 7f8a71b3c700 1 --2- 172.21.15.81:0/4046078497 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f8a5802ff60 0x7f8a58032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f8a680096a0 tx=0x7f8a68006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:28:20.870 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:20.864+0000 7f8a62ffd700 1 -- 172.21.15.81:0/4046078497 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8a5c0095f0 con 0x7f8a6c0f5410 2023-11-15T06:28:21.079 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:21.076+0000 7f8a745a1700 1 -- 172.21.15.81:0/4046078497 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8a6c0fc080 con 0x7f8a6c0f5410 2023-11-15T06:28:21.079 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:21.077+0000 7f8a62ffd700 1 -- 172.21.15.81:0/4046078497 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8a5c01db70 con 0x7f8a6c0f5410 2023-11-15T06:28:21.079 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:28:21.080 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:28:21.081 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:21.078+0000 7f8a745a1700 1 -- 172.21.15.81:0/4046078497 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f8a5802ff60 msgr2=0x7f8a58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:21.081 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:21.078+0000 7f8a745a1700 1 --2- 172.21.15.81:0/4046078497 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f8a5802ff60 0x7f8a58032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f8a680096a0 tx=0x7f8a68006b40).stop 2023-11-15T06:28:21.081 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:21.078+0000 7f8a745a1700 1 -- 172.21.15.81:0/4046078497 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a6c0f5410 msgr2=0x7f8a6c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:21.081 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:21.078+0000 7f8a745a1700 1 --2- 172.21.15.81:0/4046078497 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a6c0f5410 0x7f8a6c106730 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f8a5c013510 tx=0x7f8a5c00b040).stop 2023-11-15T06:28:21.082 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:21.078+0000 7f8a745a1700 1 -- 172.21.15.81:0/4046078497 shutdown_connections 2023-11-15T06:28:21.082 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:21.078+0000 7f8a745a1700 1 --2- 172.21.15.81:0/4046078497 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f8a5802ff60 0x7f8a58032420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:21.082 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:21.078+0000 7f8a745a1700 1 --2- 172.21.15.81:0/4046078497 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a6c0f5410 0x7f8a6c106730 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:21.082 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:21.078+0000 7f8a745a1700 1 -- 172.21.15.81:0/4046078497 >> 172.21.15.81:0/4046078497 conn(0x7f8a6c0f0b70 msgr2=0x7f8a6c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:28:21.082 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:21.078+0000 7f8a745a1700 1 -- 172.21.15.81:0/4046078497 shutdown_connections 2023-11-15T06:28:21.083 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:21.078+0000 7f8a745a1700 1 -- 172.21.15.81:0/4046078497 wait complete. 2023-11-15T06:28:21.083 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:28:21.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:20 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:28:19.800596+0000 mgr.smithi062.rwurtk (mgr.14164) 74 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:28:22.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:21 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:28:21.077648+0000 mon.smithi062 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.81:0/4046078497' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:28:22.510 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:28:22.510 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:28:23.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:22 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:28:21.800956+0000 mgr.smithi062.rwurtk (mgr.14164) 75 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:28:23.779 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:23.777+0000 7f0081346700 1 -- 172.21.15.81:0/2362663120 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f007c0f51a0 msgr2=0x7f007c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:23.779 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:23.777+0000 7f0081346700 1 --2- 172.21.15.81:0/2362663120 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f007c0f51a0 0x7f007c0f55c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f006c009fb0 tx=0x7f006c009b30).stop 2023-11-15T06:28:23.780 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:23.777+0000 7f0081346700 1 -- 172.21.15.81:0/2362663120 shutdown_connections 2023-11-15T06:28:23.780 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:23.777+0000 7f0081346700 1 --2- 172.21.15.81:0/2362663120 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f007c0f51a0 0x7f007c0f55c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:23.780 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:23.777+0000 7f0081346700 1 -- 172.21.15.81:0/2362663120 >> 172.21.15.81:0/2362663120 conn(0x7f007c0f09a0 msgr2=0x7f007c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:28:23.780 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:23.777+0000 7f0081346700 1 -- 172.21.15.81:0/2362663120 shutdown_connections 2023-11-15T06:28:23.780 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:23.777+0000 7f0081346700 1 -- 172.21.15.81:0/2362663120 wait complete. 2023-11-15T06:28:23.781 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:23.777+0000 7f0081346700 1 Processor -- start 2023-11-15T06:28:23.781 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:23.777+0000 7f0081346700 1 -- start start 2023-11-15T06:28:23.781 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:23.777+0000 7f0081346700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f007c0f51a0 0x7f007c1091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:28:23.781 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:23.777+0000 7f0081346700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f007c060240 con 0x7f007c0f51a0 2023-11-15T06:28:23.783 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:23.778+0000 7f007bfff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f007c0f51a0 0x7f007c1091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:28:23.783 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:23.778+0000 7f007bfff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f007c0f51a0 0x7f007c1091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:34732/0 (socket says 172.21.15.81:34732) 2023-11-15T06:28:23.783 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:23.778+0000 7f007bfff700 1 -- 172.21.15.81:0/3681353715 learned_addr learned my addr 172.21.15.81:0/3681353715 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:28:23.784 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:23.778+0000 7f007bfff700 1 -- 172.21.15.81:0/3681353715 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f006c005040 con 0x7f007c0f51a0 2023-11-15T06:28:23.784 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:23.778+0000 7f007bfff700 1 --2- 172.21.15.81:0/3681353715 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f007c0f51a0 0x7f007c1091b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f006c013510 tx=0x7f006c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:28:23.784 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:23.778+0000 7f0078ff9700 1 -- 172.21.15.81:0/3681353715 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f006c009320 con 0x7f007c0f51a0 2023-11-15T06:28:23.784 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:23.778+0000 7f0081346700 1 -- 172.21.15.81:0/3681353715 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f007c060440 con 0x7f007c0f51a0 2023-11-15T06:28:23.784 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:23.778+0000 7f0078ff9700 1 -- 172.21.15.81:0/3681353715 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f006c013f00 con 0x7f007c0f51a0 2023-11-15T06:28:23.785 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:23.778+0000 7f0078ff9700 1 -- 172.21.15.81:0/3681353715 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f006c014dc0 con 0x7f007c0f51a0 2023-11-15T06:28:23.785 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:23.778+0000 7f0081346700 1 -- 172.21.15.81:0/3681353715 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f007c109860 con 0x7f007c0f51a0 2023-11-15T06:28:23.785 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:23.779+0000 7f0078ff9700 1 -- 172.21.15.81:0/3681353715 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f006c014420 con 0x7f007c0f51a0 2023-11-15T06:28:23.785 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:23.779+0000 7f0081346700 1 -- 172.21.15.81:0/3681353715 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f007c045ad0 con 0x7f007c0f51a0 2023-11-15T06:28:23.785 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:23.779+0000 7f0078ff9700 1 --2- 172.21.15.81:0/3681353715 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f006402ff60 0x7f0064032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:28:23.786 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:23.780+0000 7f0078ff9700 1 -- 172.21.15.81:0/3681353715 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f006c041020 con 0x7f007c0f51a0 2023-11-15T06:28:23.786 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:23.780+0000 7f007b7fe700 1 --2- 172.21.15.81:0/3681353715 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f006402ff60 0x7f0064032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:28:23.786 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:23.780+0000 7f007b7fe700 1 --2- 172.21.15.81:0/3681353715 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f006402ff60 0x7f0064032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f00700096a0 tx=0x7f0070006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:28:23.786 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:23.784+0000 7f0078ff9700 1 -- 172.21.15.81:0/3681353715 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f006c0095f0 con 0x7f007c0f51a0 2023-11-15T06:28:23.987 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:23.985+0000 7f0081346700 1 -- 172.21.15.81:0/3681353715 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f007c10a070 con 0x7f007c0f51a0 2023-11-15T06:28:23.987 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:23.985+0000 7f0078ff9700 1 -- 172.21.15.81:0/3681353715 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f006c01cb10 con 0x7f007c0f51a0 2023-11-15T06:28:23.988 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:28:23.988 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:28:23.989 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:23.987+0000 7f0081346700 1 -- 172.21.15.81:0/3681353715 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f006402ff60 msgr2=0x7f0064032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:23.989 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:23.987+0000 7f0081346700 1 --2- 172.21.15.81:0/3681353715 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f006402ff60 0x7f0064032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f00700096a0 tx=0x7f0070006b40).stop 2023-11-15T06:28:23.989 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:23.987+0000 7f0081346700 1 -- 172.21.15.81:0/3681353715 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f007c0f51a0 msgr2=0x7f007c1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:23.989 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:23.987+0000 7f0081346700 1 --2- 172.21.15.81:0/3681353715 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f007c0f51a0 0x7f007c1091b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f006c013510 tx=0x7f006c00b040).stop 2023-11-15T06:28:23.990 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:23.987+0000 7f0081346700 1 -- 172.21.15.81:0/3681353715 shutdown_connections 2023-11-15T06:28:23.990 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:23.987+0000 7f0081346700 1 --2- 172.21.15.81:0/3681353715 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f006402ff60 0x7f0064032420 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:23.990 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:23.987+0000 7f0081346700 1 --2- 172.21.15.81:0/3681353715 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f007c0f51a0 0x7f007c1091b0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:23.990 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:23.987+0000 7f0081346700 1 -- 172.21.15.81:0/3681353715 >> 172.21.15.81:0/3681353715 conn(0x7f007c0f09a0 msgr2=0x7f007c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:28:23.990 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:23.987+0000 7f0081346700 1 -- 172.21.15.81:0/3681353715 shutdown_connections 2023-11-15T06:28:23.991 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:23.987+0000 7f0081346700 1 -- 172.21.15.81:0/3681353715 wait complete. 2023-11-15T06:28:23.991 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:28:25.169 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:24 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:28:23.801435+0000 mgr.smithi062.rwurtk (mgr.14164) 76 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:28:25.169 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:24 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:28:23.986226+0000 mon.smithi062 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.81:0/3681353715' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:28:25.367 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:28:25.368 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:28:26.607 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:26.604+0000 7f7bf84c0700 1 -- 172.21.15.81:0/4117441440 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7bf00f6510 msgr2=0x7f7bf00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:26.608 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:26.604+0000 7f7bf84c0700 1 --2- 172.21.15.81:0/4117441440 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7bf00f6510 0x7f7bf00f6930 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f7be0009fb0 tx=0x7f7be0009b30).stop 2023-11-15T06:28:26.608 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:26.604+0000 7f7bf84c0700 1 -- 172.21.15.81:0/4117441440 shutdown_connections 2023-11-15T06:28:26.608 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:26.604+0000 7f7bf84c0700 1 --2- 172.21.15.81:0/4117441440 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7bf00f6510 0x7f7bf00f6930 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:26.608 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:26.604+0000 7f7bf84c0700 1 -- 172.21.15.81:0/4117441440 >> 172.21.15.81:0/4117441440 conn(0x7f7bf00f1c70 msgr2=0x7f7bf00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:28:26.609 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:26.604+0000 7f7bf84c0700 1 -- 172.21.15.81:0/4117441440 shutdown_connections 2023-11-15T06:28:26.609 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:26.604+0000 7f7bf84c0700 1 -- 172.21.15.81:0/4117441440 wait complete. 2023-11-15T06:28:26.609 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:26.605+0000 7f7bf84c0700 1 Processor -- start 2023-11-15T06:28:26.609 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:26.605+0000 7f7bf84c0700 1 -- start start 2023-11-15T06:28:26.609 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:26.605+0000 7f7bf84c0700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7bf00f6510 0x7f7bf0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:28:26.610 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:26.605+0000 7f7bf84c0700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf0106c20 con 0x7f7bf00f6510 2023-11-15T06:28:26.610 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:26.605+0000 7f7bf625c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7bf00f6510 0x7f7bf0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:28:26.610 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:26.605+0000 7f7bf625c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7bf00f6510 0x7f7bf0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:34744/0 (socket says 172.21.15.81:34744) 2023-11-15T06:28:26.610 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:26.605+0000 7f7bf625c700 1 -- 172.21.15.81:0/1000326333 learned_addr learned my addr 172.21.15.81:0/1000326333 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:28:26.610 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:26.605+0000 7f7bf625c700 1 -- 172.21.15.81:0/1000326333 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7be0005040 con 0x7f7bf00f6510 2023-11-15T06:28:26.611 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:26.605+0000 7f7bf625c700 1 --2- 172.21.15.81:0/1000326333 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7bf00f6510 0x7f7bf0106730 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f7be0013510 tx=0x7f7be000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:28:26.611 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:26.606+0000 7f7be6ffd700 1 -- 172.21.15.81:0/1000326333 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7be0009320 con 0x7f7bf00f6510 2023-11-15T06:28:26.611 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:26.606+0000 7f7be6ffd700 1 -- 172.21.15.81:0/1000326333 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7be0013f00 con 0x7f7bf00f6510 2023-11-15T06:28:26.611 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:26.606+0000 7f7bf84c0700 1 -- 172.21.15.81:0/1000326333 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bf0106e20 con 0x7f7bf00f6510 2023-11-15T06:28:26.611 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:26.606+0000 7f7be6ffd700 1 -- 172.21.15.81:0/1000326333 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7be0014d90 con 0x7f7bf00f6510 2023-11-15T06:28:26.611 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:26.606+0000 7f7bf84c0700 1 -- 172.21.15.81:0/1000326333 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bf01072b0 con 0x7f7bf00f6510 2023-11-15T06:28:26.612 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:26.606+0000 7f7be6ffd700 1 -- 172.21.15.81:0/1000326333 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7be00263a0 con 0x7f7bf00f6510 2023-11-15T06:28:26.613 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:26.607+0000 7f7be6ffd700 1 --2- 172.21.15.81:0/1000326333 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f7bdc02ffb0 0x7f7bdc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:28:26.613 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:26.607+0000 7f7be6ffd700 1 -- 172.21.15.81:0/1000326333 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7be00410e0 con 0x7f7bf00f6510 2023-11-15T06:28:26.613 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:26.607+0000 7f7bf84c0700 1 -- 172.21.15.81:0/1000326333 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bf0045ad0 con 0x7f7bf00f6510 2023-11-15T06:28:26.613 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:26.607+0000 7f7bf5a5b700 1 --2- 172.21.15.81:0/1000326333 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f7bdc02ffb0 0x7f7bdc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:28:26.613 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:26.607+0000 7f7bf5a5b700 1 --2- 172.21.15.81:0/1000326333 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f7bdc02ffb0 0x7f7bdc032470 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f7bec0096a0 tx=0x7f7bec006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:28:26.614 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:26.612+0000 7f7be6ffd700 1 -- 172.21.15.81:0/1000326333 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7be0012920 con 0x7f7bf00f6510 2023-11-15T06:28:26.818 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:26.817+0000 7f7bf84c0700 1 -- 172.21.15.81:0/1000326333 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7bf00fc050 con 0x7f7bf00f6510 2023-11-15T06:28:26.819 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:26.817+0000 7f7be6ffd700 1 -- 172.21.15.81:0/1000326333 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7be002b020 con 0x7f7bf00f6510 2023-11-15T06:28:26.819 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:28:26.820 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:28:26.821 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:26.819+0000 7f7bf84c0700 1 -- 172.21.15.81:0/1000326333 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f7bdc02ffb0 msgr2=0x7f7bdc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:26.821 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:26.819+0000 7f7bf84c0700 1 --2- 172.21.15.81:0/1000326333 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f7bdc02ffb0 0x7f7bdc032470 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f7bec0096a0 tx=0x7f7bec006b40).stop 2023-11-15T06:28:26.821 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:26.819+0000 7f7bf84c0700 1 -- 172.21.15.81:0/1000326333 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7bf00f6510 msgr2=0x7f7bf0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:26.821 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:26.819+0000 7f7bf84c0700 1 --2- 172.21.15.81:0/1000326333 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7bf00f6510 0x7f7bf0106730 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f7be0013510 tx=0x7f7be000b040).stop 2023-11-15T06:28:26.822 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:26.819+0000 7f7bf84c0700 1 -- 172.21.15.81:0/1000326333 shutdown_connections 2023-11-15T06:28:26.822 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:26.819+0000 7f7bf84c0700 1 --2- 172.21.15.81:0/1000326333 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f7bdc02ffb0 0x7f7bdc032470 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:26.822 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:26.819+0000 7f7bf84c0700 1 --2- 172.21.15.81:0/1000326333 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7bf00f6510 0x7f7bf0106730 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:26.823 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:26.819+0000 7f7bf84c0700 1 -- 172.21.15.81:0/1000326333 >> 172.21.15.81:0/1000326333 conn(0x7f7bf00f1c70 msgr2=0x7f7bf00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:28:26.823 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:26.819+0000 7f7bf84c0700 1 -- 172.21.15.81:0/1000326333 shutdown_connections 2023-11-15T06:28:26.823 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:26.819+0000 7f7bf84c0700 1 -- 172.21.15.81:0/1000326333 wait complete. 2023-11-15T06:28:26.825 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:28:27.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:26 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:28:25.801729+0000 mgr.smithi062.rwurtk (mgr.14164) 77 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:28:27.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:26 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:28:26.818162+0000 mon.smithi062 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.81:0/1000326333' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:28:28.206 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:28:28.206 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:28:28.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:28 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:28:27.801953+0000 mgr.smithi062.rwurtk (mgr.14164) 78 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:28:29.407 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:29.402+0000 7f9ae9a08700 1 -- 172.21.15.81:0/173334361 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ae40f5410 msgr2=0x7f9ae40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:29.408 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:29.402+0000 7f9ae9a08700 1 --2- 172.21.15.81:0/173334361 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ae40f5410 0x7f9ae40f5830 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f9ad4009fb0 tx=0x7f9ad4009b30).stop 2023-11-15T06:28:29.408 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:29.402+0000 7f9ae9a08700 1 -- 172.21.15.81:0/173334361 shutdown_connections 2023-11-15T06:28:29.408 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:29.402+0000 7f9ae9a08700 1 --2- 172.21.15.81:0/173334361 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ae40f5410 0x7f9ae40f5830 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:29.408 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:29.402+0000 7f9ae9a08700 1 -- 172.21.15.81:0/173334361 >> 172.21.15.81:0/173334361 conn(0x7f9ae40f0b70 msgr2=0x7f9ae40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:28:29.409 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:29.402+0000 7f9ae9a08700 1 -- 172.21.15.81:0/173334361 shutdown_connections 2023-11-15T06:28:29.409 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:29.402+0000 7f9ae9a08700 1 -- 172.21.15.81:0/173334361 wait complete. 2023-11-15T06:28:29.409 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:29.402+0000 7f9ae9a08700 1 Processor -- start 2023-11-15T06:28:29.409 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:29.402+0000 7f9ae9a08700 1 -- start start 2023-11-15T06:28:29.409 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:29.402+0000 7f9ae9a08700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ae40f5410 0x7f9ae41066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:28:29.410 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:29.402+0000 7f9ae9a08700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae4106bd0 con 0x7f9ae40f5410 2023-11-15T06:28:29.410 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:29.402+0000 7f9ae2ffd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ae40f5410 0x7f9ae41066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:28:29.410 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:29.402+0000 7f9ae2ffd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ae40f5410 0x7f9ae41066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:34758/0 (socket says 172.21.15.81:34758) 2023-11-15T06:28:29.410 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:29.402+0000 7f9ae2ffd700 1 -- 172.21.15.81:0/4059661733 learned_addr learned my addr 172.21.15.81:0/4059661733 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:28:29.410 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:29.403+0000 7f9ae2ffd700 1 -- 172.21.15.81:0/4059661733 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad4005040 con 0x7f9ae40f5410 2023-11-15T06:28:29.411 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:29.403+0000 7f9ae2ffd700 1 --2- 172.21.15.81:0/4059661733 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ae40f5410 0x7f9ae41066e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f9ad4013510 tx=0x7f9ad400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:28:29.411 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:29.403+0000 7f9adb7fe700 1 -- 172.21.15.81:0/4059661733 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ad4009320 con 0x7f9ae40f5410 2023-11-15T06:28:29.411 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:29.403+0000 7f9ae9a08700 1 -- 172.21.15.81:0/4059661733 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ae4106dd0 con 0x7f9ae40f5410 2023-11-15T06:28:29.411 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:29.403+0000 7f9adb7fe700 1 -- 172.21.15.81:0/4059661733 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ad4013f00 con 0x7f9ae40f5410 2023-11-15T06:28:29.411 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:29.403+0000 7f9adb7fe700 1 -- 172.21.15.81:0/4059661733 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ad4014d90 con 0x7f9ae40f5410 2023-11-15T06:28:29.412 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:29.403+0000 7f9ae9a08700 1 -- 172.21.15.81:0/4059661733 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ae4107260 con 0x7f9ae40f5410 2023-11-15T06:28:29.412 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:29.404+0000 7f9adb7fe700 1 -- 172.21.15.81:0/4059661733 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9ad40095f0 con 0x7f9ae40f5410 2023-11-15T06:28:29.412 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:29.404+0000 7f9ae9a08700 1 -- 172.21.15.81:0/4059661733 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ae4045ad0 con 0x7f9ae40f5410 2023-11-15T06:28:29.413 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:29.404+0000 7f9adb7fe700 1 --2- 172.21.15.81:0/4059661733 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f9ad002ff60 0x7f9ad0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:28:29.413 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:29.404+0000 7f9adb7fe700 1 -- 172.21.15.81:0/4059661733 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9ad4040fe0 con 0x7f9ae40f5410 2023-11-15T06:28:29.413 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:29.404+0000 7f9adbfff700 1 --2- 172.21.15.81:0/4059661733 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f9ad002ff60 0x7f9ad0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:28:29.413 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:29.405+0000 7f9adbfff700 1 --2- 172.21.15.81:0/4059661733 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f9ad002ff60 0x7f9ad0032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f9acc0096a0 tx=0x7f9acc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:28:29.414 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:29.409+0000 7f9adb7fe700 1 -- 172.21.15.81:0/4059661733 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9ad401d7f0 con 0x7f9ae40f5410 2023-11-15T06:28:29.615 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:29.613+0000 7f9ae9a08700 1 -- 172.21.15.81:0/4059661733 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9ae40fae60 con 0x7f9ae40f5410 2023-11-15T06:28:29.616 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:29.614+0000 7f9adb7fe700 1 -- 172.21.15.81:0/4059661733 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9ad4010230 con 0x7f9ae40f5410 2023-11-15T06:28:29.617 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:28:29.617 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:28:29.618 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:29.616+0000 7f9ae9a08700 1 -- 172.21.15.81:0/4059661733 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f9ad002ff60 msgr2=0x7f9ad0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:29.618 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:29.616+0000 7f9ae9a08700 1 --2- 172.21.15.81:0/4059661733 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f9ad002ff60 0x7f9ad0032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f9acc0096a0 tx=0x7f9acc006b40).stop 2023-11-15T06:28:29.618 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:29.616+0000 7f9ae9a08700 1 -- 172.21.15.81:0/4059661733 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ae40f5410 msgr2=0x7f9ae41066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:29.619 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:29.616+0000 7f9ae9a08700 1 --2- 172.21.15.81:0/4059661733 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ae40f5410 0x7f9ae41066e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f9ad4013510 tx=0x7f9ad400b040).stop 2023-11-15T06:28:29.619 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:29.616+0000 7f9ae9a08700 1 -- 172.21.15.81:0/4059661733 shutdown_connections 2023-11-15T06:28:29.619 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:29.616+0000 7f9ae9a08700 1 --2- 172.21.15.81:0/4059661733 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f9ad002ff60 0x7f9ad0032420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:29.619 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:29.616+0000 7f9ae9a08700 1 --2- 172.21.15.81:0/4059661733 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ae40f5410 0x7f9ae41066e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:29.620 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:29.616+0000 7f9ae9a08700 1 -- 172.21.15.81:0/4059661733 >> 172.21.15.81:0/4059661733 conn(0x7f9ae40f0b70 msgr2=0x7f9ae405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:28:29.620 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:29.616+0000 7f9ae9a08700 1 -- 172.21.15.81:0/4059661733 shutdown_connections 2023-11-15T06:28:29.620 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:29.616+0000 7f9ae9a08700 1 -- 172.21.15.81:0/4059661733 wait complete. 2023-11-15T06:28:29.620 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:28:29.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:29 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:28:29.615044+0000 mon.smithi062 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.81:0/4059661733' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:28:31.053 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:28:31.054 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:28:31.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:30 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:28:29.802285+0000 mgr.smithi062.rwurtk (mgr.14164) 79 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:28:32.349 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:32.346+0000 7fe6fed6c700 1 -- 172.21.15.81:0/1881967104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6f80f51a0 msgr2=0x7fe6f80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:32.349 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:32.346+0000 7fe6fed6c700 1 --2- 172.21.15.81:0/1881967104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6f80f51a0 0x7fe6f80f55c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fe6f4009fb0 tx=0x7fe6f4009b30).stop 2023-11-15T06:28:32.350 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:32.346+0000 7fe6fed6c700 1 -- 172.21.15.81:0/1881967104 shutdown_connections 2023-11-15T06:28:32.350 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:32.346+0000 7fe6fed6c700 1 --2- 172.21.15.81:0/1881967104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6f80f51a0 0x7fe6f80f55c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:32.350 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:32.346+0000 7fe6fed6c700 1 -- 172.21.15.81:0/1881967104 >> 172.21.15.81:0/1881967104 conn(0x7fe6f80f09a0 msgr2=0x7fe6f80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:28:32.350 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:32.346+0000 7fe6fed6c700 1 -- 172.21.15.81:0/1881967104 shutdown_connections 2023-11-15T06:28:32.350 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:32.346+0000 7fe6fed6c700 1 -- 172.21.15.81:0/1881967104 wait complete. 2023-11-15T06:28:32.351 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:32.346+0000 7fe6fed6c700 1 Processor -- start 2023-11-15T06:28:32.351 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:32.346+0000 7fe6fed6c700 1 -- start start 2023-11-15T06:28:32.351 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:32.346+0000 7fe6fed6c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6f80f51a0 0x7fe6f81053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:28:32.351 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:32.347+0000 7fe6fed6c700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f81058a0 con 0x7fe6f80f51a0 2023-11-15T06:28:32.351 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:32.347+0000 7fe6fdd6a700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6f80f51a0 0x7fe6f81053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:28:32.352 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:32.347+0000 7fe6fdd6a700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6f80f51a0 0x7fe6f81053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:49120/0 (socket says 172.21.15.81:49120) 2023-11-15T06:28:32.352 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:32.347+0000 7fe6fdd6a700 1 -- 172.21.15.81:0/1383610348 learned_addr learned my addr 172.21.15.81:0/1383610348 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:28:32.352 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:32.347+0000 7fe6fdd6a700 1 -- 172.21.15.81:0/1383610348 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f4005040 con 0x7fe6f80f51a0 2023-11-15T06:28:32.352 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:32.347+0000 7fe6fdd6a700 1 --2- 172.21.15.81:0/1383610348 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6f80f51a0 0x7fe6f81053b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fe6f4013510 tx=0x7fe6f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:28:32.352 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:32.347+0000 7fe6ee7fc700 1 -- 172.21.15.81:0/1383610348 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6f4009320 con 0x7fe6f80f51a0 2023-11-15T06:28:32.353 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:32.347+0000 7fe6fed6c700 1 -- 172.21.15.81:0/1383610348 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6f8105aa0 con 0x7fe6f80f51a0 2023-11-15T06:28:32.353 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:32.347+0000 7fe6ee7fc700 1 -- 172.21.15.81:0/1383610348 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe6f4013f00 con 0x7fe6f80f51a0 2023-11-15T06:28:32.353 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:32.347+0000 7fe6ee7fc700 1 -- 172.21.15.81:0/1383610348 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6f4014dc0 con 0x7fe6f80f51a0 2023-11-15T06:28:32.353 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:32.348+0000 7fe6fed6c700 1 -- 172.21.15.81:0/1383610348 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6f8105f30 con 0x7fe6f80f51a0 2023-11-15T06:28:32.354 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:32.348+0000 7fe6fed6c700 1 -- 172.21.15.81:0/1383610348 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6f8045ad0 con 0x7fe6f80f51a0 2023-11-15T06:28:32.354 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:32.348+0000 7fe6ee7fc700 1 -- 172.21.15.81:0/1383610348 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe6f40095f0 con 0x7fe6f80f51a0 2023-11-15T06:28:32.355 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:32.349+0000 7fe6ee7fc700 1 --2- 172.21.15.81:0/1383610348 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fe6e402ff60 0x7fe6e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:28:32.355 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:32.349+0000 7fe6ee7fc700 1 -- 172.21.15.81:0/1383610348 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe6f4041000 con 0x7fe6f80f51a0 2023-11-15T06:28:32.355 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:32.349+0000 7fe6fd569700 1 --2- 172.21.15.81:0/1383610348 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fe6e402ff60 0x7fe6e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:28:32.355 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:32.349+0000 7fe6fd569700 1 --2- 172.21.15.81:0/1383610348 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fe6e402ff60 0x7fe6e4032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fe6e80096a0 tx=0x7fe6e8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:28:32.356 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:32.353+0000 7fe6ee7fc700 1 -- 172.21.15.81:0/1383610348 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe6f4014420 con 0x7fe6f80f51a0 2023-11-15T06:28:32.558 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:32.556+0000 7fe6fed6c700 1 -- 172.21.15.81:0/1383610348 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe6f8106de0 con 0x7fe6f80f51a0 2023-11-15T06:28:32.559 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:32.557+0000 7fe6ee7fc700 1 -- 172.21.15.81:0/1383610348 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe6f4010230 con 0x7fe6f80f51a0 2023-11-15T06:28:32.559 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:28:32.559 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:28:32.560 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:32.558+0000 7fe6fed6c700 1 -- 172.21.15.81:0/1383610348 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fe6e402ff60 msgr2=0x7fe6e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:32.560 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:32.559+0000 7fe6fed6c700 1 --2- 172.21.15.81:0/1383610348 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fe6e402ff60 0x7fe6e4032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fe6e80096a0 tx=0x7fe6e8006b40).stop 2023-11-15T06:28:32.561 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:32.559+0000 7fe6fed6c700 1 -- 172.21.15.81:0/1383610348 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6f80f51a0 msgr2=0x7fe6f81053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:32.561 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:32.559+0000 7fe6fed6c700 1 --2- 172.21.15.81:0/1383610348 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6f80f51a0 0x7fe6f81053b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fe6f4013510 tx=0x7fe6f400b040).stop 2023-11-15T06:28:32.561 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:32.559+0000 7fe6fed6c700 1 -- 172.21.15.81:0/1383610348 shutdown_connections 2023-11-15T06:28:32.561 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:32.559+0000 7fe6fed6c700 1 --2- 172.21.15.81:0/1383610348 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fe6e402ff60 0x7fe6e4032420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:32.561 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:32.559+0000 7fe6fed6c700 1 --2- 172.21.15.81:0/1383610348 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6f80f51a0 0x7fe6f81053b0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:32.562 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:32.559+0000 7fe6fed6c700 1 -- 172.21.15.81:0/1383610348 >> 172.21.15.81:0/1383610348 conn(0x7fe6f80f09a0 msgr2=0x7fe6f80f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:28:32.562 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:32.559+0000 7fe6fed6c700 1 -- 172.21.15.81:0/1383610348 shutdown_connections 2023-11-15T06:28:32.562 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:32.559+0000 7fe6fed6c700 1 -- 172.21.15.81:0/1383610348 wait complete. 2023-11-15T06:28:32.562 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:28:33.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:32 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:28:31.802695+0000 mgr.smithi062.rwurtk (mgr.14164) 80 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:28:33.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:32 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:28:32.557733+0000 mon.smithi062 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.81:0/1383610348' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:28:33.939 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:28:33.939 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:28:35.158 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:35.155+0000 7f6871e73700 1 -- 172.21.15.81:0/3933084265 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f686c0f62a0 msgr2=0x7f686c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:35.158 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:35.155+0000 7f6871e73700 1 --2- 172.21.15.81:0/3933084265 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f686c0f62a0 0x7f686c0f66c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f6860009fb0 tx=0x7f6860009b30).stop 2023-11-15T06:28:35.158 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:35.155+0000 7f6871e73700 1 -- 172.21.15.81:0/3933084265 shutdown_connections 2023-11-15T06:28:35.159 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:35.155+0000 7f6871e73700 1 --2- 172.21.15.81:0/3933084265 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f686c0f62a0 0x7f686c0f66c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:35.159 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:35.155+0000 7f6871e73700 1 -- 172.21.15.81:0/3933084265 >> 172.21.15.81:0/3933084265 conn(0x7f686c0f1ac0 msgr2=0x7f686c0f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:28:35.159 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:35.155+0000 7f6871e73700 1 -- 172.21.15.81:0/3933084265 shutdown_connections 2023-11-15T06:28:35.159 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:35.155+0000 7f6871e73700 1 -- 172.21.15.81:0/3933084265 wait complete. 2023-11-15T06:28:35.159 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:35.155+0000 7f6871e73700 1 Processor -- start 2023-11-15T06:28:35.160 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:35.155+0000 7f6871e73700 1 -- start start 2023-11-15T06:28:35.160 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:35.156+0000 7f6871e73700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f686c0f62a0 0x7f686c1086b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:28:35.160 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:35.156+0000 7f6871e73700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f686c108ba0 con 0x7f686c0f62a0 2023-11-15T06:28:35.160 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:35.156+0000 7f6870e71700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f686c0f62a0 0x7f686c1086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:28:35.160 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:35.156+0000 7f6870e71700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f686c0f62a0 0x7f686c1086b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:49136/0 (socket says 172.21.15.81:49136) 2023-11-15T06:28:35.161 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:35.156+0000 7f6870e71700 1 -- 172.21.15.81:0/18735688 learned_addr learned my addr 172.21.15.81:0/18735688 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:28:35.161 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:35.156+0000 7f6870e71700 1 -- 172.21.15.81:0/18735688 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6860005040 con 0x7f686c0f62a0 2023-11-15T06:28:35.161 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:35.156+0000 7f6870e71700 1 --2- 172.21.15.81:0/18735688 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f686c0f62a0 0x7f686c1086b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f6860013510 tx=0x7f686000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:28:35.162 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:35.156+0000 7f68697fa700 1 -- 172.21.15.81:0/18735688 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6860009320 con 0x7f686c0f62a0 2023-11-15T06:28:35.163 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:35.157+0000 7f6871e73700 1 -- 172.21.15.81:0/18735688 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f686c108da0 con 0x7f686c0f62a0 2023-11-15T06:28:35.163 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:35.157+0000 7f68697fa700 1 -- 172.21.15.81:0/18735688 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6860013f00 con 0x7f686c0f62a0 2023-11-15T06:28:35.164 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:35.157+0000 7f68697fa700 1 -- 172.21.15.81:0/18735688 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6860014dc0 con 0x7f686c0f62a0 2023-11-15T06:28:35.164 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:35.157+0000 7f6871e73700 1 -- 172.21.15.81:0/18735688 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f686c109230 con 0x7f686c0f62a0 2023-11-15T06:28:35.164 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:35.157+0000 7f68697fa700 1 -- 172.21.15.81:0/18735688 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f686001d3d0 con 0x7f686c0f62a0 2023-11-15T06:28:35.165 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:35.157+0000 7f6871e73700 1 -- 172.21.15.81:0/18735688 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f686c045ad0 con 0x7f686c0f62a0 2023-11-15T06:28:35.165 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:35.157+0000 7f68697fa700 1 --2- 172.21.15.81:0/18735688 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f685c02ff60 0x7f685c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:28:35.165 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:35.157+0000 7f68697fa700 1 -- 172.21.15.81:0/18735688 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f68600410f0 con 0x7f686c0f62a0 2023-11-15T06:28:35.166 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:35.158+0000 7f686bfff700 1 --2- 172.21.15.81:0/18735688 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f685c02ff60 0x7f685c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:28:35.166 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:35.158+0000 7f686bfff700 1 --2- 172.21.15.81:0/18735688 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f685c02ff60 0x7f685c032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f68580096a0 tx=0x7f6858006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:28:35.166 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:35.162+0000 7f68697fa700 1 -- 172.21.15.81:0/18735688 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f68600126f0 con 0x7f686c0f62a0 2023-11-15T06:28:35.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:34 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:28:33.803085+0000 mgr.smithi062.rwurtk (mgr.14164) 81 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:28:35.368 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:35.366+0000 7f6871e73700 1 -- 172.21.15.81:0/18735688 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f686c06b1d0 con 0x7f686c0f62a0 2023-11-15T06:28:35.369 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:35.367+0000 7f68697fa700 1 -- 172.21.15.81:0/18735688 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6860012330 con 0x7f686c0f62a0 2023-11-15T06:28:35.369 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:28:35.370 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:28:35.371 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:35.369+0000 7f6871e73700 1 -- 172.21.15.81:0/18735688 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f685c02ff60 msgr2=0x7f685c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:35.371 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:35.369+0000 7f6871e73700 1 --2- 172.21.15.81:0/18735688 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f685c02ff60 0x7f685c032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f68580096a0 tx=0x7f6858006b40).stop 2023-11-15T06:28:35.371 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:35.369+0000 7f6871e73700 1 -- 172.21.15.81:0/18735688 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f686c0f62a0 msgr2=0x7f686c1086b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:35.371 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:35.369+0000 7f6871e73700 1 --2- 172.21.15.81:0/18735688 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f686c0f62a0 0x7f686c1086b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f6860013510 tx=0x7f686000b040).stop 2023-11-15T06:28:35.371 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:35.369+0000 7f6871e73700 1 -- 172.21.15.81:0/18735688 shutdown_connections 2023-11-15T06:28:35.372 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:35.369+0000 7f6871e73700 1 --2- 172.21.15.81:0/18735688 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f685c02ff60 0x7f685c032420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:35.372 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:35.369+0000 7f6871e73700 1 --2- 172.21.15.81:0/18735688 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f686c0f62a0 0x7f686c1086b0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:35.372 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:35.369+0000 7f6871e73700 1 -- 172.21.15.81:0/18735688 >> 172.21.15.81:0/18735688 conn(0x7f686c0f1ac0 msgr2=0x7f686c0f28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:28:35.372 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:35.369+0000 7f6871e73700 1 -- 172.21.15.81:0/18735688 shutdown_connections 2023-11-15T06:28:35.372 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:35.369+0000 7f6871e73700 1 -- 172.21.15.81:0/18735688 wait complete. 2023-11-15T06:28:35.373 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:28:36.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:35 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:28:35.368010+0000 mon.smithi062 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.81:0/18735688' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:28:36.785 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:28:36.786 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:28:37.169 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:36 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:28:35.803497+0000 mgr.smithi062.rwurtk (mgr.14164) 82 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:28:37.981 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:37.978+0000 7f494fe6f700 1 -- 172.21.15.81:0/1105747102 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4948060280 msgr2=0x7f49480f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:37.981 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:37.978+0000 7f494fe6f700 1 --2- 172.21.15.81:0/1105747102 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4948060280 0x7f49480f90f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f4938009fb0 tx=0x7f4938009b30).stop 2023-11-15T06:28:37.981 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:37.978+0000 7f494fe6f700 1 -- 172.21.15.81:0/1105747102 shutdown_connections 2023-11-15T06:28:37.982 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:37.978+0000 7f494fe6f700 1 --2- 172.21.15.81:0/1105747102 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4948060280 0x7f49480f90f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:37.982 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:37.978+0000 7f494fe6f700 1 -- 172.21.15.81:0/1105747102 >> 172.21.15.81:0/1105747102 conn(0x7f49480f0b70 msgr2=0x7f49480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:28:37.982 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:37.978+0000 7f494fe6f700 1 -- 172.21.15.81:0/1105747102 shutdown_connections 2023-11-15T06:28:37.982 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:37.978+0000 7f494fe6f700 1 -- 172.21.15.81:0/1105747102 wait complete. 2023-11-15T06:28:37.982 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:37.979+0000 7f494fe6f700 1 Processor -- start 2023-11-15T06:28:37.983 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:37.979+0000 7f494fe6f700 1 -- start start 2023-11-15T06:28:37.983 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:37.979+0000 7f494fe6f700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4948060280 0x7f49480f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:28:37.983 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:37.979+0000 7f494fe6f700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49480f4ff0 con 0x7f4948060280 2023-11-15T06:28:37.983 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:37.979+0000 7f494dc0b700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4948060280 0x7f49480f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:28:37.983 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:37.979+0000 7f494dc0b700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4948060280 0x7f49480f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:49152/0 (socket says 172.21.15.81:49152) 2023-11-15T06:28:37.983 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:37.979+0000 7f494dc0b700 1 -- 172.21.15.81:0/311848801 learned_addr learned my addr 172.21.15.81:0/311848801 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:28:37.984 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:37.979+0000 7f494dc0b700 1 -- 172.21.15.81:0/311848801 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4938005040 con 0x7f4948060280 2023-11-15T06:28:37.985 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:37.979+0000 7f494dc0b700 1 --2- 172.21.15.81:0/311848801 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4948060280 0x7f49480f6b70 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f4938013510 tx=0x7f493800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:28:37.985 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:37.980+0000 7f493e7fc700 1 -- 172.21.15.81:0/311848801 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4938009320 con 0x7f4948060280 2023-11-15T06:28:37.986 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:37.980+0000 7f494fe6f700 1 -- 172.21.15.81:0/311848801 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49480f5190 con 0x7f4948060280 2023-11-15T06:28:37.986 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:37.980+0000 7f493e7fc700 1 -- 172.21.15.81:0/311848801 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4938013f00 con 0x7f4948060280 2023-11-15T06:28:37.986 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:37.980+0000 7f493e7fc700 1 -- 172.21.15.81:0/311848801 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4938014dc0 con 0x7f4948060280 2023-11-15T06:28:37.986 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:37.980+0000 7f494fe6f700 1 -- 172.21.15.81:0/311848801 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49480f5620 con 0x7f4948060280 2023-11-15T06:28:37.986 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:37.980+0000 7f493e7fc700 1 -- 172.21.15.81:0/311848801 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f49380095f0 con 0x7f4948060280 2023-11-15T06:28:37.989 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:37.981+0000 7f494fe6f700 1 -- 172.21.15.81:0/311848801 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4948059100 con 0x7f4948060280 2023-11-15T06:28:37.989 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:37.981+0000 7f493e7fc700 1 --2- 172.21.15.81:0/311848801 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f493402ff60 0x7f4934032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:28:37.990 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:37.981+0000 7f493e7fc700 1 -- 172.21.15.81:0/311848801 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f49380410c0 con 0x7f4948060280 2023-11-15T06:28:37.990 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:37.981+0000 7f494d40a700 1 --2- 172.21.15.81:0/311848801 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f493402ff60 0x7f4934032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:28:37.990 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:37.981+0000 7f494d40a700 1 --2- 172.21.15.81:0/311848801 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f493402ff60 0x7f4934032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f49440096a0 tx=0x7f4944006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:28:37.991 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:37.985+0000 7f493e7fc700 1 -- 172.21.15.81:0/311848801 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4938014420 con 0x7f4948060280 2023-11-15T06:28:38.195 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:38.193+0000 7f494fe6f700 1 -- 172.21.15.81:0/311848801 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f49480fbf50 con 0x7f4948060280 2023-11-15T06:28:38.195 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:38.194+0000 7f493e7fc700 1 -- 172.21.15.81:0/311848801 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4938010230 con 0x7f4948060280 2023-11-15T06:28:38.196 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:28:38.196 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:28:38.197 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:38.195+0000 7f494fe6f700 1 -- 172.21.15.81:0/311848801 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f493402ff60 msgr2=0x7f4934032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:38.197 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:38.195+0000 7f494fe6f700 1 --2- 172.21.15.81:0/311848801 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f493402ff60 0x7f4934032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f49440096a0 tx=0x7f4944006b40).stop 2023-11-15T06:28:38.198 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:38.195+0000 7f494fe6f700 1 -- 172.21.15.81:0/311848801 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4948060280 msgr2=0x7f49480f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:38.198 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:38.195+0000 7f494fe6f700 1 --2- 172.21.15.81:0/311848801 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4948060280 0x7f49480f6b70 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f4938013510 tx=0x7f493800b040).stop 2023-11-15T06:28:38.198 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:38.195+0000 7f494fe6f700 1 -- 172.21.15.81:0/311848801 shutdown_connections 2023-11-15T06:28:38.198 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:38.195+0000 7f494fe6f700 1 --2- 172.21.15.81:0/311848801 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f493402ff60 0x7f4934032420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:38.198 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:38.195+0000 7f494fe6f700 1 --2- 172.21.15.81:0/311848801 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4948060280 0x7f49480f6b70 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:38.199 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:38.195+0000 7f494fe6f700 1 -- 172.21.15.81:0/311848801 >> 172.21.15.81:0/311848801 conn(0x7f49480f0b70 msgr2=0x7f49480f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:28:38.199 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:38.195+0000 7f494fe6f700 1 -- 172.21.15.81:0/311848801 shutdown_connections 2023-11-15T06:28:38.199 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:38.195+0000 7f494fe6f700 1 -- 172.21.15.81:0/311848801 wait complete. 2023-11-15T06:28:38.199 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:28:38.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:38 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:28:37.803836+0000 mgr.smithi062.rwurtk (mgr.14164) 83 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:28:38.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:38 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:28:38.194425+0000 mon.smithi062 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.81:0/311848801' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:28:39.571 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:28:39.571 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:28:40.825 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:40.822+0000 7f2f32cf6700 1 -- 172.21.15.81:0/1159460916 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f2c0f47c0 msgr2=0x7f2f2c0f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:40.825 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:40.822+0000 7f2f32cf6700 1 --2- 172.21.15.81:0/1159460916 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f2c0f47c0 0x7f2f2c0f4be0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f2f20009fb0 tx=0x7f2f20009b30).stop 2023-11-15T06:28:40.825 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:40.822+0000 7f2f32cf6700 1 -- 172.21.15.81:0/1159460916 shutdown_connections 2023-11-15T06:28:40.826 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:40.822+0000 7f2f32cf6700 1 --2- 172.21.15.81:0/1159460916 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f2c0f47c0 0x7f2f2c0f4be0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:40.826 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:40.822+0000 7f2f32cf6700 1 -- 172.21.15.81:0/1159460916 >> 172.21.15.81:0/1159460916 conn(0x7f2f2c0f02e0 msgr2=0x7f2f2c0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:28:40.826 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:40.822+0000 7f2f32cf6700 1 -- 172.21.15.81:0/1159460916 shutdown_connections 2023-11-15T06:28:40.826 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:40.822+0000 7f2f32cf6700 1 -- 172.21.15.81:0/1159460916 wait complete. 2023-11-15T06:28:40.827 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:40.823+0000 7f2f32cf6700 1 Processor -- start 2023-11-15T06:28:40.827 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:40.823+0000 7f2f32cf6700 1 -- start start 2023-11-15T06:28:40.827 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:40.823+0000 7f2f32cf6700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f2c0f47c0 0x7f2f2c106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:28:40.827 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:40.823+0000 7f2f32cf6700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2c106bf0 con 0x7f2f2c0f47c0 2023-11-15T06:28:40.829 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:40.823+0000 7f2f30a92700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f2c0f47c0 0x7f2f2c106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:28:40.829 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:40.823+0000 7f2f30a92700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f2c0f47c0 0x7f2f2c106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:49164/0 (socket says 172.21.15.81:49164) 2023-11-15T06:28:40.829 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:40.823+0000 7f2f30a92700 1 -- 172.21.15.81:0/1540800411 learned_addr learned my addr 172.21.15.81:0/1540800411 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:28:40.830 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:40.824+0000 7f2f30a92700 1 -- 172.21.15.81:0/1540800411 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f20005040 con 0x7f2f2c0f47c0 2023-11-15T06:28:40.830 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:40.824+0000 7f2f30a92700 1 --2- 172.21.15.81:0/1540800411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f2c0f47c0 0x7f2f2c106700 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f2f20013510 tx=0x7f2f2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:28:40.830 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:40.824+0000 7f2f297fa700 1 -- 172.21.15.81:0/1540800411 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f20009320 con 0x7f2f2c0f47c0 2023-11-15T06:28:40.831 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:40.824+0000 7f2f32cf6700 1 -- 172.21.15.81:0/1540800411 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f2c106df0 con 0x7f2f2c0f47c0 2023-11-15T06:28:40.831 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:40.824+0000 7f2f297fa700 1 -- 172.21.15.81:0/1540800411 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2f20013f00 con 0x7f2f2c0f47c0 2023-11-15T06:28:40.831 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:40.824+0000 7f2f297fa700 1 -- 172.21.15.81:0/1540800411 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f20014d90 con 0x7f2f2c0f47c0 2023-11-15T06:28:40.831 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:40.824+0000 7f2f32cf6700 1 -- 172.21.15.81:0/1540800411 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f2c107280 con 0x7f2f2c0f47c0 2023-11-15T06:28:40.832 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:40.825+0000 7f2f297fa700 1 -- 172.21.15.81:0/1540800411 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2f20014420 con 0x7f2f2c0f47c0 2023-11-15T06:28:40.832 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:40.825+0000 7f2f32cf6700 1 -- 172.21.15.81:0/1540800411 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f2c045ad0 con 0x7f2f2c0f47c0 2023-11-15T06:28:40.832 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:40.825+0000 7f2f297fa700 1 --2- 172.21.15.81:0/1540800411 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f2f1c02ff60 0x7f2f1c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:28:40.832 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:40.825+0000 7f2f297fa700 1 -- 172.21.15.81:0/1540800411 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2f20041000 con 0x7f2f2c0f47c0 2023-11-15T06:28:40.833 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:40.825+0000 7f2f2bfff700 1 --2- 172.21.15.81:0/1540800411 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f2f1c02ff60 0x7f2f1c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:28:40.833 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:40.826+0000 7f2f2bfff700 1 --2- 172.21.15.81:0/1540800411 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f2f1c02ff60 0x7f2f1c032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f2f180096a0 tx=0x7f2f18006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:28:40.833 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:40.830+0000 7f2f297fa700 1 -- 172.21.15.81:0/1540800411 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2f200095f0 con 0x7f2f2c0f47c0 2023-11-15T06:28:41.035 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:41.033+0000 7f2f32cf6700 1 -- 172.21.15.81:0/1540800411 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2f2c108130 con 0x7f2f2c0f47c0 2023-11-15T06:28:41.036 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:41.034+0000 7f2f297fa700 1 -- 172.21.15.81:0/1540800411 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2f2001db70 con 0x7f2f2c0f47c0 2023-11-15T06:28:41.036 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:28:41.036 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:28:41.037 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:41.035+0000 7f2f32cf6700 1 -- 172.21.15.81:0/1540800411 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f2f1c02ff60 msgr2=0x7f2f1c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:41.037 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:41.035+0000 7f2f32cf6700 1 --2- 172.21.15.81:0/1540800411 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f2f1c02ff60 0x7f2f1c032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f2f180096a0 tx=0x7f2f18006b40).stop 2023-11-15T06:28:41.038 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:41.035+0000 7f2f32cf6700 1 -- 172.21.15.81:0/1540800411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f2c0f47c0 msgr2=0x7f2f2c106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:41.038 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:41.035+0000 7f2f32cf6700 1 --2- 172.21.15.81:0/1540800411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f2c0f47c0 0x7f2f2c106700 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f2f20013510 tx=0x7f2f2000b040).stop 2023-11-15T06:28:41.038 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:41.035+0000 7f2f32cf6700 1 -- 172.21.15.81:0/1540800411 shutdown_connections 2023-11-15T06:28:41.038 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:41.035+0000 7f2f32cf6700 1 --2- 172.21.15.81:0/1540800411 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f2f1c02ff60 0x7f2f1c032420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:41.039 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:41.035+0000 7f2f32cf6700 1 --2- 172.21.15.81:0/1540800411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f2c0f47c0 0x7f2f2c106700 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:41.039 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:41.035+0000 7f2f32cf6700 1 -- 172.21.15.81:0/1540800411 >> 172.21.15.81:0/1540800411 conn(0x7f2f2c0f02e0 msgr2=0x7f2f2c0f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:28:41.039 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:41.035+0000 7f2f32cf6700 1 -- 172.21.15.81:0/1540800411 shutdown_connections 2023-11-15T06:28:41.039 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:41.035+0000 7f2f32cf6700 1 -- 172.21.15.81:0/1540800411 wait complete. 2023-11-15T06:28:41.039 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:28:41.172 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:40 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:28:39.804132+0000 mgr.smithi062.rwurtk (mgr.14164) 84 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:28:42.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:41 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:28:41.034503+0000 mon.smithi062 (mon.0) 169 : audit [DBG] from='client.? 172.21.15.81:0/1540800411' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:28:42.430 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:28:42.430 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:28:43.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:42 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:28:41.804404+0000 mgr.smithi062.rwurtk (mgr.14164) 85 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:28:43.608 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:43.605+0000 7f3c7fc9d700 1 -- 172.21.15.81:0/3387478742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c780f5410 msgr2=0x7f3c780f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:43.608 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:43.605+0000 7f3c7fc9d700 1 --2- 172.21.15.81:0/3387478742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c780f5410 0x7f3c780f5830 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f3c68009fb0 tx=0x7f3c68009b30).stop 2023-11-15T06:28:43.609 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:43.605+0000 7f3c7fc9d700 1 -- 172.21.15.81:0/3387478742 shutdown_connections 2023-11-15T06:28:43.609 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:43.605+0000 7f3c7fc9d700 1 --2- 172.21.15.81:0/3387478742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c780f5410 0x7f3c780f5830 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:43.609 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:43.605+0000 7f3c7fc9d700 1 -- 172.21.15.81:0/3387478742 >> 172.21.15.81:0/3387478742 conn(0x7f3c780f0b70 msgr2=0x7f3c780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:28:43.609 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:43.605+0000 7f3c7fc9d700 1 -- 172.21.15.81:0/3387478742 shutdown_connections 2023-11-15T06:28:43.609 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:43.605+0000 7f3c7fc9d700 1 -- 172.21.15.81:0/3387478742 wait complete. 2023-11-15T06:28:43.610 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:43.606+0000 7f3c7fc9d700 1 Processor -- start 2023-11-15T06:28:43.610 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:43.606+0000 7f3c7fc9d700 1 -- start start 2023-11-15T06:28:43.610 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:43.606+0000 7f3c7fc9d700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c780f5410 0x7f3c78109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:28:43.610 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:43.606+0000 7f3c7fc9d700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c78060240 con 0x7f3c780f5410 2023-11-15T06:28:43.610 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:43.606+0000 7f3c7da39700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c780f5410 0x7f3c78109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:28:43.611 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:43.606+0000 7f3c7da39700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c780f5410 0x7f3c78109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:33690/0 (socket says 172.21.15.81:33690) 2023-11-15T06:28:43.611 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:43.606+0000 7f3c7da39700 1 -- 172.21.15.81:0/78858570 learned_addr learned my addr 172.21.15.81:0/78858570 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:28:43.611 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:43.606+0000 7f3c7da39700 1 -- 172.21.15.81:0/78858570 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c68005040 con 0x7f3c780f5410 2023-11-15T06:28:43.611 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:43.606+0000 7f3c7da39700 1 --2- 172.21.15.81:0/78858570 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c780f5410 0x7f3c78109430 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f3c68013510 tx=0x7f3c6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:28:43.613 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:43.607+0000 7f3c6e7fc700 1 -- 172.21.15.81:0/78858570 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c68009320 con 0x7f3c780f5410 2023-11-15T06:28:43.613 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:43.607+0000 7f3c7fc9d700 1 -- 172.21.15.81:0/78858570 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c78060440 con 0x7f3c780f5410 2023-11-15T06:28:43.613 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:43.607+0000 7f3c7fc9d700 1 -- 172.21.15.81:0/78858570 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c78109ae0 con 0x7f3c780f5410 2023-11-15T06:28:43.613 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:43.607+0000 7f3c6e7fc700 1 -- 172.21.15.81:0/78858570 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3c68013f00 con 0x7f3c780f5410 2023-11-15T06:28:43.613 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:43.607+0000 7f3c6e7fc700 1 -- 172.21.15.81:0/78858570 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c68014cc0 con 0x7f3c780f5410 2023-11-15T06:28:43.614 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:43.608+0000 7f3c7fc9d700 1 -- 172.21.15.81:0/78858570 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c60004fa0 con 0x7f3c780f5410 2023-11-15T06:28:43.614 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:43.608+0000 7f3c6e7fc700 1 -- 172.21.15.81:0/78858570 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3c68014420 con 0x7f3c780f5410 2023-11-15T06:28:43.614 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:43.608+0000 7f3c6e7fc700 1 --2- 172.21.15.81:0/78858570 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f3c6402ff60 0x7f3c64032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:28:43.614 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:43.608+0000 7f3c6e7fc700 1 -- 172.21.15.81:0/78858570 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3c68040f60 con 0x7f3c780f5410 2023-11-15T06:28:43.614 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:43.608+0000 7f3c7d238700 1 --2- 172.21.15.81:0/78858570 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f3c6402ff60 0x7f3c64032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:28:43.615 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:43.608+0000 7f3c7d238700 1 --2- 172.21.15.81:0/78858570 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f3c6402ff60 0x7f3c64032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f3c740096a0 tx=0x7f3c74006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:28:43.615 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:43.613+0000 7f3c6e7fc700 1 -- 172.21.15.81:0/78858570 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3c680095f0 con 0x7f3c780f5410 2023-11-15T06:28:43.821 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:43.820+0000 7f3c7fc9d700 1 -- 172.21.15.81:0/78858570 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3c60005670 con 0x7f3c780f5410 2023-11-15T06:28:43.822 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:43.820+0000 7f3c6e7fc700 1 -- 172.21.15.81:0/78858570 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3c68010e60 con 0x7f3c780f5410 2023-11-15T06:28:43.823 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:28:43.823 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:28:43.824 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:43.822+0000 7f3c7fc9d700 1 -- 172.21.15.81:0/78858570 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f3c6402ff60 msgr2=0x7f3c64032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:43.824 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:43.822+0000 7f3c7fc9d700 1 --2- 172.21.15.81:0/78858570 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f3c6402ff60 0x7f3c64032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f3c740096a0 tx=0x7f3c74006b40).stop 2023-11-15T06:28:43.824 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:43.822+0000 7f3c7fc9d700 1 -- 172.21.15.81:0/78858570 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c780f5410 msgr2=0x7f3c78109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:43.824 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:43.822+0000 7f3c7fc9d700 1 --2- 172.21.15.81:0/78858570 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c780f5410 0x7f3c78109430 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f3c68013510 tx=0x7f3c6800b040).stop 2023-11-15T06:28:43.824 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:43.822+0000 7f3c7fc9d700 1 -- 172.21.15.81:0/78858570 shutdown_connections 2023-11-15T06:28:43.825 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:43.822+0000 7f3c7fc9d700 1 --2- 172.21.15.81:0/78858570 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f3c6402ff60 0x7f3c64032420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:43.825 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:43.822+0000 7f3c7fc9d700 1 --2- 172.21.15.81:0/78858570 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c780f5410 0x7f3c78109430 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:43.825 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:43.822+0000 7f3c7fc9d700 1 -- 172.21.15.81:0/78858570 >> 172.21.15.81:0/78858570 conn(0x7f3c780f0b70 msgr2=0x7f3c780f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:28:43.825 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:43.822+0000 7f3c7fc9d700 1 -- 172.21.15.81:0/78858570 shutdown_connections 2023-11-15T06:28:43.826 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:43.822+0000 7f3c7fc9d700 1 -- 172.21.15.81:0/78858570 wait complete. 2023-11-15T06:28:43.826 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:28:44.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:43 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:28:43.821176+0000 mon.smithi062 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.81:0/78858570' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:28:45.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:44 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:28:43.804757+0000 mgr.smithi062.rwurtk (mgr.14164) 86 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:28:45.226 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:28:45.227 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:28:46.411 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:46.409+0000 7fe62b3f1700 1 -- 172.21.15.81:0/2067594830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6240f5410 msgr2=0x7fe6240f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:46.412 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:46.409+0000 7fe62b3f1700 1 --2- 172.21.15.81:0/2067594830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6240f5410 0x7fe6240f5830 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fe614009fb0 tx=0x7fe614009b30).stop 2023-11-15T06:28:46.412 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:46.409+0000 7fe62b3f1700 1 -- 172.21.15.81:0/2067594830 shutdown_connections 2023-11-15T06:28:46.413 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:46.409+0000 7fe62b3f1700 1 --2- 172.21.15.81:0/2067594830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6240f5410 0x7fe6240f5830 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:46.413 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:46.409+0000 7fe62b3f1700 1 -- 172.21.15.81:0/2067594830 >> 172.21.15.81:0/2067594830 conn(0x7fe6240f0b70 msgr2=0x7fe6240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:28:46.413 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:46.409+0000 7fe62b3f1700 1 -- 172.21.15.81:0/2067594830 shutdown_connections 2023-11-15T06:28:46.413 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:46.409+0000 7fe62b3f1700 1 -- 172.21.15.81:0/2067594830 wait complete. 2023-11-15T06:28:46.414 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:46.410+0000 7fe62b3f1700 1 Processor -- start 2023-11-15T06:28:46.414 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:46.410+0000 7fe62b3f1700 1 -- start start 2023-11-15T06:28:46.414 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:46.410+0000 7fe62b3f1700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6240f5410 0x7fe624109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:28:46.414 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:46.410+0000 7fe62b3f1700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe624060220 con 0x7fe6240f5410 2023-11-15T06:28:46.416 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:46.410+0000 7fe62918d700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6240f5410 0x7fe624109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:28:46.417 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:46.410+0000 7fe62918d700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6240f5410 0x7fe624109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:33700/0 (socket says 172.21.15.81:33700) 2023-11-15T06:28:46.417 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:46.410+0000 7fe62918d700 1 -- 172.21.15.81:0/862639237 learned_addr learned my addr 172.21.15.81:0/862639237 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:28:46.417 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:46.410+0000 7fe62918d700 1 -- 172.21.15.81:0/862639237 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe614005040 con 0x7fe6240f5410 2023-11-15T06:28:46.417 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:46.411+0000 7fe62918d700 1 --2- 172.21.15.81:0/862639237 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6240f5410 0x7fe624109450 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fe614014000 tx=0x7fe61400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:28:46.418 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:46.411+0000 7fe619ffb700 1 -- 172.21.15.81:0/862639237 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6140095f0 con 0x7fe6240f5410 2023-11-15T06:28:46.418 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:46.411+0000 7fe619ffb700 1 -- 172.21.15.81:0/862639237 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe614007e70 con 0x7fe6240f5410 2023-11-15T06:28:46.418 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:46.411+0000 7fe62b3f1700 1 -- 172.21.15.81:0/862639237 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe624060420 con 0x7fe6240f5410 2023-11-15T06:28:46.418 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:46.411+0000 7fe619ffb700 1 -- 172.21.15.81:0/862639237 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe614014ea0 con 0x7fe6240f5410 2023-11-15T06:28:46.419 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:46.411+0000 7fe62b3f1700 1 -- 172.21.15.81:0/862639237 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe624109b00 con 0x7fe6240f5410 2023-11-15T06:28:46.419 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:46.412+0000 7fe619ffb700 1 -- 172.21.15.81:0/862639237 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe614025070 con 0x7fe6240f5410 2023-11-15T06:28:46.419 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:46.412+0000 7fe62b3f1700 1 -- 172.21.15.81:0/862639237 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe624059100 con 0x7fe6240f5410 2023-11-15T06:28:46.419 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:46.412+0000 7fe619ffb700 1 --2- 172.21.15.81:0/862639237 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fe61002ff60 0x7fe610032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:28:46.419 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:46.412+0000 7fe619ffb700 1 -- 172.21.15.81:0/862639237 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe614040970 con 0x7fe6240f5410 2023-11-15T06:28:46.420 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:46.412+0000 7fe62898c700 1 --2- 172.21.15.81:0/862639237 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fe61002ff60 0x7fe610032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:28:46.420 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:46.412+0000 7fe62898c700 1 --2- 172.21.15.81:0/862639237 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fe61002ff60 0x7fe610032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fe6200096a0 tx=0x7fe620006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:28:46.421 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:46.416+0000 7fe619ffb700 1 -- 172.21.15.81:0/862639237 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe6140079c0 con 0x7fe6240f5410 2023-11-15T06:28:46.624 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:46.622+0000 7fe62b3f1700 1 -- 172.21.15.81:0/862639237 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe624045ad0 con 0x7fe6240f5410 2023-11-15T06:28:46.624 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:46.623+0000 7fe619ffb700 1 -- 172.21.15.81:0/862639237 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe61400f300 con 0x7fe6240f5410 2023-11-15T06:28:46.625 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:28:46.625 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:28:46.626 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:46.624+0000 7fe62b3f1700 1 -- 172.21.15.81:0/862639237 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fe61002ff60 msgr2=0x7fe610032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:46.626 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:46.624+0000 7fe62b3f1700 1 --2- 172.21.15.81:0/862639237 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fe61002ff60 0x7fe610032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fe6200096a0 tx=0x7fe620006b40).stop 2023-11-15T06:28:46.627 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:46.624+0000 7fe62b3f1700 1 -- 172.21.15.81:0/862639237 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6240f5410 msgr2=0x7fe624109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:46.627 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:46.624+0000 7fe62b3f1700 1 --2- 172.21.15.81:0/862639237 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6240f5410 0x7fe624109450 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fe614014000 tx=0x7fe61400b040).stop 2023-11-15T06:28:46.627 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:46.624+0000 7fe62b3f1700 1 -- 172.21.15.81:0/862639237 shutdown_connections 2023-11-15T06:28:46.627 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:46.624+0000 7fe62b3f1700 1 --2- 172.21.15.81:0/862639237 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fe61002ff60 0x7fe610032420 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:46.628 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:46.624+0000 7fe62b3f1700 1 --2- 172.21.15.81:0/862639237 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6240f5410 0x7fe624109450 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:46.628 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:46.624+0000 7fe62b3f1700 1 -- 172.21.15.81:0/862639237 >> 172.21.15.81:0/862639237 conn(0x7fe6240f0b70 msgr2=0x7fe6240f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:28:46.628 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:46.624+0000 7fe62b3f1700 1 -- 172.21.15.81:0/862639237 shutdown_connections 2023-11-15T06:28:46.628 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:46.624+0000 7fe62b3f1700 1 -- 172.21.15.81:0/862639237 wait complete. 2023-11-15T06:28:46.629 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:28:47.171 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:46 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:28:45.805136+0000 mgr.smithi062.rwurtk (mgr.14164) 87 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:28:47.171 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:46 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:28:46.623466+0000 mon.smithi062 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.81:0/862639237' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:28:48.036 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:28:48.036 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:28:48.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:48 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:28:47.805493+0000 mgr.smithi062.rwurtk (mgr.14164) 88 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:28:49.281 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:49.275+0000 7f761deaa700 1 -- 172.21.15.81:0/1480055425 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76180f6510 msgr2=0x7f76180f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:49.282 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:49.275+0000 7f761deaa700 1 --2- 172.21.15.81:0/1480055425 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76180f6510 0x7f76180f6930 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f7600009fb0 tx=0x7f7600009b30).stop 2023-11-15T06:28:49.282 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:49.275+0000 7f761deaa700 1 -- 172.21.15.81:0/1480055425 shutdown_connections 2023-11-15T06:28:49.282 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:49.275+0000 7f761deaa700 1 --2- 172.21.15.81:0/1480055425 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76180f6510 0x7f76180f6930 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:49.282 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:49.275+0000 7f761deaa700 1 -- 172.21.15.81:0/1480055425 >> 172.21.15.81:0/1480055425 conn(0x7f76180f1c70 msgr2=0x7f76180f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:28:49.283 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:49.275+0000 7f761deaa700 1 -- 172.21.15.81:0/1480055425 shutdown_connections 2023-11-15T06:28:49.284 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:49.275+0000 7f761deaa700 1 -- 172.21.15.81:0/1480055425 wait complete. 2023-11-15T06:28:49.284 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:49.276+0000 7f761deaa700 1 Processor -- start 2023-11-15T06:28:49.285 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:49.276+0000 7f761deaa700 1 -- start start 2023-11-15T06:28:49.285 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:49.276+0000 7f761deaa700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76180f6510 0x7f7618106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:28:49.285 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:49.276+0000 7f761deaa700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7618106c20 con 0x7f76180f6510 2023-11-15T06:28:49.285 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:49.276+0000 7f76177fe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76180f6510 0x7f7618106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:28:49.285 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:49.276+0000 7f76177fe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76180f6510 0x7f7618106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:33714/0 (socket says 172.21.15.81:33714) 2023-11-15T06:28:49.286 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:49.276+0000 7f76177fe700 1 -- 172.21.15.81:0/2162120629 learned_addr learned my addr 172.21.15.81:0/2162120629 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:28:49.286 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:49.277+0000 7f76177fe700 1 -- 172.21.15.81:0/2162120629 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7600005040 con 0x7f76180f6510 2023-11-15T06:28:49.286 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:49.277+0000 7f76177fe700 1 --2- 172.21.15.81:0/2162120629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76180f6510 0x7f7618106730 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f7600013510 tx=0x7f760000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:28:49.286 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:49.277+0000 7f760ffff700 1 -- 172.21.15.81:0/2162120629 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7600009320 con 0x7f76180f6510 2023-11-15T06:28:49.286 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:49.277+0000 7f761deaa700 1 -- 172.21.15.81:0/2162120629 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7618106e20 con 0x7f76180f6510 2023-11-15T06:28:49.287 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:49.277+0000 7f760ffff700 1 -- 172.21.15.81:0/2162120629 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7600013f00 con 0x7f76180f6510 2023-11-15T06:28:49.287 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:49.277+0000 7f760ffff700 1 -- 172.21.15.81:0/2162120629 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7600014d90 con 0x7f76180f6510 2023-11-15T06:28:49.287 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:49.277+0000 7f761deaa700 1 -- 172.21.15.81:0/2162120629 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76181072b0 con 0x7f76180f6510 2023-11-15T06:28:49.287 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:49.278+0000 7f760ffff700 1 -- 172.21.15.81:0/2162120629 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7600014420 con 0x7f76180f6510 2023-11-15T06:28:49.287 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:49.278+0000 7f761deaa700 1 -- 172.21.15.81:0/2162120629 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7618045ad0 con 0x7f76180f6510 2023-11-15T06:28:49.288 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:49.278+0000 7f760ffff700 1 --2- 172.21.15.81:0/2162120629 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f760402ff60 0x7f7604032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:28:49.288 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:49.278+0000 7f760ffff700 1 -- 172.21.15.81:0/2162120629 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7600041000 con 0x7f76180f6510 2023-11-15T06:28:49.288 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:49.278+0000 7f7616ffd700 1 --2- 172.21.15.81:0/2162120629 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f760402ff60 0x7f7604032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:28:49.288 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:49.279+0000 7f7616ffd700 1 --2- 172.21.15.81:0/2162120629 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f760402ff60 0x7f7604032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f76080096a0 tx=0x7f7608006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:28:49.289 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:49.283+0000 7f760ffff700 1 -- 172.21.15.81:0/2162120629 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f76000095f0 con 0x7f76180f6510 2023-11-15T06:28:49.490 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:49.487+0000 7f761deaa700 1 -- 172.21.15.81:0/2162120629 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f76180fbf60 con 0x7f76180f6510 2023-11-15T06:28:49.490 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:49.488+0000 7f760ffff700 1 -- 172.21.15.81:0/2162120629 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f760001db70 con 0x7f76180f6510 2023-11-15T06:28:49.491 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:28:49.491 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:28:49.492 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:49.489+0000 7f761deaa700 1 -- 172.21.15.81:0/2162120629 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f760402ff60 msgr2=0x7f7604032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:49.493 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:49.489+0000 7f761deaa700 1 --2- 172.21.15.81:0/2162120629 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f760402ff60 0x7f7604032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f76080096a0 tx=0x7f7608006b40).stop 2023-11-15T06:28:49.493 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:49.489+0000 7f761deaa700 1 -- 172.21.15.81:0/2162120629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76180f6510 msgr2=0x7f7618106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:49.493 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:49.489+0000 7f761deaa700 1 --2- 172.21.15.81:0/2162120629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76180f6510 0x7f7618106730 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f7600013510 tx=0x7f760000b040).stop 2023-11-15T06:28:49.493 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:49.489+0000 7f761deaa700 1 -- 172.21.15.81:0/2162120629 shutdown_connections 2023-11-15T06:28:49.493 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:49.489+0000 7f761deaa700 1 --2- 172.21.15.81:0/2162120629 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f760402ff60 0x7f7604032420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:49.494 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:49.489+0000 7f761deaa700 1 --2- 172.21.15.81:0/2162120629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76180f6510 0x7f7618106730 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:49.494 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:49.489+0000 7f761deaa700 1 -- 172.21.15.81:0/2162120629 >> 172.21.15.81:0/2162120629 conn(0x7f76180f1c70 msgr2=0x7f76180f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:28:49.494 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:49.490+0000 7f761deaa700 1 -- 172.21.15.81:0/2162120629 shutdown_connections 2023-11-15T06:28:49.494 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:49.490+0000 7f761deaa700 1 -- 172.21.15.81:0/2162120629 wait complete. 2023-11-15T06:28:49.495 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:28:49.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:49 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:28:49.489576+0000 mon.smithi062 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.81:0/2162120629' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:28:50.832 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:28:50.832 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:28:50.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:50 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:28:49.805828+0000 mgr.smithi062.rwurtk (mgr.14164) 89 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:28:52.058 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:52.053+0000 7f0ca47bb700 1 -- 172.21.15.81:0/2724372161 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c9c0f5410 msgr2=0x7f0c9c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:52.058 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:52.053+0000 7f0ca47bb700 1 --2- 172.21.15.81:0/2724372161 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c9c0f5410 0x7f0c9c0f5830 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f0c8c009fb0 tx=0x7f0c8c009b30).stop 2023-11-15T06:28:52.058 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:52.053+0000 7f0ca47bb700 1 -- 172.21.15.81:0/2724372161 shutdown_connections 2023-11-15T06:28:52.059 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:52.053+0000 7f0ca47bb700 1 --2- 172.21.15.81:0/2724372161 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c9c0f5410 0x7f0c9c0f5830 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:52.059 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:52.053+0000 7f0ca47bb700 1 -- 172.21.15.81:0/2724372161 >> 172.21.15.81:0/2724372161 conn(0x7f0c9c0f0b70 msgr2=0x7f0c9c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:28:52.059 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:52.053+0000 7f0ca47bb700 1 -- 172.21.15.81:0/2724372161 shutdown_connections 2023-11-15T06:28:52.059 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:52.053+0000 7f0ca47bb700 1 -- 172.21.15.81:0/2724372161 wait complete. 2023-11-15T06:28:52.059 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:52.054+0000 7f0ca47bb700 1 Processor -- start 2023-11-15T06:28:52.060 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:52.054+0000 7f0ca47bb700 1 -- start start 2023-11-15T06:28:52.060 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:52.054+0000 7f0ca47bb700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c9c0f5410 0x7f0c9c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:28:52.060 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:52.054+0000 7f0ca47bb700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c9c105b20 con 0x7f0c9c0f5410 2023-11-15T06:28:52.060 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:52.054+0000 7f0ca2557700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c9c0f5410 0x7f0c9c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:28:52.060 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:52.054+0000 7f0ca2557700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c9c0f5410 0x7f0c9c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:33736/0 (socket says 172.21.15.81:33736) 2023-11-15T06:28:52.061 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:52.054+0000 7f0ca2557700 1 -- 172.21.15.81:0/2057186277 learned_addr learned my addr 172.21.15.81:0/2057186277 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:28:52.061 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:52.055+0000 7f0ca2557700 1 -- 172.21.15.81:0/2057186277 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c8c005040 con 0x7f0c9c0f5410 2023-11-15T06:28:52.061 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:52.055+0000 7f0ca2557700 1 --2- 172.21.15.81:0/2057186277 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c9c0f5410 0x7f0c9c105630 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f0c8c013510 tx=0x7f0c8c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:28:52.061 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:52.055+0000 7f0c92ffd700 1 -- 172.21.15.81:0/2057186277 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c8c009320 con 0x7f0c9c0f5410 2023-11-15T06:28:52.061 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:52.055+0000 7f0c92ffd700 1 -- 172.21.15.81:0/2057186277 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0c8c013f00 con 0x7f0c9c0f5410 2023-11-15T06:28:52.062 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:52.055+0000 7f0c92ffd700 1 -- 172.21.15.81:0/2057186277 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c8c014d90 con 0x7f0c9c0f5410 2023-11-15T06:28:52.062 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:52.055+0000 7f0ca47bb700 1 -- 172.21.15.81:0/2057186277 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c9c105d20 con 0x7f0c9c0f5410 2023-11-15T06:28:52.062 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:52.055+0000 7f0ca47bb700 1 -- 172.21.15.81:0/2057186277 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c9c1061b0 con 0x7f0c9c0f5410 2023-11-15T06:28:52.063 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:52.056+0000 7f0c92ffd700 1 -- 172.21.15.81:0/2057186277 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0c8c014420 con 0x7f0c9c0f5410 2023-11-15T06:28:52.063 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:52.056+0000 7f0ca47bb700 1 -- 172.21.15.81:0/2057186277 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c9c045ad0 con 0x7f0c9c0f5410 2023-11-15T06:28:52.063 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:52.056+0000 7f0c92ffd700 1 --2- 172.21.15.81:0/2057186277 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f0c8802ff60 0x7f0c88032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:28:52.064 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:52.056+0000 7f0c92ffd700 1 -- 172.21.15.81:0/2057186277 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0c8c0410c0 con 0x7f0c9c0f5410 2023-11-15T06:28:52.064 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:52.056+0000 7f0ca1d56700 1 --2- 172.21.15.81:0/2057186277 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f0c8802ff60 0x7f0c88032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:28:52.064 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:52.057+0000 7f0ca1d56700 1 --2- 172.21.15.81:0/2057186277 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f0c8802ff60 0x7f0c88032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f0c980096a0 tx=0x7f0c98006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:28:52.065 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:52.061+0000 7f0c92ffd700 1 -- 172.21.15.81:0/2057186277 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0c8c026bf0 con 0x7f0c9c0f5410 2023-11-15T06:28:52.268 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:52.265+0000 7f0ca47bb700 1 -- 172.21.15.81:0/2057186277 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0c9c0fae60 con 0x7f0c9c0f5410 2023-11-15T06:28:52.269 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:52.266+0000 7f0c92ffd700 1 -- 172.21.15.81:0/2057186277 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0c8c01db70 con 0x7f0c9c0f5410 2023-11-15T06:28:52.269 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:28:52.270 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:28:52.271 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:52.268+0000 7f0ca47bb700 1 -- 172.21.15.81:0/2057186277 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f0c8802ff60 msgr2=0x7f0c88032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:52.271 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:52.268+0000 7f0ca47bb700 1 --2- 172.21.15.81:0/2057186277 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f0c8802ff60 0x7f0c88032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f0c980096a0 tx=0x7f0c98006b40).stop 2023-11-15T06:28:52.271 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:52.268+0000 7f0ca47bb700 1 -- 172.21.15.81:0/2057186277 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c9c0f5410 msgr2=0x7f0c9c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:52.271 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:52.268+0000 7f0ca47bb700 1 --2- 172.21.15.81:0/2057186277 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c9c0f5410 0x7f0c9c105630 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f0c8c013510 tx=0x7f0c8c00b040).stop 2023-11-15T06:28:52.271 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:52.268+0000 7f0ca47bb700 1 -- 172.21.15.81:0/2057186277 shutdown_connections 2023-11-15T06:28:52.272 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:52.268+0000 7f0ca47bb700 1 --2- 172.21.15.81:0/2057186277 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f0c8802ff60 0x7f0c88032420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:52.272 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:52.268+0000 7f0ca47bb700 1 --2- 172.21.15.81:0/2057186277 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c9c0f5410 0x7f0c9c105630 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:52.272 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:52.268+0000 7f0ca47bb700 1 -- 172.21.15.81:0/2057186277 >> 172.21.15.81:0/2057186277 conn(0x7f0c9c0f0b70 msgr2=0x7f0c9c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:28:52.272 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:52.268+0000 7f0ca47bb700 1 -- 172.21.15.81:0/2057186277 shutdown_connections 2023-11-15T06:28:52.272 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:52.268+0000 7f0ca47bb700 1 -- 172.21.15.81:0/2057186277 wait complete. 2023-11-15T06:28:52.273 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:28:53.169 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:52 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:28:51.806063+0000 mgr.smithi062.rwurtk (mgr.14164) 90 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:28:53.169 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:52 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:28:52.267923+0000 mon.smithi062 (mon.0) 173 : audit [DBG] from='client.? 172.21.15.81:0/2057186277' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:28:53.656 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:28:53.656 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:28:54.834 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:54.830+0000 7f5841b03700 1 -- 172.21.15.81:0/835188663 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f583c0f6510 msgr2=0x7f583c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:54.834 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:54.830+0000 7f5841b03700 1 --2- 172.21.15.81:0/835188663 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f583c0f6510 0x7f583c0f6930 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f5830009fb0 tx=0x7f5830009b30).stop 2023-11-15T06:28:54.834 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:54.830+0000 7f5841b03700 1 -- 172.21.15.81:0/835188663 shutdown_connections 2023-11-15T06:28:54.835 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:54.830+0000 7f5841b03700 1 --2- 172.21.15.81:0/835188663 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f583c0f6510 0x7f583c0f6930 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:54.835 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:54.830+0000 7f5841b03700 1 -- 172.21.15.81:0/835188663 >> 172.21.15.81:0/835188663 conn(0x7f583c0f1c70 msgr2=0x7f583c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:28:54.835 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:54.830+0000 7f5841b03700 1 -- 172.21.15.81:0/835188663 shutdown_connections 2023-11-15T06:28:54.835 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:54.830+0000 7f5841b03700 1 -- 172.21.15.81:0/835188663 wait complete. 2023-11-15T06:28:54.835 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:54.831+0000 7f5841b03700 1 Processor -- start 2023-11-15T06:28:54.836 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:54.831+0000 7f5841b03700 1 -- start start 2023-11-15T06:28:54.836 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:54.831+0000 7f5841b03700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f583c106840 0x7f583c106c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:28:54.836 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:54.831+0000 7f5841b03700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583c107150 con 0x7f583c106840 2023-11-15T06:28:54.836 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:54.831+0000 7f583b7fe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f583c106840 0x7f583c106c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:28:54.836 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:54.831+0000 7f583b7fe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f583c106840 0x7f583c106c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:47080/0 (socket says 172.21.15.81:47080) 2023-11-15T06:28:54.837 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:54.831+0000 7f583b7fe700 1 -- 172.21.15.81:0/2145993580 learned_addr learned my addr 172.21.15.81:0/2145993580 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:28:54.838 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:54.832+0000 7f583b7fe700 1 -- 172.21.15.81:0/2145993580 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5830005040 con 0x7f583c106840 2023-11-15T06:28:54.838 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:54.832+0000 7f583b7fe700 1 --2- 172.21.15.81:0/2145993580 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f583c106840 0x7f583c106c60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f5830009f80 tx=0x7f5830009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:28:54.838 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:54.832+0000 7f5823fff700 1 -- 172.21.15.81:0/2145993580 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58300136c0 con 0x7f583c106840 2023-11-15T06:28:54.839 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:54.832+0000 7f5841b03700 1 -- 172.21.15.81:0/2145993580 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f583c107350 con 0x7f583c106840 2023-11-15T06:28:54.839 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:54.832+0000 7f5823fff700 1 -- 172.21.15.81:0/2145993580 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5830013e70 con 0x7f583c106840 2023-11-15T06:28:54.839 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:54.832+0000 7f5823fff700 1 -- 172.21.15.81:0/2145993580 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5830014420 con 0x7f583c106840 2023-11-15T06:28:54.839 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:54.832+0000 7f5841b03700 1 -- 172.21.15.81:0/2145993580 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f583c107ea0 con 0x7f583c106840 2023-11-15T06:28:54.839 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:54.833+0000 7f5823fff700 1 -- 172.21.15.81:0/2145993580 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5830025060 con 0x7f583c106840 2023-11-15T06:28:54.840 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:54.833+0000 7f5841b03700 1 -- 172.21.15.81:0/2145993580 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f583c059100 con 0x7f583c106840 2023-11-15T06:28:54.840 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:54.833+0000 7f5823fff700 1 --2- 172.21.15.81:0/2145993580 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f582402ff60 0x7f5824032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:28:54.840 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:54.833+0000 7f5823fff700 1 -- 172.21.15.81:0/2145993580 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5830040e40 con 0x7f583c106840 2023-11-15T06:28:54.840 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:54.833+0000 7f583affd700 1 --2- 172.21.15.81:0/2145993580 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f582402ff60 0x7f5824032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:28:54.840 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:54.833+0000 7f583affd700 1 --2- 172.21.15.81:0/2145993580 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f582402ff60 0x7f5824032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f582c0096a0 tx=0x7f582c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:28:54.841 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:54.838+0000 7f5823fff700 1 -- 172.21.15.81:0/2145993580 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f583001cd10 con 0x7f583c106840 2023-11-15T06:28:55.047 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:55.044+0000 7f5841b03700 1 -- 172.21.15.81:0/2145993580 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f583c045ad0 con 0x7f583c106840 2023-11-15T06:28:55.048 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:55.045+0000 7f5823fff700 1 -- 172.21.15.81:0/2145993580 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f583001cb30 con 0x7f583c106840 2023-11-15T06:28:55.048 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:28:55.048 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:28:55.049 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:55.046+0000 7f5841b03700 1 -- 172.21.15.81:0/2145993580 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f582402ff60 msgr2=0x7f5824032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:55.049 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:55.046+0000 7f5841b03700 1 --2- 172.21.15.81:0/2145993580 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f582402ff60 0x7f5824032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f582c0096a0 tx=0x7f582c006b40).stop 2023-11-15T06:28:55.049 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:55.046+0000 7f5841b03700 1 -- 172.21.15.81:0/2145993580 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f583c106840 msgr2=0x7f583c106c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:55.050 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:55.046+0000 7f5841b03700 1 --2- 172.21.15.81:0/2145993580 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f583c106840 0x7f583c106c60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f5830009f80 tx=0x7f5830009b30).stop 2023-11-15T06:28:55.050 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:55.046+0000 7f5841b03700 1 -- 172.21.15.81:0/2145993580 shutdown_connections 2023-11-15T06:28:55.050 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:55.046+0000 7f5841b03700 1 --2- 172.21.15.81:0/2145993580 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f582402ff60 0x7f5824032420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:55.050 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:55.046+0000 7f5841b03700 1 --2- 172.21.15.81:0/2145993580 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f583c106840 0x7f583c106c60 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:55.051 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:55.046+0000 7f5841b03700 1 -- 172.21.15.81:0/2145993580 >> 172.21.15.81:0/2145993580 conn(0x7f583c0f1c70 msgr2=0x7f583c0f2800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:28:55.051 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:55.046+0000 7f5841b03700 1 -- 172.21.15.81:0/2145993580 shutdown_connections 2023-11-15T06:28:55.051 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:55.046+0000 7f5841b03700 1 -- 172.21.15.81:0/2145993580 wait complete. 2023-11-15T06:28:55.051 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:28:55.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:54 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:28:53.806388+0000 mgr.smithi062.rwurtk (mgr.14164) 91 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:28:55.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:54 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:28:53.916341+0000 mon.smithi062 (mon.0) 174 : audit [INF] from='mgr.14164 172.21.15.62:0/147479783' entity='mgr.smithi062.rwurtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi062.rwurtk/mirror_snapshot_schedule"}]: dispatch 2023-11-15T06:28:55.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:54 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:28:53.918186+0000 mon.smithi062 (mon.0) 175 : audit [INF] from='mgr.14164 172.21.15.62:0/147479783' entity='mgr.smithi062.rwurtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi062.rwurtk/trash_purge_schedule"}]: dispatch 2023-11-15T06:28:56.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:55 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:28:55.046682+0000 mon.smithi062 (mon.0) 176 : audit [DBG] from='client.? 172.21.15.81:0/2145993580' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:28:56.435 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:28:56.436 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:28:57.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:56 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:28:55.806756+0000 mgr.smithi062.rwurtk (mgr.14164) 92 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:28:57.619 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:57.615+0000 7ff5805c2700 1 -- 172.21.15.81:0/2062550820 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5780f5410 msgr2=0x7ff5780f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:57.620 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:57.615+0000 7ff5805c2700 1 --2- 172.21.15.81:0/2062550820 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5780f5410 0x7ff5780f5830 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7ff568009fb0 tx=0x7ff568009b30).stop 2023-11-15T06:28:57.620 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:57.615+0000 7ff5805c2700 1 -- 172.21.15.81:0/2062550820 shutdown_connections 2023-11-15T06:28:57.620 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:57.615+0000 7ff5805c2700 1 --2- 172.21.15.81:0/2062550820 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5780f5410 0x7ff5780f5830 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:57.621 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:57.615+0000 7ff5805c2700 1 -- 172.21.15.81:0/2062550820 >> 172.21.15.81:0/2062550820 conn(0x7ff5780f0b70 msgr2=0x7ff5780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:28:57.621 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:57.615+0000 7ff5805c2700 1 -- 172.21.15.81:0/2062550820 shutdown_connections 2023-11-15T06:28:57.621 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:57.615+0000 7ff5805c2700 1 -- 172.21.15.81:0/2062550820 wait complete. 2023-11-15T06:28:57.621 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:57.616+0000 7ff5805c2700 1 Processor -- start 2023-11-15T06:28:57.621 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:57.616+0000 7ff5805c2700 1 -- start start 2023-11-15T06:28:57.621 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:57.616+0000 7ff5805c2700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5780f5410 0x7ff578106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:28:57.622 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:57.616+0000 7ff5805c2700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff578106c10 con 0x7ff5780f5410 2023-11-15T06:28:57.622 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:57.616+0000 7ff57e35e700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5780f5410 0x7ff578106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:28:57.622 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:57.616+0000 7ff57e35e700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5780f5410 0x7ff578106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:47106/0 (socket says 172.21.15.81:47106) 2023-11-15T06:28:57.622 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:57.616+0000 7ff57e35e700 1 -- 172.21.15.81:0/1289548323 learned_addr learned my addr 172.21.15.81:0/1289548323 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:28:57.622 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:57.616+0000 7ff57e35e700 1 -- 172.21.15.81:0/1289548323 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff568005040 con 0x7ff5780f5410 2023-11-15T06:28:57.623 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:57.617+0000 7ff57e35e700 1 --2- 172.21.15.81:0/1289548323 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5780f5410 0x7ff578106720 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7ff568014000 tx=0x7ff56800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:28:57.623 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:57.617+0000 7ff56effd700 1 -- 172.21.15.81:0/1289548323 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff5680095f0 con 0x7ff5780f5410 2023-11-15T06:28:57.623 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:57.617+0000 7ff5805c2700 1 -- 172.21.15.81:0/1289548323 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff578106e10 con 0x7ff5780f5410 2023-11-15T06:28:57.623 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:57.617+0000 7ff56effd700 1 -- 172.21.15.81:0/1289548323 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff568007e70 con 0x7ff5780f5410 2023-11-15T06:28:57.624 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:57.617+0000 7ff56effd700 1 -- 172.21.15.81:0/1289548323 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff568014ea0 con 0x7ff5780f5410 2023-11-15T06:28:57.624 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:57.617+0000 7ff5805c2700 1 -- 172.21.15.81:0/1289548323 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5781072a0 con 0x7ff5780f5410 2023-11-15T06:28:57.625 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:57.618+0000 7ff56effd700 1 -- 172.21.15.81:0/1289548323 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff568025070 con 0x7ff5780f5410 2023-11-15T06:28:57.625 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:57.618+0000 7ff5805c2700 1 -- 172.21.15.81:0/1289548323 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff578045ad0 con 0x7ff5780f5410 2023-11-15T06:28:57.626 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:57.618+0000 7ff56effd700 1 --2- 172.21.15.81:0/1289548323 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff56402ff60 0x7ff564032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:28:57.626 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:57.618+0000 7ff56effd700 1 -- 172.21.15.81:0/1289548323 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff5680409f0 con 0x7ff5780f5410 2023-11-15T06:28:57.627 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:57.618+0000 7ff57db5d700 1 --2- 172.21.15.81:0/1289548323 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff56402ff60 0x7ff564032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:28:57.627 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:57.619+0000 7ff57db5d700 1 --2- 172.21.15.81:0/1289548323 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff56402ff60 0x7ff564032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7ff5740096a0 tx=0x7ff574006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:28:57.627 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:57.623+0000 7ff56effd700 1 -- 172.21.15.81:0/1289548323 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff5680079c0 con 0x7ff5780f5410 2023-11-15T06:28:57.830 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:57.828+0000 7ff5805c2700 1 -- 172.21.15.81:0/1289548323 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff578108150 con 0x7ff5780f5410 2023-11-15T06:28:57.831 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:57.828+0000 7ff56effd700 1 -- 172.21.15.81:0/1289548323 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff56800f300 con 0x7ff5780f5410 2023-11-15T06:28:57.832 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:28:57.832 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:28:57.833 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:57.830+0000 7ff5805c2700 1 -- 172.21.15.81:0/1289548323 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff56402ff60 msgr2=0x7ff564032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:57.833 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:57.830+0000 7ff5805c2700 1 --2- 172.21.15.81:0/1289548323 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff56402ff60 0x7ff564032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7ff5740096a0 tx=0x7ff574006b40).stop 2023-11-15T06:28:57.833 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:57.830+0000 7ff5805c2700 1 -- 172.21.15.81:0/1289548323 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5780f5410 msgr2=0x7ff578106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:28:57.834 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:57.830+0000 7ff5805c2700 1 --2- 172.21.15.81:0/1289548323 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5780f5410 0x7ff578106720 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7ff568014000 tx=0x7ff56800b040).stop 2023-11-15T06:28:57.834 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:57.830+0000 7ff5805c2700 1 -- 172.21.15.81:0/1289548323 shutdown_connections 2023-11-15T06:28:57.834 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:57.830+0000 7ff5805c2700 1 --2- 172.21.15.81:0/1289548323 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff56402ff60 0x7ff564032420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:57.834 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:57.830+0000 7ff5805c2700 1 --2- 172.21.15.81:0/1289548323 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5780f5410 0x7ff578106720 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:28:57.834 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:57.830+0000 7ff5805c2700 1 -- 172.21.15.81:0/1289548323 >> 172.21.15.81:0/1289548323 conn(0x7ff5780f0b70 msgr2=0x7ff57805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:28:57.835 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:57.830+0000 7ff5805c2700 1 -- 172.21.15.81:0/1289548323 shutdown_connections 2023-11-15T06:28:57.835 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:28:57.830+0000 7ff5805c2700 1 -- 172.21.15.81:0/1289548323 wait complete. 2023-11-15T06:28:57.835 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:28:58.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:58 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:28:57.807046+0000 mgr.smithi062.rwurtk (mgr.14164) 93 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:28:58.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:28:58 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:28:57.830113+0000 mon.smithi062 (mon.0) 177 : audit [DBG] from='client.? 172.21.15.81:0/1289548323' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:28:59.208 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:28:59.209 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:29:00.440 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:00.433+0000 7f7d517a7700 1 -- 172.21.15.81:0/2965226979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d4c0f5410 msgr2=0x7f7d4c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:00.440 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:00.433+0000 7f7d517a7700 1 --2- 172.21.15.81:0/2965226979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d4c0f5410 0x7f7d4c0f5830 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f7d40009fb0 tx=0x7f7d40009b30).stop 2023-11-15T06:29:00.441 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:00.433+0000 7f7d517a7700 1 -- 172.21.15.81:0/2965226979 shutdown_connections 2023-11-15T06:29:00.441 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:00.433+0000 7f7d517a7700 1 --2- 172.21.15.81:0/2965226979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d4c0f5410 0x7f7d4c0f5830 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:00.441 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:00.433+0000 7f7d517a7700 1 -- 172.21.15.81:0/2965226979 >> 172.21.15.81:0/2965226979 conn(0x7f7d4c0f0b70 msgr2=0x7f7d4c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:29:00.441 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:00.433+0000 7f7d517a7700 1 -- 172.21.15.81:0/2965226979 shutdown_connections 2023-11-15T06:29:00.441 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:00.433+0000 7f7d517a7700 1 -- 172.21.15.81:0/2965226979 wait complete. 2023-11-15T06:29:00.442 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:00.434+0000 7f7d517a7700 1 Processor -- start 2023-11-15T06:29:00.442 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:00.434+0000 7f7d517a7700 1 -- start start 2023-11-15T06:29:00.442 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:00.434+0000 7f7d517a7700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d4c0f5410 0x7f7d4c105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:29:00.442 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:00.434+0000 7f7d517a7700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4c105af0 con 0x7f7d4c0f5410 2023-11-15T06:29:00.443 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:00.435+0000 7f7d4affd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d4c0f5410 0x7f7d4c105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:29:00.446 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:00.435+0000 7f7d4affd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d4c0f5410 0x7f7d4c105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:47124/0 (socket says 172.21.15.81:47124) 2023-11-15T06:29:00.446 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:00.435+0000 7f7d4affd700 1 -- 172.21.15.81:0/3300771449 learned_addr learned my addr 172.21.15.81:0/3300771449 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:29:00.448 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:00.435+0000 7f7d4affd700 1 -- 172.21.15.81:0/3300771449 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d40005040 con 0x7f7d4c0f5410 2023-11-15T06:29:00.449 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:00.435+0000 7f7d4affd700 1 --2- 172.21.15.81:0/3300771449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d4c0f5410 0x7f7d4c105600 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f7d40013510 tx=0x7f7d4000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:29:00.449 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:00.435+0000 7f7d337fe700 1 -- 172.21.15.81:0/3300771449 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d40009320 con 0x7f7d4c0f5410 2023-11-15T06:29:00.449 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:00.435+0000 7f7d337fe700 1 -- 172.21.15.81:0/3300771449 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7d40013f00 con 0x7f7d4c0f5410 2023-11-15T06:29:00.449 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:00.435+0000 7f7d517a7700 1 -- 172.21.15.81:0/3300771449 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d4c105cf0 con 0x7f7d4c0f5410 2023-11-15T06:29:00.450 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:00.435+0000 7f7d337fe700 1 -- 172.21.15.81:0/3300771449 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d40014d90 con 0x7f7d4c0f5410 2023-11-15T06:29:00.450 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:00.435+0000 7f7d517a7700 1 -- 172.21.15.81:0/3300771449 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d4c106180 con 0x7f7d4c0f5410 2023-11-15T06:29:00.450 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:00.436+0000 7f7d337fe700 1 -- 172.21.15.81:0/3300771449 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7d400263a0 con 0x7f7d4c0f5410 2023-11-15T06:29:00.450 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:00.436+0000 7f7d337fe700 1 --2- 172.21.15.81:0/3300771449 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f7d3402ff60 0x7f7d34032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:29:00.450 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:00.436+0000 7f7d337fe700 1 -- 172.21.15.81:0/3300771449 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7d40041090 con 0x7f7d4c0f5410 2023-11-15T06:29:00.451 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:00.436+0000 7f7d517a7700 1 -- 172.21.15.81:0/3300771449 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d4c045ad0 con 0x7f7d4c0f5410 2023-11-15T06:29:00.451 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:00.436+0000 7f7d4a7fc700 1 --2- 172.21.15.81:0/3300771449 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f7d3402ff60 0x7f7d34032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:29:00.451 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:00.437+0000 7f7d4a7fc700 1 --2- 172.21.15.81:0/3300771449 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f7d3402ff60 0x7f7d34032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f7d3c0096a0 tx=0x7f7d3c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:29:00.452 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:00.441+0000 7f7d337fe700 1 -- 172.21.15.81:0/3300771449 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7d40012920 con 0x7f7d4c0f5410 2023-11-15T06:29:00.648 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:00.645+0000 7f7d517a7700 1 -- 172.21.15.81:0/3300771449 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7d4c0faf50 con 0x7f7d4c0f5410 2023-11-15T06:29:00.649 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:00.646+0000 7f7d337fe700 1 -- 172.21.15.81:0/3300771449 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7d4002b020 con 0x7f7d4c0f5410 2023-11-15T06:29:00.649 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:29:00.650 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:29:00.651 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:00.648+0000 7f7d517a7700 1 -- 172.21.15.81:0/3300771449 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f7d3402ff60 msgr2=0x7f7d34032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:00.651 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:00.648+0000 7f7d517a7700 1 --2- 172.21.15.81:0/3300771449 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f7d3402ff60 0x7f7d34032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f7d3c0096a0 tx=0x7f7d3c006b40).stop 2023-11-15T06:29:00.651 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:00.648+0000 7f7d517a7700 1 -- 172.21.15.81:0/3300771449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d4c0f5410 msgr2=0x7f7d4c105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:00.651 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:00.648+0000 7f7d517a7700 1 --2- 172.21.15.81:0/3300771449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d4c0f5410 0x7f7d4c105600 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f7d40013510 tx=0x7f7d4000b040).stop 2023-11-15T06:29:00.652 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:00.648+0000 7f7d517a7700 1 -- 172.21.15.81:0/3300771449 shutdown_connections 2023-11-15T06:29:00.652 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:00.648+0000 7f7d517a7700 1 --2- 172.21.15.81:0/3300771449 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f7d3402ff60 0x7f7d34032420 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:00.652 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:00.648+0000 7f7d517a7700 1 --2- 172.21.15.81:0/3300771449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d4c0f5410 0x7f7d4c105600 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:00.652 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:00.648+0000 7f7d517a7700 1 -- 172.21.15.81:0/3300771449 >> 172.21.15.81:0/3300771449 conn(0x7f7d4c0f0b70 msgr2=0x7f7d4c0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:29:00.653 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:00.648+0000 7f7d517a7700 1 -- 172.21.15.81:0/3300771449 shutdown_connections 2023-11-15T06:29:00.653 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:00.648+0000 7f7d517a7700 1 -- 172.21.15.81:0/3300771449 wait complete. 2023-11-15T06:29:00.653 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:29:01.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:00 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:28:59.807355+0000 mgr.smithi062.rwurtk (mgr.14164) 94 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:29:01.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:00 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:29:00.647846+0000 mon.smithi062 (mon.0) 178 : audit [DBG] from='client.? 172.21.15.81:0/3300771449' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:29:02.070 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:29:02.070 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:29:03.170 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:29:01.807661+0000 mgr.smithi062.rwurtk (mgr.14164) 95 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:29:03.341 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:03.337+0000 7fe5482b7700 1 -- 172.21.15.81:0/2893172306 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe5400f6510 msgr2=0x7fe5400f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:03.341 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:03.337+0000 7fe5482b7700 1 --2- 172.21.15.81:0/2893172306 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe5400f6510 0x7fe5400f6930 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7fe53c009fb0 tx=0x7fe53c009b30).stop 2023-11-15T06:29:03.341 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:03.337+0000 7fe5482b7700 1 -- 172.21.15.81:0/2893172306 shutdown_connections 2023-11-15T06:29:03.341 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:03.337+0000 7fe5482b7700 1 --2- 172.21.15.81:0/2893172306 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe5400f6510 0x7fe5400f6930 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:03.342 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:03.337+0000 7fe5482b7700 1 -- 172.21.15.81:0/2893172306 >> 172.21.15.81:0/2893172306 conn(0x7fe5400f1c70 msgr2=0x7fe5400f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:29:03.342 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:03.337+0000 7fe5482b7700 1 -- 172.21.15.81:0/2893172306 shutdown_connections 2023-11-15T06:29:03.342 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:03.337+0000 7fe5482b7700 1 -- 172.21.15.81:0/2893172306 wait complete. 2023-11-15T06:29:03.342 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:03.337+0000 7fe5482b7700 1 Processor -- start 2023-11-15T06:29:03.342 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:03.337+0000 7fe5482b7700 1 -- start start 2023-11-15T06:29:03.343 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:03.337+0000 7fe5482b7700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe5400f6510 0x7fe540106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:29:03.343 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:03.337+0000 7fe5482b7700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe540106c20 con 0x7fe5400f6510 2023-11-15T06:29:03.343 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:03.338+0000 7fe546053700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe5400f6510 0x7fe540106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:29:03.343 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:03.338+0000 7fe546053700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe5400f6510 0x7fe540106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:54592/0 (socket says 172.21.15.81:54592) 2023-11-15T06:29:03.343 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:03.338+0000 7fe546053700 1 -- 172.21.15.81:0/229345737 learned_addr learned my addr 172.21.15.81:0/229345737 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:29:03.344 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:03.338+0000 7fe546053700 1 -- 172.21.15.81:0/229345737 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe53c005040 con 0x7fe5400f6510 2023-11-15T06:29:03.344 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:03.338+0000 7fe546053700 1 --2- 172.21.15.81:0/229345737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe5400f6510 0x7fe540106730 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fe53c014000 tx=0x7fe53c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:29:03.345 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:03.338+0000 7fe536ffd700 1 -- 172.21.15.81:0/229345737 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe53c0095f0 con 0x7fe5400f6510 2023-11-15T06:29:03.345 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:03.338+0000 7fe5482b7700 1 -- 172.21.15.81:0/229345737 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe540106e20 con 0x7fe5400f6510 2023-11-15T06:29:03.346 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:03.338+0000 7fe536ffd700 1 -- 172.21.15.81:0/229345737 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe53c007e70 con 0x7fe5400f6510 2023-11-15T06:29:03.346 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:03.338+0000 7fe536ffd700 1 -- 172.21.15.81:0/229345737 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe53c014ea0 con 0x7fe5400f6510 2023-11-15T06:29:03.346 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:03.338+0000 7fe5482b7700 1 -- 172.21.15.81:0/229345737 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5401072b0 con 0x7fe5400f6510 2023-11-15T06:29:03.346 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:03.339+0000 7fe536ffd700 1 -- 172.21.15.81:0/229345737 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe53c01d4e0 con 0x7fe5400f6510 2023-11-15T06:29:03.346 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:03.339+0000 7fe5482b7700 1 -- 172.21.15.81:0/229345737 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe540045ad0 con 0x7fe5400f6510 2023-11-15T06:29:03.347 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:03.339+0000 7fe536ffd700 1 --2- 172.21.15.81:0/229345737 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fe52c02ff60 0x7fe52c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:29:03.347 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:03.339+0000 7fe536ffd700 1 -- 172.21.15.81:0/229345737 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe53c040c80 con 0x7fe5400f6510 2023-11-15T06:29:03.347 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:03.340+0000 7fe545852700 1 --2- 172.21.15.81:0/229345737 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fe52c02ff60 0x7fe52c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:29:03.347 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:03.340+0000 7fe545852700 1 --2- 172.21.15.81:0/229345737 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fe52c02ff60 0x7fe52c032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fe5300096a0 tx=0x7fe530006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:29:03.348 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:03.345+0000 7fe536ffd700 1 -- 172.21.15.81:0/229345737 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe53c00f8f0 con 0x7fe5400f6510 2023-11-15T06:29:03.551 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:03.548+0000 7fe5482b7700 1 -- 172.21.15.81:0/229345737 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe5400fbf60 con 0x7fe5400f6510 2023-11-15T06:29:03.551 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:03.549+0000 7fe536ffd700 1 -- 172.21.15.81:0/229345737 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe53c012330 con 0x7fe5400f6510 2023-11-15T06:29:03.552 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:29:03.552 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:29:03.553 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:03.550+0000 7fe5482b7700 1 -- 172.21.15.81:0/229345737 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fe52c02ff60 msgr2=0x7fe52c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:03.553 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:03.550+0000 7fe5482b7700 1 --2- 172.21.15.81:0/229345737 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fe52c02ff60 0x7fe52c032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fe5300096a0 tx=0x7fe530006b40).stop 2023-11-15T06:29:03.553 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:03.550+0000 7fe5482b7700 1 -- 172.21.15.81:0/229345737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe5400f6510 msgr2=0x7fe540106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:03.554 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:03.550+0000 7fe5482b7700 1 --2- 172.21.15.81:0/229345737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe5400f6510 0x7fe540106730 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fe53c014000 tx=0x7fe53c00b040).stop 2023-11-15T06:29:03.554 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:03.550+0000 7fe5482b7700 1 -- 172.21.15.81:0/229345737 shutdown_connections 2023-11-15T06:29:03.554 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:03.550+0000 7fe5482b7700 1 --2- 172.21.15.81:0/229345737 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fe52c02ff60 0x7fe52c032420 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:03.554 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:03.550+0000 7fe5482b7700 1 --2- 172.21.15.81:0/229345737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe5400f6510 0x7fe540106730 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:03.554 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:03.550+0000 7fe5482b7700 1 -- 172.21.15.81:0/229345737 >> 172.21.15.81:0/229345737 conn(0x7fe5400f1c70 msgr2=0x7fe5400f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:29:03.555 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:03.550+0000 7fe5482b7700 1 -- 172.21.15.81:0/229345737 shutdown_connections 2023-11-15T06:29:03.555 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:03.550+0000 7fe5482b7700 1 -- 172.21.15.81:0/229345737 wait complete. 2023-11-15T06:29:03.556 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:29:04.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:03 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:29:03.550455+0000 mon.smithi062 (mon.0) 179 : audit [DBG] from='client.? 172.21.15.81:0/229345737' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:29:04.924 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:29:04.924 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:29:05.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:04 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:29:03.807961+0000 mgr.smithi062.rwurtk (mgr.14164) 96 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:29:06.137 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:06.133+0000 7fe013dea700 1 -- 172.21.15.81:0/2676468068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe00c0f5410 msgr2=0x7fe00c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:06.137 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:06.133+0000 7fe013dea700 1 --2- 172.21.15.81:0/2676468068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe00c0f5410 0x7fe00c0f5830 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7fdffc009fb0 tx=0x7fdffc009b30).stop 2023-11-15T06:29:06.137 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:06.133+0000 7fe013dea700 1 -- 172.21.15.81:0/2676468068 shutdown_connections 2023-11-15T06:29:06.137 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:06.133+0000 7fe013dea700 1 --2- 172.21.15.81:0/2676468068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe00c0f5410 0x7fe00c0f5830 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:06.138 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:06.133+0000 7fe013dea700 1 -- 172.21.15.81:0/2676468068 >> 172.21.15.81:0/2676468068 conn(0x7fe00c0f0b70 msgr2=0x7fe00c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:29:06.138 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:06.134+0000 7fe013dea700 1 -- 172.21.15.81:0/2676468068 shutdown_connections 2023-11-15T06:29:06.138 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:06.134+0000 7fe013dea700 1 -- 172.21.15.81:0/2676468068 wait complete. 2023-11-15T06:29:06.138 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:06.134+0000 7fe013dea700 1 Processor -- start 2023-11-15T06:29:06.138 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:06.134+0000 7fe013dea700 1 -- start start 2023-11-15T06:29:06.140 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:06.134+0000 7fe013dea700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe00c0f5410 0x7fe00c109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:29:06.140 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:06.134+0000 7fe013dea700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe00c060220 con 0x7fe00c0f5410 2023-11-15T06:29:06.141 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:06.134+0000 7fe011b86700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe00c0f5410 0x7fe00c109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:29:06.141 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:06.134+0000 7fe011b86700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe00c0f5410 0x7fe00c109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:54616/0 (socket says 172.21.15.81:54616) 2023-11-15T06:29:06.141 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:06.134+0000 7fe011b86700 1 -- 172.21.15.81:0/2764635219 learned_addr learned my addr 172.21.15.81:0/2764635219 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:29:06.143 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:06.135+0000 7fe011b86700 1 -- 172.21.15.81:0/2764635219 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdffc005040 con 0x7fe00c0f5410 2023-11-15T06:29:06.143 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:06.135+0000 7fe011b86700 1 --2- 172.21.15.81:0/2764635219 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe00c0f5410 0x7fe00c109450 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fdffc013510 tx=0x7fdffc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:29:06.143 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:06.135+0000 7fe0027fc700 1 -- 172.21.15.81:0/2764635219 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdffc009320 con 0x7fe00c0f5410 2023-11-15T06:29:06.144 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:06.135+0000 7fe013dea700 1 -- 172.21.15.81:0/2764635219 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe00c060420 con 0x7fe00c0f5410 2023-11-15T06:29:06.144 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:06.135+0000 7fe0027fc700 1 -- 172.21.15.81:0/2764635219 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdffc013f00 con 0x7fe00c0f5410 2023-11-15T06:29:06.144 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:06.135+0000 7fe0027fc700 1 -- 172.21.15.81:0/2764635219 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdffc014d90 con 0x7fe00c0f5410 2023-11-15T06:29:06.144 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:06.135+0000 7fe013dea700 1 -- 172.21.15.81:0/2764635219 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe00c109b00 con 0x7fe00c0f5410 2023-11-15T06:29:06.144 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:06.136+0000 7fe0027fc700 1 -- 172.21.15.81:0/2764635219 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdffc0095f0 con 0x7fe00c0f5410 2023-11-15T06:29:06.145 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:06.136+0000 7fe013dea700 1 -- 172.21.15.81:0/2764635219 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe00c059100 con 0x7fe00c0f5410 2023-11-15T06:29:06.145 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:06.136+0000 7fe0027fc700 1 --2- 172.21.15.81:0/2764635219 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fdff802ff60 0x7fdff8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:29:06.145 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:06.136+0000 7fe0027fc700 1 -- 172.21.15.81:0/2764635219 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdffc041000 con 0x7fe00c0f5410 2023-11-15T06:29:06.145 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:06.136+0000 7fe011385700 1 --2- 172.21.15.81:0/2764635219 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fdff802ff60 0x7fdff8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:29:06.145 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:06.137+0000 7fe011385700 1 --2- 172.21.15.81:0/2764635219 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fdff802ff60 0x7fdff8032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fe0080096a0 tx=0x7fe008006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:29:06.146 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:06.142+0000 7fe0027fc700 1 -- 172.21.15.81:0/2764635219 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdffc01d7f0 con 0x7fe00c0f5410 2023-11-15T06:29:06.350 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:06.348+0000 7fe013dea700 1 -- 172.21.15.81:0/2764635219 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe00c045ad0 con 0x7fe00c0f5410 2023-11-15T06:29:06.351 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:06.349+0000 7fe0027fc700 1 -- 172.21.15.81:0/2764635219 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdffc010230 con 0x7fe00c0f5410 2023-11-15T06:29:06.351 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:29:06.351 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:29:06.353 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:06.350+0000 7fe013dea700 1 -- 172.21.15.81:0/2764635219 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fdff802ff60 msgr2=0x7fdff8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:06.353 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:06.351+0000 7fe013dea700 1 --2- 172.21.15.81:0/2764635219 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fdff802ff60 0x7fdff8032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fe0080096a0 tx=0x7fe008006b40).stop 2023-11-15T06:29:06.353 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:06.351+0000 7fe013dea700 1 -- 172.21.15.81:0/2764635219 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe00c0f5410 msgr2=0x7fe00c109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:06.353 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:06.351+0000 7fe013dea700 1 --2- 172.21.15.81:0/2764635219 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe00c0f5410 0x7fe00c109450 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fdffc013510 tx=0x7fdffc00b040).stop 2023-11-15T06:29:06.353 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:06.351+0000 7fe013dea700 1 -- 172.21.15.81:0/2764635219 shutdown_connections 2023-11-15T06:29:06.354 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:06.351+0000 7fe013dea700 1 --2- 172.21.15.81:0/2764635219 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fdff802ff60 0x7fdff8032420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:06.354 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:06.351+0000 7fe013dea700 1 --2- 172.21.15.81:0/2764635219 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe00c0f5410 0x7fe00c109450 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:06.354 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:06.351+0000 7fe013dea700 1 -- 172.21.15.81:0/2764635219 >> 172.21.15.81:0/2764635219 conn(0x7fe00c0f0b70 msgr2=0x7fe00c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:29:06.354 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:06.351+0000 7fe013dea700 1 -- 172.21.15.81:0/2764635219 shutdown_connections 2023-11-15T06:29:06.354 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:06.351+0000 7fe013dea700 1 -- 172.21.15.81:0/2764635219 wait complete. 2023-11-15T06:29:06.355 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:29:07.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:06 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:29:05.808258+0000 mgr.smithi062.rwurtk (mgr.14164) 97 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:29:07.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:06 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:29:06.349590+0000 mon.smithi062 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.81:0/2764635219' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:29:07.739 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:29:07.740 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:29:08.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:08 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:29:07.808593+0000 mgr.smithi062.rwurtk (mgr.14164) 98 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:29:08.937 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:08.932+0000 7f1c4cba1700 1 -- 172.21.15.81:0/2653888116 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c480601e0 msgr2=0x7f1c480f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:08.938 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:08.932+0000 7f1c4cba1700 1 --2- 172.21.15.81:0/2653888116 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c480601e0 0x7f1c480f8f20 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f1c38009fb0 tx=0x7f1c38009b30).stop 2023-11-15T06:29:08.938 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:08.932+0000 7f1c4cba1700 1 -- 172.21.15.81:0/2653888116 shutdown_connections 2023-11-15T06:29:08.939 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:08.932+0000 7f1c4cba1700 1 --2- 172.21.15.81:0/2653888116 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c480601e0 0x7f1c480f8f20 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:08.939 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:08.932+0000 7f1c4cba1700 1 -- 172.21.15.81:0/2653888116 >> 172.21.15.81:0/2653888116 conn(0x7f1c480f09a0 msgr2=0x7f1c480f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:29:08.940 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:08.932+0000 7f1c4cba1700 1 -- 172.21.15.81:0/2653888116 shutdown_connections 2023-11-15T06:29:08.940 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:08.932+0000 7f1c4cba1700 1 -- 172.21.15.81:0/2653888116 wait complete. 2023-11-15T06:29:08.940 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:08.932+0000 7f1c4cba1700 1 Processor -- start 2023-11-15T06:29:08.941 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:08.933+0000 7f1c4cba1700 1 -- start start 2023-11-15T06:29:08.941 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:08.933+0000 7f1c4cba1700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c480601e0 0x7f1c48106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:29:08.941 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:08.933+0000 7f1c4cba1700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c481069f0 con 0x7f1c480601e0 2023-11-15T06:29:08.942 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:08.933+0000 7f1c477fe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c480601e0 0x7f1c48106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:29:08.942 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:08.933+0000 7f1c477fe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c480601e0 0x7f1c48106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:54640/0 (socket says 172.21.15.81:54640) 2023-11-15T06:29:08.943 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:08.933+0000 7f1c477fe700 1 -- 172.21.15.81:0/218757465 learned_addr learned my addr 172.21.15.81:0/218757465 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:29:08.943 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:08.933+0000 7f1c477fe700 1 -- 172.21.15.81:0/218757465 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c38005040 con 0x7f1c480601e0 2023-11-15T06:29:08.943 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:08.934+0000 7f1c477fe700 1 --2- 172.21.15.81:0/218757465 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c480601e0 0x7f1c48106500 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f1c38013510 tx=0x7f1c3800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:29:08.944 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:08.934+0000 7f1c44ff9700 1 -- 172.21.15.81:0/218757465 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c38009320 con 0x7f1c480601e0 2023-11-15T06:29:08.944 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:08.934+0000 7f1c44ff9700 1 -- 172.21.15.81:0/218757465 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1c38013f00 con 0x7f1c480601e0 2023-11-15T06:29:08.944 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:08.934+0000 7f1c4cba1700 1 -- 172.21.15.81:0/218757465 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c48106bf0 con 0x7f1c480601e0 2023-11-15T06:29:08.945 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:08.934+0000 7f1c44ff9700 1 -- 172.21.15.81:0/218757465 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c38014d90 con 0x7f1c480601e0 2023-11-15T06:29:08.945 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:08.934+0000 7f1c4cba1700 1 -- 172.21.15.81:0/218757465 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c48107080 con 0x7f1c480601e0 2023-11-15T06:29:08.945 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:08.935+0000 7f1c44ff9700 1 -- 172.21.15.81:0/218757465 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1c380263a0 con 0x7f1c480601e0 2023-11-15T06:29:08.946 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:08.935+0000 7f1c4cba1700 1 -- 172.21.15.81:0/218757465 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c48045ad0 con 0x7f1c480601e0 2023-11-15T06:29:08.946 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:08.935+0000 7f1c44ff9700 1 --2- 172.21.15.81:0/218757465 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f1c3402ff60 0x7f1c34032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:29:08.947 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:08.935+0000 7f1c44ff9700 1 -- 172.21.15.81:0/218757465 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1c380412c0 con 0x7f1c480601e0 2023-11-15T06:29:08.947 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:08.935+0000 7f1c3effd700 1 --2- 172.21.15.81:0/218757465 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f1c3402ff60 0x7f1c34032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:29:08.947 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:08.936+0000 7f1c3effd700 1 --2- 172.21.15.81:0/218757465 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f1c3402ff60 0x7f1c34032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f1c300096a0 tx=0x7f1c30006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:29:08.948 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:08.940+0000 7f1c44ff9700 1 -- 172.21.15.81:0/218757465 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1c38012510 con 0x7f1c480601e0 2023-11-15T06:29:09.147 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:09.146+0000 7f1c4cba1700 1 -- 172.21.15.81:0/218757465 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1c48107f30 con 0x7f1c480601e0 2023-11-15T06:29:09.148 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:09.146+0000 7f1c44ff9700 1 -- 172.21.15.81:0/218757465 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1c38012330 con 0x7f1c480601e0 2023-11-15T06:29:09.149 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:29:09.149 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:29:09.149 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:09.148+0000 7f1c4cba1700 1 -- 172.21.15.81:0/218757465 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f1c3402ff60 msgr2=0x7f1c34032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:09.150 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:09.148+0000 7f1c4cba1700 1 --2- 172.21.15.81:0/218757465 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f1c3402ff60 0x7f1c34032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f1c300096a0 tx=0x7f1c30006b40).stop 2023-11-15T06:29:09.150 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:09.148+0000 7f1c4cba1700 1 -- 172.21.15.81:0/218757465 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c480601e0 msgr2=0x7f1c48106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:09.150 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:09.148+0000 7f1c4cba1700 1 --2- 172.21.15.81:0/218757465 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c480601e0 0x7f1c48106500 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f1c38013510 tx=0x7f1c3800b040).stop 2023-11-15T06:29:09.150 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:09.148+0000 7f1c4cba1700 1 -- 172.21.15.81:0/218757465 shutdown_connections 2023-11-15T06:29:09.151 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:09.148+0000 7f1c4cba1700 1 --2- 172.21.15.81:0/218757465 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f1c3402ff60 0x7f1c34032420 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:09.151 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:09.148+0000 7f1c4cba1700 1 --2- 172.21.15.81:0/218757465 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c480601e0 0x7f1c48106500 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:09.151 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:09.148+0000 7f1c4cba1700 1 -- 172.21.15.81:0/218757465 >> 172.21.15.81:0/218757465 conn(0x7f1c480f09a0 msgr2=0x7f1c480f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:29:09.151 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:09.148+0000 7f1c4cba1700 1 -- 172.21.15.81:0/218757465 shutdown_connections 2023-11-15T06:29:09.151 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:09.148+0000 7f1c4cba1700 1 -- 172.21.15.81:0/218757465 wait complete. 2023-11-15T06:29:09.152 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:29:09.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:09 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:29:09.147088+0000 mon.smithi062 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.81:0/218757465' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:29:10.591 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:29:10.592 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:29:10.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:10 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:29:09.808958+0000 mgr.smithi062.rwurtk (mgr.14164) 99 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:29:11.808 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:11.801+0000 7f4af51bf700 1 -- 172.21.15.81:0/3245155770 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4af00f5410 msgr2=0x7f4af00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:11.808 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:11.801+0000 7f4af51bf700 1 --2- 172.21.15.81:0/3245155770 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4af00f5410 0x7f4af00f5830 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f4ad8009fb0 tx=0x7f4ad8009b30).stop 2023-11-15T06:29:11.808 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:11.801+0000 7f4af51bf700 1 -- 172.21.15.81:0/3245155770 shutdown_connections 2023-11-15T06:29:11.809 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:11.801+0000 7f4af51bf700 1 --2- 172.21.15.81:0/3245155770 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4af00f5410 0x7f4af00f5830 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:11.811 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:11.801+0000 7f4af51bf700 1 -- 172.21.15.81:0/3245155770 >> 172.21.15.81:0/3245155770 conn(0x7f4af00f0b70 msgr2=0x7f4af00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:29:11.811 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:11.801+0000 7f4af51bf700 1 -- 172.21.15.81:0/3245155770 shutdown_connections 2023-11-15T06:29:11.812 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:11.801+0000 7f4af51bf700 1 -- 172.21.15.81:0/3245155770 wait complete. 2023-11-15T06:29:11.812 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:11.802+0000 7f4af51bf700 1 Processor -- start 2023-11-15T06:29:11.812 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:11.802+0000 7f4af51bf700 1 -- start start 2023-11-15T06:29:11.812 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:11.802+0000 7f4af51bf700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4af00f5410 0x7f4af0109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:29:11.812 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:11.802+0000 7f4af51bf700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af0060220 con 0x7f4af00f5410 2023-11-15T06:29:11.813 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:11.802+0000 7f4aeed9d700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4af00f5410 0x7f4af0109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:29:11.813 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:11.802+0000 7f4aeed9d700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4af00f5410 0x7f4af0109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:54660/0 (socket says 172.21.15.81:54660) 2023-11-15T06:29:11.813 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:11.802+0000 7f4aeed9d700 1 -- 172.21.15.81:0/1878369346 learned_addr learned my addr 172.21.15.81:0/1878369346 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:29:11.814 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:11.803+0000 7f4aeed9d700 1 -- 172.21.15.81:0/1878369346 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad8005040 con 0x7f4af00f5410 2023-11-15T06:29:11.814 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:11.803+0000 7f4aeed9d700 1 --2- 172.21.15.81:0/1878369346 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4af00f5410 0x7f4af0109450 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f4ad8013510 tx=0x7f4ad800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:29:11.814 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:11.803+0000 7f4ae77fe700 1 -- 172.21.15.81:0/1878369346 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ad8009320 con 0x7f4af00f5410 2023-11-15T06:29:11.814 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:11.803+0000 7f4ae77fe700 1 -- 172.21.15.81:0/1878369346 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4ad8013f00 con 0x7f4af00f5410 2023-11-15T06:29:11.815 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:11.803+0000 7f4af51bf700 1 -- 172.21.15.81:0/1878369346 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4af0060420 con 0x7f4af00f5410 2023-11-15T06:29:11.815 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:11.803+0000 7f4ae77fe700 1 -- 172.21.15.81:0/1878369346 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ad8014d90 con 0x7f4af00f5410 2023-11-15T06:29:11.815 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:11.803+0000 7f4af51bf700 1 -- 172.21.15.81:0/1878369346 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4af0109b00 con 0x7f4af00f5410 2023-11-15T06:29:11.815 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:11.804+0000 7f4ae77fe700 1 -- 172.21.15.81:0/1878369346 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4ad80095f0 con 0x7f4af00f5410 2023-11-15T06:29:11.815 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:11.804+0000 7f4af51bf700 1 -- 172.21.15.81:0/1878369346 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4af0059100 con 0x7f4af00f5410 2023-11-15T06:29:11.816 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:11.804+0000 7f4ae77fe700 1 --2- 172.21.15.81:0/1878369346 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f4adc02ff60 0x7f4adc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:29:11.816 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:11.804+0000 7f4ae77fe700 1 -- 172.21.15.81:0/1878369346 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4ad80410a0 con 0x7f4af00f5410 2023-11-15T06:29:11.816 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:11.804+0000 7f4aee59c700 1 --2- 172.21.15.81:0/1878369346 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f4adc02ff60 0x7f4adc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:29:11.816 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:11.804+0000 7f4aee59c700 1 --2- 172.21.15.81:0/1878369346 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f4adc02ff60 0x7f4adc032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f4ae00096a0 tx=0x7f4ae0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:29:11.817 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:11.809+0000 7f4ae77fe700 1 -- 172.21.15.81:0/1878369346 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4ad801d7f0 con 0x7f4af00f5410 2023-11-15T06:29:12.012 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:12.010+0000 7f4af51bf700 1 -- 172.21.15.81:0/1878369346 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4af0045ad0 con 0x7f4af00f5410 2023-11-15T06:29:12.013 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:12.011+0000 7f4ae77fe700 1 -- 172.21.15.81:0/1878369346 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4ad8010230 con 0x7f4af00f5410 2023-11-15T06:29:12.013 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:29:12.014 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:29:12.015 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:12.012+0000 7f4af51bf700 1 -- 172.21.15.81:0/1878369346 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f4adc02ff60 msgr2=0x7f4adc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:12.015 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:12.012+0000 7f4af51bf700 1 --2- 172.21.15.81:0/1878369346 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f4adc02ff60 0x7f4adc032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f4ae00096a0 tx=0x7f4ae0006b40).stop 2023-11-15T06:29:12.015 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:12.012+0000 7f4af51bf700 1 -- 172.21.15.81:0/1878369346 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4af00f5410 msgr2=0x7f4af0109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:12.015 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:12.012+0000 7f4af51bf700 1 --2- 172.21.15.81:0/1878369346 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4af00f5410 0x7f4af0109450 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f4ad8013510 tx=0x7f4ad800b040).stop 2023-11-15T06:29:12.016 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:12.012+0000 7f4af51bf700 1 -- 172.21.15.81:0/1878369346 shutdown_connections 2023-11-15T06:29:12.016 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:12.012+0000 7f4af51bf700 1 --2- 172.21.15.81:0/1878369346 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f4adc02ff60 0x7f4adc032420 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:12.016 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:12.012+0000 7f4af51bf700 1 --2- 172.21.15.81:0/1878369346 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4af00f5410 0x7f4af0109450 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:12.016 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:12.012+0000 7f4af51bf700 1 -- 172.21.15.81:0/1878369346 >> 172.21.15.81:0/1878369346 conn(0x7f4af00f0b70 msgr2=0x7f4af00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:29:12.017 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:12.012+0000 7f4af51bf700 1 -- 172.21.15.81:0/1878369346 shutdown_connections 2023-11-15T06:29:12.017 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:12.012+0000 7f4af51bf700 1 -- 172.21.15.81:0/1878369346 wait complete. 2023-11-15T06:29:12.017 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:29:13.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:12 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:29:11.809178+0000 mgr.smithi062.rwurtk (mgr.14164) 100 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:29:13.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:12 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:29:12.011648+0000 mon.smithi062 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.81:0/1878369346' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:29:13.391 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:29:13.392 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:29:14.680 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:14.677+0000 7f3832930700 1 -- 172.21.15.81:0/2199725210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f382c0f78e0 msgr2=0x7f382c0f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:14.680 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:14.677+0000 7f3832930700 1 --2- 172.21.15.81:0/2199725210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f382c0f78e0 0x7f382c0f7d00 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f3828009fb0 tx=0x7f3828009b30).stop 2023-11-15T06:29:14.681 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:14.677+0000 7f3832930700 1 -- 172.21.15.81:0/2199725210 shutdown_connections 2023-11-15T06:29:14.681 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:14.677+0000 7f3832930700 1 --2- 172.21.15.81:0/2199725210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f382c0f78e0 0x7f382c0f7d00 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:14.681 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:14.677+0000 7f3832930700 1 -- 172.21.15.81:0/2199725210 >> 172.21.15.81:0/2199725210 conn(0x7f382c0f0120 msgr2=0x7f382c0f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:29:14.681 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:14.677+0000 7f3832930700 1 -- 172.21.15.81:0/2199725210 shutdown_connections 2023-11-15T06:29:14.681 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:14.677+0000 7f3832930700 1 -- 172.21.15.81:0/2199725210 wait complete. 2023-11-15T06:29:14.682 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:14.677+0000 7f3832930700 1 Processor -- start 2023-11-15T06:29:14.682 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:14.677+0000 7f3832930700 1 -- start start 2023-11-15T06:29:14.682 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:14.677+0000 7f3832930700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f382c0f78e0 0x7f382c1086e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:29:14.682 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:14.677+0000 7f3832930700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f382c108bd0 con 0x7f382c0f78e0 2023-11-15T06:29:14.682 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:14.677+0000 7f383192e700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f382c0f78e0 0x7f382c1086e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:29:14.683 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:14.678+0000 7f383192e700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f382c0f78e0 0x7f382c1086e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:33508/0 (socket says 172.21.15.81:33508) 2023-11-15T06:29:14.683 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:14.678+0000 7f383192e700 1 -- 172.21.15.81:0/3668991486 learned_addr learned my addr 172.21.15.81:0/3668991486 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:29:14.683 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:14.678+0000 7f383192e700 1 -- 172.21.15.81:0/3668991486 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3828005040 con 0x7f382c0f78e0 2023-11-15T06:29:14.683 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:14.678+0000 7f383192e700 1 --2- 172.21.15.81:0/3668991486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f382c0f78e0 0x7f382c1086e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f3828013510 tx=0x7f382800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:29:14.683 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:14.678+0000 7f38227fc700 1 -- 172.21.15.81:0/3668991486 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3828009320 con 0x7f382c0f78e0 2023-11-15T06:29:14.684 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:14.678+0000 7f38227fc700 1 -- 172.21.15.81:0/3668991486 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3828013f00 con 0x7f382c0f78e0 2023-11-15T06:29:14.684 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:14.678+0000 7f3832930700 1 -- 172.21.15.81:0/3668991486 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f382c108dd0 con 0x7f382c0f78e0 2023-11-15T06:29:14.684 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:14.678+0000 7f38227fc700 1 -- 172.21.15.81:0/3668991486 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3828014dc0 con 0x7f382c0f78e0 2023-11-15T06:29:14.684 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:14.678+0000 7f3832930700 1 -- 172.21.15.81:0/3668991486 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f382c109260 con 0x7f382c0f78e0 2023-11-15T06:29:14.686 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:14.679+0000 7f38227fc700 1 -- 172.21.15.81:0/3668991486 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f38280095f0 con 0x7f382c0f78e0 2023-11-15T06:29:14.686 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:14.679+0000 7f3832930700 1 -- 172.21.15.81:0/3668991486 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f382c045ad0 con 0x7f382c0f78e0 2023-11-15T06:29:14.686 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:14.679+0000 7f38227fc700 1 --2- 172.21.15.81:0/3668991486 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f381802ff60 0x7f3818032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:29:14.686 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:14.679+0000 7f38227fc700 1 -- 172.21.15.81:0/3668991486 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f38280410c0 con 0x7f382c0f78e0 2023-11-15T06:29:14.686 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:14.679+0000 7f383112d700 1 --2- 172.21.15.81:0/3668991486 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f381802ff60 0x7f3818032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:29:14.687 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:14.680+0000 7f383112d700 1 --2- 172.21.15.81:0/3668991486 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f381802ff60 0x7f3818032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f381c0096a0 tx=0x7f381c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:29:14.687 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:14.684+0000 7f38227fc700 1 -- 172.21.15.81:0/3668991486 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3828014420 con 0x7f382c0f78e0 2023-11-15T06:29:14.888 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:14.886+0000 7f3832930700 1 -- 172.21.15.81:0/3668991486 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f382c10a110 con 0x7f382c0f78e0 2023-11-15T06:29:14.889 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:14.887+0000 7f38227fc700 1 -- 172.21.15.81:0/3668991486 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3828010230 con 0x7f382c0f78e0 2023-11-15T06:29:14.889 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:29:14.890 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:29:14.891 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:14.889+0000 7f3832930700 1 -- 172.21.15.81:0/3668991486 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f381802ff60 msgr2=0x7f3818032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:14.891 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:14.889+0000 7f3832930700 1 --2- 172.21.15.81:0/3668991486 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f381802ff60 0x7f3818032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f381c0096a0 tx=0x7f381c006b40).stop 2023-11-15T06:29:14.891 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:14.889+0000 7f3832930700 1 -- 172.21.15.81:0/3668991486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f382c0f78e0 msgr2=0x7f382c1086e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:14.891 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:14.889+0000 7f3832930700 1 --2- 172.21.15.81:0/3668991486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f382c0f78e0 0x7f382c1086e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f3828013510 tx=0x7f382800b040).stop 2023-11-15T06:29:14.892 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:14.889+0000 7f3832930700 1 -- 172.21.15.81:0/3668991486 shutdown_connections 2023-11-15T06:29:14.892 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:14.889+0000 7f3832930700 1 --2- 172.21.15.81:0/3668991486 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f381802ff60 0x7f3818032420 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:14.892 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:14.889+0000 7f3832930700 1 --2- 172.21.15.81:0/3668991486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f382c0f78e0 0x7f382c1086e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:14.892 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:14.889+0000 7f3832930700 1 -- 172.21.15.81:0/3668991486 >> 172.21.15.81:0/3668991486 conn(0x7f382c0f0120 msgr2=0x7f382c0f0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:29:14.892 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:14.889+0000 7f3832930700 1 -- 172.21.15.81:0/3668991486 shutdown_connections 2023-11-15T06:29:14.893 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:14.889+0000 7f3832930700 1 -- 172.21.15.81:0/3668991486 wait complete. 2023-11-15T06:29:14.893 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:29:15.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:14 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:29:13.809481+0000 mgr.smithi062.rwurtk (mgr.14164) 101 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:29:16.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:15 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:29:14.887835+0000 mon.smithi062 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.81:0/3668991486' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:29:16.226 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:29:16.227 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:29:17.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:16 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:29:15.809811+0000 mgr.smithi062.rwurtk (mgr.14164) 102 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:29:17.441 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:17.435+0000 7f22bfeb3700 1 -- 172.21.15.81:0/1103491239 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22b80f5410 msgr2=0x7f22b80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:17.441 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:17.435+0000 7f22bfeb3700 1 --2- 172.21.15.81:0/1103491239 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22b80f5410 0x7f22b80f5830 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f22ac009fb0 tx=0x7f22ac009b30).stop 2023-11-15T06:29:17.441 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:17.436+0000 7f22bfeb3700 1 -- 172.21.15.81:0/1103491239 shutdown_connections 2023-11-15T06:29:17.442 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:17.436+0000 7f22bfeb3700 1 --2- 172.21.15.81:0/1103491239 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22b80f5410 0x7f22b80f5830 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:17.442 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:17.436+0000 7f22bfeb3700 1 -- 172.21.15.81:0/1103491239 >> 172.21.15.81:0/1103491239 conn(0x7f22b80f0b70 msgr2=0x7f22b80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:29:17.442 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:17.436+0000 7f22bfeb3700 1 -- 172.21.15.81:0/1103491239 shutdown_connections 2023-11-15T06:29:17.443 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:17.436+0000 7f22bfeb3700 1 -- 172.21.15.81:0/1103491239 wait complete. 2023-11-15T06:29:17.443 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:17.436+0000 7f22bfeb3700 1 Processor -- start 2023-11-15T06:29:17.443 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:17.436+0000 7f22bfeb3700 1 -- start start 2023-11-15T06:29:17.443 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:17.436+0000 7f22bfeb3700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22b80f5410 0x7f22b8109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:29:17.443 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:17.436+0000 7f22bfeb3700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22b8060220 con 0x7f22b80f5410 2023-11-15T06:29:17.444 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:17.436+0000 7f22bdc4f700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22b80f5410 0x7f22b8109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:29:17.444 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:17.436+0000 7f22bdc4f700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22b80f5410 0x7f22b8109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:33532/0 (socket says 172.21.15.81:33532) 2023-11-15T06:29:17.444 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:17.437+0000 7f22bdc4f700 1 -- 172.21.15.81:0/3495970657 learned_addr learned my addr 172.21.15.81:0/3495970657 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:29:17.444 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:17.437+0000 7f22bdc4f700 1 -- 172.21.15.81:0/3495970657 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22ac005040 con 0x7f22b80f5410 2023-11-15T06:29:17.445 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:17.437+0000 7f22bdc4f700 1 --2- 172.21.15.81:0/3495970657 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22b80f5410 0x7f22b8109450 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f22ac013510 tx=0x7f22ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:29:17.445 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:17.437+0000 7f22aa7fc700 1 -- 172.21.15.81:0/3495970657 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22ac009320 con 0x7f22b80f5410 2023-11-15T06:29:17.445 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:17.437+0000 7f22bfeb3700 1 -- 172.21.15.81:0/3495970657 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22b8060420 con 0x7f22b80f5410 2023-11-15T06:29:17.445 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:17.437+0000 7f22aa7fc700 1 -- 172.21.15.81:0/3495970657 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f22ac013f00 con 0x7f22b80f5410 2023-11-15T06:29:17.446 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:17.437+0000 7f22aa7fc700 1 -- 172.21.15.81:0/3495970657 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22ac014d90 con 0x7f22b80f5410 2023-11-15T06:29:17.446 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:17.437+0000 7f22bfeb3700 1 -- 172.21.15.81:0/3495970657 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22b8109b00 con 0x7f22b80f5410 2023-11-15T06:29:17.446 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:17.438+0000 7f22aa7fc700 1 -- 172.21.15.81:0/3495970657 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f22ac0095f0 con 0x7f22b80f5410 2023-11-15T06:29:17.446 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:17.438+0000 7f22bfeb3700 1 -- 172.21.15.81:0/3495970657 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22b8059100 con 0x7f22b80f5410 2023-11-15T06:29:17.447 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:17.438+0000 7f22aa7fc700 1 --2- 172.21.15.81:0/3495970657 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f22a402ffb0 0x7f22a4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:29:17.447 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:17.438+0000 7f22aa7fc700 1 -- 172.21.15.81:0/3495970657 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f22ac0410c0 con 0x7f22b80f5410 2023-11-15T06:29:17.448 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:17.438+0000 7f22bd44e700 1 --2- 172.21.15.81:0/3495970657 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f22a402ffb0 0x7f22a4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:29:17.448 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:17.439+0000 7f22bd44e700 1 --2- 172.21.15.81:0/3495970657 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f22a402ffb0 0x7f22a4032470 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f22b40096a0 tx=0x7f22b4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:29:17.449 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:17.443+0000 7f22aa7fc700 1 -- 172.21.15.81:0/3495970657 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f22ac01d7f0 con 0x7f22b80f5410 2023-11-15T06:29:17.648 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:17.646+0000 7f22bfeb3700 1 -- 172.21.15.81:0/3495970657 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f22b8045ad0 con 0x7f22b80f5410 2023-11-15T06:29:17.649 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:17.647+0000 7f22aa7fc700 1 -- 172.21.15.81:0/3495970657 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f22ac010230 con 0x7f22b80f5410 2023-11-15T06:29:17.649 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:29:17.650 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:29:17.650 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:17.649+0000 7f22bfeb3700 1 -- 172.21.15.81:0/3495970657 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f22a402ffb0 msgr2=0x7f22a4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:17.651 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:17.649+0000 7f22bfeb3700 1 --2- 172.21.15.81:0/3495970657 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f22a402ffb0 0x7f22a4032470 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f22b40096a0 tx=0x7f22b4006b40).stop 2023-11-15T06:29:17.651 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:17.649+0000 7f22bfeb3700 1 -- 172.21.15.81:0/3495970657 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22b80f5410 msgr2=0x7f22b8109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:17.651 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:17.649+0000 7f22bfeb3700 1 --2- 172.21.15.81:0/3495970657 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22b80f5410 0x7f22b8109450 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f22ac013510 tx=0x7f22ac00b040).stop 2023-11-15T06:29:17.651 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:17.649+0000 7f22bfeb3700 1 -- 172.21.15.81:0/3495970657 shutdown_connections 2023-11-15T06:29:17.652 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:17.649+0000 7f22bfeb3700 1 --2- 172.21.15.81:0/3495970657 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f22a402ffb0 0x7f22a4032470 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:17.652 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:17.649+0000 7f22bfeb3700 1 --2- 172.21.15.81:0/3495970657 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22b80f5410 0x7f22b8109450 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:17.652 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:17.649+0000 7f22bfeb3700 1 -- 172.21.15.81:0/3495970657 >> 172.21.15.81:0/3495970657 conn(0x7f22b80f0b70 msgr2=0x7f22b80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:29:17.652 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:17.649+0000 7f22bfeb3700 1 -- 172.21.15.81:0/3495970657 shutdown_connections 2023-11-15T06:29:17.652 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:17.649+0000 7f22bfeb3700 1 -- 172.21.15.81:0/3495970657 wait complete. 2023-11-15T06:29:17.653 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:29:18.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:29:17.648049+0000 mon.smithi062 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.81:0/3495970657' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:29:18.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:29:17.810148+0000 mgr.smithi062.rwurtk (mgr.14164) 103 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:29:19.081 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:29:19.082 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:29:20.344 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:20.341+0000 7f9f4d41e700 1 -- 172.21.15.81:0/2055885600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f480f62a0 msgr2=0x7f9f480f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:20.345 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:20.341+0000 7f9f4d41e700 1 --2- 172.21.15.81:0/2055885600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f480f62a0 0x7f9f480f66c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f9f38009fb0 tx=0x7f9f38009b30).stop 2023-11-15T06:29:20.345 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:20.341+0000 7f9f4d41e700 1 -- 172.21.15.81:0/2055885600 shutdown_connections 2023-11-15T06:29:20.345 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:20.341+0000 7f9f4d41e700 1 --2- 172.21.15.81:0/2055885600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f480f62a0 0x7f9f480f66c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:20.345 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:20.341+0000 7f9f4d41e700 1 -- 172.21.15.81:0/2055885600 >> 172.21.15.81:0/2055885600 conn(0x7f9f480f1aa0 msgr2=0x7f9f480f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:29:20.345 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:20.341+0000 7f9f4d41e700 1 -- 172.21.15.81:0/2055885600 shutdown_connections 2023-11-15T06:29:20.346 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:20.342+0000 7f9f4d41e700 1 -- 172.21.15.81:0/2055885600 wait complete. 2023-11-15T06:29:20.346 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:20.342+0000 7f9f4d41e700 1 Processor -- start 2023-11-15T06:29:20.346 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:20.342+0000 7f9f4d41e700 1 -- start start 2023-11-15T06:29:20.346 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:20.342+0000 7f9f4d41e700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f480f62a0 0x7f9f481064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:29:20.347 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:20.342+0000 7f9f4d41e700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f481069a0 con 0x7f9f480f62a0 2023-11-15T06:29:20.347 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:20.342+0000 7f9f47fff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f480f62a0 0x7f9f481064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:29:20.347 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:20.343+0000 7f9f47fff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f480f62a0 0x7f9f481064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:33550/0 (socket says 172.21.15.81:33550) 2023-11-15T06:29:20.347 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:20.343+0000 7f9f47fff700 1 -- 172.21.15.81:0/1969915182 learned_addr learned my addr 172.21.15.81:0/1969915182 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:29:20.347 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:20.343+0000 7f9f47fff700 1 -- 172.21.15.81:0/1969915182 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f38005040 con 0x7f9f480f62a0 2023-11-15T06:29:20.349 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:20.343+0000 7f9f47fff700 1 --2- 172.21.15.81:0/1969915182 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f480f62a0 0x7f9f481064b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f9f38013510 tx=0x7f9f3800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:29:20.349 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:20.343+0000 7f9f44ff9700 1 -- 172.21.15.81:0/1969915182 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f38009320 con 0x7f9f480f62a0 2023-11-15T06:29:20.349 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:20.343+0000 7f9f44ff9700 1 -- 172.21.15.81:0/1969915182 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9f38013f00 con 0x7f9f480f62a0 2023-11-15T06:29:20.349 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:20.343+0000 7f9f4d41e700 1 -- 172.21.15.81:0/1969915182 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f48106ba0 con 0x7f9f480f62a0 2023-11-15T06:29:20.350 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:20.343+0000 7f9f44ff9700 1 -- 172.21.15.81:0/1969915182 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f38014dc0 con 0x7f9f480f62a0 2023-11-15T06:29:20.350 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:20.343+0000 7f9f4d41e700 1 -- 172.21.15.81:0/1969915182 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f48107030 con 0x7f9f480f62a0 2023-11-15T06:29:20.350 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:20.344+0000 7f9f44ff9700 1 -- 172.21.15.81:0/1969915182 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9f380095f0 con 0x7f9f480f62a0 2023-11-15T06:29:20.350 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:20.344+0000 7f9f4d41e700 1 -- 172.21.15.81:0/1969915182 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f48045ad0 con 0x7f9f480f62a0 2023-11-15T06:29:20.350 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:20.344+0000 7f9f44ff9700 1 --2- 172.21.15.81:0/1969915182 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f9f3002ff60 0x7f9f30032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:29:20.351 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:20.344+0000 7f9f44ff9700 1 -- 172.21.15.81:0/1969915182 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9f38041000 con 0x7f9f480f62a0 2023-11-15T06:29:20.351 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:20.344+0000 7f9f477fe700 1 --2- 172.21.15.81:0/1969915182 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f9f3002ff60 0x7f9f30032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:29:20.351 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:20.345+0000 7f9f477fe700 1 --2- 172.21.15.81:0/1969915182 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f9f3002ff60 0x7f9f30032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f9f3c0096a0 tx=0x7f9f3c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:29:20.352 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:20.349+0000 7f9f44ff9700 1 -- 172.21.15.81:0/1969915182 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9f3801c790 con 0x7f9f480f62a0 2023-11-15T06:29:20.555 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:20.553+0000 7f9f4d41e700 1 -- 172.21.15.81:0/1969915182 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9f48107ee0 con 0x7f9f480f62a0 2023-11-15T06:29:20.555 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:20.554+0000 7f9f44ff9700 1 -- 172.21.15.81:0/1969915182 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9f38010230 con 0x7f9f480f62a0 2023-11-15T06:29:20.556 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:29:20.556 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:29:20.557 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:20.555+0000 7f9f4d41e700 1 -- 172.21.15.81:0/1969915182 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f9f3002ff60 msgr2=0x7f9f30032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:20.557 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:20.555+0000 7f9f4d41e700 1 --2- 172.21.15.81:0/1969915182 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f9f3002ff60 0x7f9f30032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f9f3c0096a0 tx=0x7f9f3c006b40).stop 2023-11-15T06:29:20.558 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:20.555+0000 7f9f4d41e700 1 -- 172.21.15.81:0/1969915182 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f480f62a0 msgr2=0x7f9f481064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:20.558 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:20.555+0000 7f9f4d41e700 1 --2- 172.21.15.81:0/1969915182 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f480f62a0 0x7f9f481064b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f9f38013510 tx=0x7f9f3800b040).stop 2023-11-15T06:29:20.558 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:20.556+0000 7f9f4d41e700 1 -- 172.21.15.81:0/1969915182 shutdown_connections 2023-11-15T06:29:20.558 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:20.556+0000 7f9f4d41e700 1 --2- 172.21.15.81:0/1969915182 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f9f3002ff60 0x7f9f30032420 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:20.559 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:20.556+0000 7f9f4d41e700 1 --2- 172.21.15.81:0/1969915182 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f480f62a0 0x7f9f481064b0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:20.559 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:20.556+0000 7f9f4d41e700 1 -- 172.21.15.81:0/1969915182 >> 172.21.15.81:0/1969915182 conn(0x7f9f480f1aa0 msgr2=0x7f9f480f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:29:20.559 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:20.556+0000 7f9f4d41e700 1 -- 172.21.15.81:0/1969915182 shutdown_connections 2023-11-15T06:29:20.559 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:20.556+0000 7f9f4d41e700 1 -- 172.21.15.81:0/1969915182 wait complete. 2023-11-15T06:29:20.560 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:29:21.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:20 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:29:19.810436+0000 mgr.smithi062.rwurtk (mgr.14164) 104 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:29:21.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:20 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:29:20.554713+0000 mon.smithi062 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.81:0/1969915182' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:29:21.968 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:29:21.969 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:29:23.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:22 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:29:21.810761+0000 mgr.smithi062.rwurtk (mgr.14164) 105 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:29:23.171 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:23.169+0000 7f003500a700 1 -- 172.21.15.81:0/1263959291 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0030060160 msgr2=0x7f00300fb3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:23.171 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:23.169+0000 7f003500a700 1 --2- 172.21.15.81:0/1263959291 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0030060160 0x7f00300fb3a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f0020009fb0 tx=0x7f0020009b30).stop 2023-11-15T06:29:23.171 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:23.169+0000 7f003500a700 1 -- 172.21.15.81:0/1263959291 shutdown_connections 2023-11-15T06:29:23.171 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:23.169+0000 7f003500a700 1 --2- 172.21.15.81:0/1263959291 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0030060160 0x7f00300fb3a0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:23.172 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:23.169+0000 7f003500a700 1 -- 172.21.15.81:0/1263959291 >> 172.21.15.81:0/1263959291 conn(0x7f00300f09a0 msgr2=0x7f00300f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:29:23.172 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:23.169+0000 7f003500a700 1 -- 172.21.15.81:0/1263959291 shutdown_connections 2023-11-15T06:29:23.172 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:23.169+0000 7f003500a700 1 -- 172.21.15.81:0/1263959291 wait complete. 2023-11-15T06:29:23.172 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:23.169+0000 7f003500a700 1 Processor -- start 2023-11-15T06:29:23.174 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:23.169+0000 7f003500a700 1 -- start start 2023-11-15T06:29:23.175 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:23.170+0000 7f003500a700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0030060160 0x7f00301086f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:29:23.175 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:23.170+0000 7f003500a700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0030108be0 con 0x7f0030060160 2023-11-15T06:29:23.175 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:23.170+0000 7f002ffff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0030060160 0x7f00301086f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:29:23.175 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:23.170+0000 7f002ffff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0030060160 0x7f00301086f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:57090/0 (socket says 172.21.15.81:57090) 2023-11-15T06:29:23.175 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:23.170+0000 7f002ffff700 1 -- 172.21.15.81:0/1110276668 learned_addr learned my addr 172.21.15.81:0/1110276668 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:29:23.176 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:23.170+0000 7f002ffff700 1 -- 172.21.15.81:0/1110276668 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0020005040 con 0x7f0030060160 2023-11-15T06:29:23.176 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:23.170+0000 7f002ffff700 1 --2- 172.21.15.81:0/1110276668 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0030060160 0x7f00301086f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f0020013510 tx=0x7f002000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:29:23.176 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:23.171+0000 7f002cff9700 1 -- 172.21.15.81:0/1110276668 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0020009320 con 0x7f0030060160 2023-11-15T06:29:23.177 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:23.171+0000 7f003500a700 1 -- 172.21.15.81:0/1110276668 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0030108de0 con 0x7f0030060160 2023-11-15T06:29:23.177 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:23.171+0000 7f003500a700 1 -- 172.21.15.81:0/1110276668 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0030109270 con 0x7f0030060160 2023-11-15T06:29:23.177 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:23.171+0000 7f002cff9700 1 -- 172.21.15.81:0/1110276668 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0020013f00 con 0x7f0030060160 2023-11-15T06:29:23.177 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:23.171+0000 7f002cff9700 1 -- 172.21.15.81:0/1110276668 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0020014d60 con 0x7f0030060160 2023-11-15T06:29:23.178 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:23.172+0000 7f002cff9700 1 -- 172.21.15.81:0/1110276668 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f002001e420 con 0x7f0030060160 2023-11-15T06:29:23.178 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:23.172+0000 7f003500a700 1 -- 172.21.15.81:0/1110276668 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0010004fa0 con 0x7f0030060160 2023-11-15T06:29:23.178 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:23.172+0000 7f002cff9700 1 --2- 172.21.15.81:0/1110276668 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f001802ff60 0x7f0018032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:29:23.178 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:23.172+0000 7f002cff9700 1 -- 172.21.15.81:0/1110276668 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0020041f30 con 0x7f0030060160 2023-11-15T06:29:23.178 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:23.172+0000 7f002f7fe700 1 --2- 172.21.15.81:0/1110276668 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f001802ff60 0x7f0018032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:29:23.179 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:23.173+0000 7f002f7fe700 1 --2- 172.21.15.81:0/1110276668 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f001802ff60 0x7f0018032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f00240096a0 tx=0x7f0024006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:29:23.179 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:23.177+0000 7f002cff9700 1 -- 172.21.15.81:0/1110276668 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f00200263a0 con 0x7f0030060160 2023-11-15T06:29:23.387 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:23.385+0000 7f003500a700 1 -- 172.21.15.81:0/1110276668 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0010005670 con 0x7f0030060160 2023-11-15T06:29:23.388 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:23.386+0000 7f002cff9700 1 -- 172.21.15.81:0/1110276668 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f002001a230 con 0x7f0030060160 2023-11-15T06:29:23.388 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:29:23.388 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:29:23.390 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:23.387+0000 7f003500a700 1 -- 172.21.15.81:0/1110276668 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f001802ff60 msgr2=0x7f0018032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:23.390 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:23.387+0000 7f003500a700 1 --2- 172.21.15.81:0/1110276668 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f001802ff60 0x7f0018032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f00240096a0 tx=0x7f0024006b40).stop 2023-11-15T06:29:23.390 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:23.387+0000 7f003500a700 1 -- 172.21.15.81:0/1110276668 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0030060160 msgr2=0x7f00301086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:23.390 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:23.387+0000 7f003500a700 1 --2- 172.21.15.81:0/1110276668 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0030060160 0x7f00301086f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f0020013510 tx=0x7f002000b040).stop 2023-11-15T06:29:23.391 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:23.387+0000 7f003500a700 1 -- 172.21.15.81:0/1110276668 shutdown_connections 2023-11-15T06:29:23.391 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:23.387+0000 7f003500a700 1 --2- 172.21.15.81:0/1110276668 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f001802ff60 0x7f0018032420 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:23.391 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:23.387+0000 7f003500a700 1 --2- 172.21.15.81:0/1110276668 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0030060160 0x7f00301086f0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:23.391 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:23.387+0000 7f003500a700 1 -- 172.21.15.81:0/1110276668 >> 172.21.15.81:0/1110276668 conn(0x7f00300f09a0 msgr2=0x7f00300f14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:29:23.392 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:23.387+0000 7f003500a700 1 -- 172.21.15.81:0/1110276668 shutdown_connections 2023-11-15T06:29:23.392 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:23.387+0000 7f003500a700 1 -- 172.21.15.81:0/1110276668 wait complete. 2023-11-15T06:29:23.392 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:29:24.169 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:23 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:29:23.386500+0000 mon.smithi062 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.81:0/1110276668' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:29:24.857 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:29:24.858 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:29:25.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:24 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:29:23.811133+0000 mgr.smithi062.rwurtk (mgr.14164) 106 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:29:26.077 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:26.074+0000 7f5e69664700 1 -- 172.21.15.81:0/4043136252 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e640f5410 msgr2=0x7f5e640f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:26.078 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:26.074+0000 7f5e69664700 1 --2- 172.21.15.81:0/4043136252 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e640f5410 0x7f5e640f5830 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f5e58009fb0 tx=0x7f5e58009b30).stop 2023-11-15T06:29:26.078 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:26.074+0000 7f5e69664700 1 -- 172.21.15.81:0/4043136252 shutdown_connections 2023-11-15T06:29:26.078 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:26.074+0000 7f5e69664700 1 --2- 172.21.15.81:0/4043136252 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e640f5410 0x7f5e640f5830 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:26.079 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:26.074+0000 7f5e69664700 1 -- 172.21.15.81:0/4043136252 >> 172.21.15.81:0/4043136252 conn(0x7f5e640f0b70 msgr2=0x7f5e640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:29:26.079 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:26.074+0000 7f5e69664700 1 -- 172.21.15.81:0/4043136252 shutdown_connections 2023-11-15T06:29:26.079 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:26.074+0000 7f5e69664700 1 -- 172.21.15.81:0/4043136252 wait complete. 2023-11-15T06:29:26.079 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:26.075+0000 7f5e69664700 1 Processor -- start 2023-11-15T06:29:26.079 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:26.075+0000 7f5e69664700 1 -- start start 2023-11-15T06:29:26.080 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:26.075+0000 7f5e69664700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e640f5410 0x7f5e64106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:29:26.080 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:26.075+0000 7f5e69664700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e64106c00 con 0x7f5e640f5410 2023-11-15T06:29:26.080 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:26.075+0000 7f5e62ffd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e640f5410 0x7f5e64106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:29:26.080 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:26.075+0000 7f5e62ffd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e640f5410 0x7f5e64106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:57106/0 (socket says 172.21.15.81:57106) 2023-11-15T06:29:26.080 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:26.075+0000 7f5e62ffd700 1 -- 172.21.15.81:0/652008662 learned_addr learned my addr 172.21.15.81:0/652008662 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:29:26.081 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:26.075+0000 7f5e62ffd700 1 -- 172.21.15.81:0/652008662 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e58005040 con 0x7f5e640f5410 2023-11-15T06:29:26.081 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:26.075+0000 7f5e62ffd700 1 --2- 172.21.15.81:0/652008662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e640f5410 0x7f5e64106710 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f5e58013510 tx=0x7f5e5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:29:26.081 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:26.076+0000 7f5e4b7fe700 1 -- 172.21.15.81:0/652008662 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e58009320 con 0x7f5e640f5410 2023-11-15T06:29:26.081 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:26.076+0000 7f5e69664700 1 -- 172.21.15.81:0/652008662 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e64106e00 con 0x7f5e640f5410 2023-11-15T06:29:26.081 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:26.076+0000 7f5e4b7fe700 1 -- 172.21.15.81:0/652008662 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e58013f00 con 0x7f5e640f5410 2023-11-15T06:29:26.081 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:26.076+0000 7f5e4b7fe700 1 -- 172.21.15.81:0/652008662 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e58014d90 con 0x7f5e640f5410 2023-11-15T06:29:26.082 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:26.076+0000 7f5e69664700 1 -- 172.21.15.81:0/652008662 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e64107290 con 0x7f5e640f5410 2023-11-15T06:29:26.083 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:26.076+0000 7f5e4b7fe700 1 -- 172.21.15.81:0/652008662 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5e58014420 con 0x7f5e640f5410 2023-11-15T06:29:26.083 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:26.077+0000 7f5e4b7fe700 1 --2- 172.21.15.81:0/652008662 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f5e4c02ff60 0x7f5e4c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:29:26.083 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:26.077+0000 7f5e4b7fe700 1 -- 172.21.15.81:0/652008662 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5e580410c0 con 0x7f5e640f5410 2023-11-15T06:29:26.083 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:26.077+0000 7f5e69664700 1 -- 172.21.15.81:0/652008662 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e64045ad0 con 0x7f5e640f5410 2023-11-15T06:29:26.084 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:26.077+0000 7f5e627fc700 1 --2- 172.21.15.81:0/652008662 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f5e4c02ff60 0x7f5e4c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:29:26.084 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:26.077+0000 7f5e627fc700 1 --2- 172.21.15.81:0/652008662 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f5e4c02ff60 0x7f5e4c032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f5e540096a0 tx=0x7f5e54006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:29:26.084 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:26.082+0000 7f5e4b7fe700 1 -- 172.21.15.81:0/652008662 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5e580095f0 con 0x7f5e640f5410 2023-11-15T06:29:26.287 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:26.285+0000 7f5e69664700 1 -- 172.21.15.81:0/652008662 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5e640faf50 con 0x7f5e640f5410 2023-11-15T06:29:26.287 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:26.286+0000 7f5e4b7fe700 1 -- 172.21.15.81:0/652008662 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5e5801db70 con 0x7f5e640f5410 2023-11-15T06:29:26.288 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:29:26.288 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:29:26.290 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:26.288+0000 7f5e69664700 1 -- 172.21.15.81:0/652008662 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f5e4c02ff60 msgr2=0x7f5e4c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:26.290 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:26.288+0000 7f5e69664700 1 --2- 172.21.15.81:0/652008662 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f5e4c02ff60 0x7f5e4c032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f5e540096a0 tx=0x7f5e54006b40).stop 2023-11-15T06:29:26.290 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:26.288+0000 7f5e69664700 1 -- 172.21.15.81:0/652008662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e640f5410 msgr2=0x7f5e64106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:26.290 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:26.288+0000 7f5e69664700 1 --2- 172.21.15.81:0/652008662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e640f5410 0x7f5e64106710 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f5e58013510 tx=0x7f5e5800b040).stop 2023-11-15T06:29:26.290 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:26.288+0000 7f5e69664700 1 -- 172.21.15.81:0/652008662 shutdown_connections 2023-11-15T06:29:26.291 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:26.288+0000 7f5e69664700 1 --2- 172.21.15.81:0/652008662 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f5e4c02ff60 0x7f5e4c032420 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:26.291 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:26.288+0000 7f5e69664700 1 --2- 172.21.15.81:0/652008662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e640f5410 0x7f5e64106710 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:26.291 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:26.288+0000 7f5e69664700 1 -- 172.21.15.81:0/652008662 >> 172.21.15.81:0/652008662 conn(0x7f5e640f0b70 msgr2=0x7f5e6405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:29:26.291 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:26.288+0000 7f5e69664700 1 -- 172.21.15.81:0/652008662 shutdown_connections 2023-11-15T06:29:26.291 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:26.288+0000 7f5e69664700 1 -- 172.21.15.81:0/652008662 wait complete. 2023-11-15T06:29:26.292 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:29:27.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:26 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:29:25.811471+0000 mgr.smithi062.rwurtk (mgr.14164) 107 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:29:27.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:26 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:29:26.286499+0000 mon.smithi062 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.81:0/652008662' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:29:27.686 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:29:27.686 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:29:28.884 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:28.882+0000 7f1b3f075700 1 -- 172.21.15.81:0/1436050769 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b380f6510 msgr2=0x7f1b380f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:28.884 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:28.882+0000 7f1b3f075700 1 --2- 172.21.15.81:0/1436050769 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b380f6510 0x7f1b380f6930 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f1b2c009fb0 tx=0x7f1b2c009b30).stop 2023-11-15T06:29:28.885 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:28.882+0000 7f1b3f075700 1 -- 172.21.15.81:0/1436050769 shutdown_connections 2023-11-15T06:29:28.885 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:28.882+0000 7f1b3f075700 1 --2- 172.21.15.81:0/1436050769 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b380f6510 0x7f1b380f6930 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:28.885 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:28.882+0000 7f1b3f075700 1 -- 172.21.15.81:0/1436050769 >> 172.21.15.81:0/1436050769 conn(0x7f1b380f1c70 msgr2=0x7f1b380f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:29:28.885 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:28.882+0000 7f1b3f075700 1 -- 172.21.15.81:0/1436050769 shutdown_connections 2023-11-15T06:29:28.885 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:28.882+0000 7f1b3f075700 1 -- 172.21.15.81:0/1436050769 wait complete. 2023-11-15T06:29:28.887 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:28.883+0000 7f1b3f075700 1 Processor -- start 2023-11-15T06:29:28.887 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:28.883+0000 7f1b3f075700 1 -- start start 2023-11-15T06:29:28.888 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:28.883+0000 7f1b3f075700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b380f6510 0x7f1b38106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:29:28.888 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:28.883+0000 7f1b3f075700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b38106c20 con 0x7f1b380f6510 2023-11-15T06:29:28.888 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:28.883+0000 7f1b3ce11700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b380f6510 0x7f1b38106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:29:28.888 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:28.883+0000 7f1b3ce11700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b380f6510 0x7f1b38106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:57126/0 (socket says 172.21.15.81:57126) 2023-11-15T06:29:28.889 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:28.883+0000 7f1b3ce11700 1 -- 172.21.15.81:0/473828586 learned_addr learned my addr 172.21.15.81:0/473828586 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:29:28.889 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:28.884+0000 7f1b3ce11700 1 -- 172.21.15.81:0/473828586 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b2c005040 con 0x7f1b380f6510 2023-11-15T06:29:28.889 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:28.884+0000 7f1b3ce11700 1 --2- 172.21.15.81:0/473828586 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b380f6510 0x7f1b38106730 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f1b2c013510 tx=0x7f1b2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:29:28.889 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:28.884+0000 7f1b357fa700 1 -- 172.21.15.81:0/473828586 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b2c009320 con 0x7f1b380f6510 2023-11-15T06:29:28.889 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:28.884+0000 7f1b3f075700 1 -- 172.21.15.81:0/473828586 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b38106e20 con 0x7f1b380f6510 2023-11-15T06:29:28.889 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:28.884+0000 7f1b357fa700 1 -- 172.21.15.81:0/473828586 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1b2c013f00 con 0x7f1b380f6510 2023-11-15T06:29:28.890 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:28.884+0000 7f1b357fa700 1 -- 172.21.15.81:0/473828586 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b2c014d90 con 0x7f1b380f6510 2023-11-15T06:29:28.890 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:28.884+0000 7f1b3f075700 1 -- 172.21.15.81:0/473828586 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b381072b0 con 0x7f1b380f6510 2023-11-15T06:29:28.890 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:28.885+0000 7f1b357fa700 1 -- 172.21.15.81:0/473828586 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1b2c0263a0 con 0x7f1b380f6510 2023-11-15T06:29:28.891 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:28.885+0000 7f1b357fa700 1 --2- 172.21.15.81:0/473828586 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f1b2802ff60 0x7f1b28032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:29:28.892 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:28.885+0000 7f1b357fa700 1 -- 172.21.15.81:0/473828586 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1b2c041300 con 0x7f1b380f6510 2023-11-15T06:29:28.892 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:28.885+0000 7f1b3f075700 1 -- 172.21.15.81:0/473828586 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b38045ad0 con 0x7f1b380f6510 2023-11-15T06:29:28.892 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:28.885+0000 7f1b37fff700 1 --2- 172.21.15.81:0/473828586 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f1b2802ff60 0x7f1b28032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:29:28.893 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:28.886+0000 7f1b37fff700 1 --2- 172.21.15.81:0/473828586 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f1b2802ff60 0x7f1b28032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f1b240096a0 tx=0x7f1b24006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:29:28.893 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:28.890+0000 7f1b357fa700 1 -- 172.21.15.81:0/473828586 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1b2c012920 con 0x7f1b380f6510 2023-11-15T06:29:28.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:28 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:29:27.811836+0000 mgr.smithi062.rwurtk (mgr.14164) 108 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:29:29.096 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:29.094+0000 7f1b3f075700 1 -- 172.21.15.81:0/473828586 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1b380fc050 con 0x7f1b380f6510 2023-11-15T06:29:29.097 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:29.094+0000 7f1b357fa700 1 -- 172.21.15.81:0/473828586 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1b2c02b030 con 0x7f1b380f6510 2023-11-15T06:29:29.097 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:29:29.097 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:29:29.099 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:29.096+0000 7f1b3f075700 1 -- 172.21.15.81:0/473828586 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f1b2802ff60 msgr2=0x7f1b28032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:29.099 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:29.096+0000 7f1b3f075700 1 --2- 172.21.15.81:0/473828586 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f1b2802ff60 0x7f1b28032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f1b240096a0 tx=0x7f1b24006b40).stop 2023-11-15T06:29:29.099 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:29.096+0000 7f1b3f075700 1 -- 172.21.15.81:0/473828586 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b380f6510 msgr2=0x7f1b38106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:29.099 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:29.096+0000 7f1b3f075700 1 --2- 172.21.15.81:0/473828586 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b380f6510 0x7f1b38106730 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f1b2c013510 tx=0x7f1b2c00b040).stop 2023-11-15T06:29:29.100 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:29.096+0000 7f1b3f075700 1 -- 172.21.15.81:0/473828586 shutdown_connections 2023-11-15T06:29:29.100 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:29.096+0000 7f1b3f075700 1 --2- 172.21.15.81:0/473828586 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f1b2802ff60 0x7f1b28032420 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:29.100 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:29.096+0000 7f1b3f075700 1 --2- 172.21.15.81:0/473828586 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b380f6510 0x7f1b38106730 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:29.100 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:29.096+0000 7f1b3f075700 1 -- 172.21.15.81:0/473828586 >> 172.21.15.81:0/473828586 conn(0x7f1b380f1c70 msgr2=0x7f1b380f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:29:29.100 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:29.096+0000 7f1b3f075700 1 -- 172.21.15.81:0/473828586 shutdown_connections 2023-11-15T06:29:29.100 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:29.096+0000 7f1b3f075700 1 -- 172.21.15.81:0/473828586 wait complete. 2023-11-15T06:29:29.101 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:29:29.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:29 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:29:29.095155+0000 mon.smithi062 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.81:0/473828586' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:29:30.516 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:29:30.517 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:29:30.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:30 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:29:29.812151+0000 mgr.smithi062.rwurtk (mgr.14164) 109 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:29:31.682 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:31.678+0000 7f6c6bfa8700 1 -- 172.21.15.81:0/4078262934 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c640f6510 msgr2=0x7f6c640f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:31.682 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:31.678+0000 7f6c6bfa8700 1 --2- 172.21.15.81:0/4078262934 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c640f6510 0x7f6c640f6930 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f6c54009fb0 tx=0x7f6c54009b30).stop 2023-11-15T06:29:31.683 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:31.678+0000 7f6c6bfa8700 1 -- 172.21.15.81:0/4078262934 shutdown_connections 2023-11-15T06:29:31.683 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:31.678+0000 7f6c6bfa8700 1 --2- 172.21.15.81:0/4078262934 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c640f6510 0x7f6c640f6930 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:31.683 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:31.678+0000 7f6c6bfa8700 1 -- 172.21.15.81:0/4078262934 >> 172.21.15.81:0/4078262934 conn(0x7f6c640f1c70 msgr2=0x7f6c640f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:29:31.683 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:31.678+0000 7f6c6bfa8700 1 -- 172.21.15.81:0/4078262934 shutdown_connections 2023-11-15T06:29:31.684 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:31.678+0000 7f6c6bfa8700 1 -- 172.21.15.81:0/4078262934 wait complete. 2023-11-15T06:29:31.684 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:31.679+0000 7f6c6bfa8700 1 Processor -- start 2023-11-15T06:29:31.684 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:31.679+0000 7f6c6bfa8700 1 -- start start 2023-11-15T06:29:31.684 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:31.679+0000 7f6c6bfa8700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c640f6510 0x7f6c64106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:29:31.684 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:31.679+0000 7f6c6bfa8700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c64106c20 con 0x7f6c640f6510 2023-11-15T06:29:31.685 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:31.680+0000 7f6c69d44700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c640f6510 0x7f6c64106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:29:31.685 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:31.680+0000 7f6c69d44700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c640f6510 0x7f6c64106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:57146/0 (socket says 172.21.15.81:57146) 2023-11-15T06:29:31.685 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:31.680+0000 7f6c69d44700 1 -- 172.21.15.81:0/987746792 learned_addr learned my addr 172.21.15.81:0/987746792 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:29:31.685 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:31.680+0000 7f6c69d44700 1 -- 172.21.15.81:0/987746792 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c54005040 con 0x7f6c640f6510 2023-11-15T06:29:31.685 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:31.680+0000 7f6c69d44700 1 --2- 172.21.15.81:0/987746792 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c640f6510 0x7f6c64106730 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f6c54013510 tx=0x7f6c5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:29:31.686 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:31.680+0000 7f6c5a7fc700 1 -- 172.21.15.81:0/987746792 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c54009320 con 0x7f6c640f6510 2023-11-15T06:29:31.686 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:31.680+0000 7f6c6bfa8700 1 -- 172.21.15.81:0/987746792 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c64106e20 con 0x7f6c640f6510 2023-11-15T06:29:31.686 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:31.680+0000 7f6c5a7fc700 1 -- 172.21.15.81:0/987746792 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6c54013f00 con 0x7f6c640f6510 2023-11-15T06:29:31.686 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:31.680+0000 7f6c5a7fc700 1 -- 172.21.15.81:0/987746792 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c54014d90 con 0x7f6c640f6510 2023-11-15T06:29:31.686 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:31.680+0000 7f6c6bfa8700 1 -- 172.21.15.81:0/987746792 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c641072b0 con 0x7f6c640f6510 2023-11-15T06:29:31.687 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:31.681+0000 7f6c5a7fc700 1 -- 172.21.15.81:0/987746792 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6c540095f0 con 0x7f6c640f6510 2023-11-15T06:29:31.688 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:31.681+0000 7f6c6bfa8700 1 -- 172.21.15.81:0/987746792 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c64045ad0 con 0x7f6c640f6510 2023-11-15T06:29:31.688 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:31.681+0000 7f6c5a7fc700 1 --2- 172.21.15.81:0/987746792 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f6c5002ff60 0x7f6c50032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:29:31.688 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:31.681+0000 7f6c5a7fc700 1 -- 172.21.15.81:0/987746792 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6c54040fe0 con 0x7f6c640f6510 2023-11-15T06:29:31.688 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:31.681+0000 7f6c69543700 1 --2- 172.21.15.81:0/987746792 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f6c5002ff60 0x7f6c50032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:29:31.689 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:31.682+0000 7f6c69543700 1 --2- 172.21.15.81:0/987746792 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f6c5002ff60 0x7f6c50032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f6c600096a0 tx=0x7f6c60006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:29:31.689 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:31.686+0000 7f6c5a7fc700 1 -- 172.21.15.81:0/987746792 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6c5401d7f0 con 0x7f6c640f6510 2023-11-15T06:29:31.894 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:31.892+0000 7f6c6bfa8700 1 -- 172.21.15.81:0/987746792 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6c640fbf60 con 0x7f6c640f6510 2023-11-15T06:29:31.895 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:31.893+0000 7f6c5a7fc700 1 -- 172.21.15.81:0/987746792 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6c54010230 con 0x7f6c640f6510 2023-11-15T06:29:31.895 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:29:31.895 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:29:31.896 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:31.894+0000 7f6c6bfa8700 1 -- 172.21.15.81:0/987746792 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f6c5002ff60 msgr2=0x7f6c50032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:31.897 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:31.894+0000 7f6c6bfa8700 1 --2- 172.21.15.81:0/987746792 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f6c5002ff60 0x7f6c50032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f6c600096a0 tx=0x7f6c60006b40).stop 2023-11-15T06:29:31.897 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:31.894+0000 7f6c6bfa8700 1 -- 172.21.15.81:0/987746792 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c640f6510 msgr2=0x7f6c64106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:31.897 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:31.894+0000 7f6c6bfa8700 1 --2- 172.21.15.81:0/987746792 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c640f6510 0x7f6c64106730 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f6c54013510 tx=0x7f6c5400b040).stop 2023-11-15T06:29:31.897 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:31.894+0000 7f6c6bfa8700 1 -- 172.21.15.81:0/987746792 shutdown_connections 2023-11-15T06:29:31.897 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:31.894+0000 7f6c6bfa8700 1 --2- 172.21.15.81:0/987746792 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f6c5002ff60 0x7f6c50032420 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:31.898 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:31.894+0000 7f6c6bfa8700 1 --2- 172.21.15.81:0/987746792 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c640f6510 0x7f6c64106730 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:31.898 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:31.894+0000 7f6c6bfa8700 1 -- 172.21.15.81:0/987746792 >> 172.21.15.81:0/987746792 conn(0x7f6c640f1c70 msgr2=0x7f6c640f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:29:31.898 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:31.894+0000 7f6c6bfa8700 1 -- 172.21.15.81:0/987746792 shutdown_connections 2023-11-15T06:29:31.898 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:31.894+0000 7f6c6bfa8700 1 -- 172.21.15.81:0/987746792 wait complete. 2023-11-15T06:29:31.899 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:29:33.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:32 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:29:31.812456+0000 mgr.smithi062.rwurtk (mgr.14164) 110 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:29:33.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:32 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:29:31.893668+0000 mon.smithi062 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.81:0/987746792' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:29:33.325 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:29:33.326 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:29:34.550 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:34.548+0000 7f75c4c2f700 1 -- 172.21.15.81:0/4176637974 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75c00f6510 msgr2=0x7f75c00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:34.551 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:34.548+0000 7f75c4c2f700 1 --2- 172.21.15.81:0/4176637974 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75c00f6510 0x7f75c00f6930 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f75a8009fb0 tx=0x7f75a8009b30).stop 2023-11-15T06:29:34.551 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:34.548+0000 7f75c4c2f700 1 -- 172.21.15.81:0/4176637974 shutdown_connections 2023-11-15T06:29:34.551 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:34.548+0000 7f75c4c2f700 1 --2- 172.21.15.81:0/4176637974 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75c00f6510 0x7f75c00f6930 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:34.551 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:34.548+0000 7f75c4c2f700 1 -- 172.21.15.81:0/4176637974 >> 172.21.15.81:0/4176637974 conn(0x7f75c00f1c70 msgr2=0x7f75c00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:29:34.551 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:34.548+0000 7f75c4c2f700 1 -- 172.21.15.81:0/4176637974 shutdown_connections 2023-11-15T06:29:34.552 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:34.549+0000 7f75c4c2f700 1 -- 172.21.15.81:0/4176637974 wait complete. 2023-11-15T06:29:34.554 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:34.549+0000 7f75c4c2f700 1 Processor -- start 2023-11-15T06:29:34.555 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:34.549+0000 7f75c4c2f700 1 -- start start 2023-11-15T06:29:34.555 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:34.549+0000 7f75c4c2f700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75c00f6510 0x7f75c0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:29:34.555 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:34.549+0000 7f75c4c2f700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c0106c20 con 0x7f75c00f6510 2023-11-15T06:29:34.556 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:34.549+0000 7f75be59c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75c00f6510 0x7f75c0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:29:34.556 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:34.549+0000 7f75be59c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75c00f6510 0x7f75c0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:41560/0 (socket says 172.21.15.81:41560) 2023-11-15T06:29:34.556 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:34.549+0000 7f75be59c700 1 -- 172.21.15.81:0/4228431705 learned_addr learned my addr 172.21.15.81:0/4228431705 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:29:34.556 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:34.550+0000 7f75be59c700 1 -- 172.21.15.81:0/4228431705 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a8005040 con 0x7f75c00f6510 2023-11-15T06:29:34.556 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:34.550+0000 7f75be59c700 1 --2- 172.21.15.81:0/4228431705 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75c00f6510 0x7f75c0106730 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f75a8013510 tx=0x7f75a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:29:34.557 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:34.550+0000 7f75b6ffd700 1 -- 172.21.15.81:0/4228431705 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75a8009320 con 0x7f75c00f6510 2023-11-15T06:29:34.557 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:34.550+0000 7f75c4c2f700 1 -- 172.21.15.81:0/4228431705 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75c0106e20 con 0x7f75c00f6510 2023-11-15T06:29:34.557 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:34.550+0000 7f75b6ffd700 1 -- 172.21.15.81:0/4228431705 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f75a8013f00 con 0x7f75c00f6510 2023-11-15T06:29:34.557 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:34.550+0000 7f75b6ffd700 1 -- 172.21.15.81:0/4228431705 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75a8014d90 con 0x7f75c00f6510 2023-11-15T06:29:34.564 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:34.550+0000 7f75c4c2f700 1 -- 172.21.15.81:0/4228431705 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75c01072b0 con 0x7f75c00f6510 2023-11-15T06:29:34.565 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:34.551+0000 7f75b6ffd700 1 -- 172.21.15.81:0/4228431705 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f75a8014420 con 0x7f75c00f6510 2023-11-15T06:29:34.565 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:34.551+0000 7f75c4c2f700 1 -- 172.21.15.81:0/4228431705 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75c0045ad0 con 0x7f75c00f6510 2023-11-15T06:29:34.565 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:34.551+0000 7f75b6ffd700 1 --2- 172.21.15.81:0/4228431705 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f75ac02ff60 0x7f75ac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:29:34.565 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:34.551+0000 7f75b6ffd700 1 -- 172.21.15.81:0/4228431705 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f75a8041000 con 0x7f75c00f6510 2023-11-15T06:29:34.565 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:34.551+0000 7f75bdd9b700 1 --2- 172.21.15.81:0/4228431705 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f75ac02ff60 0x7f75ac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:29:34.566 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:34.551+0000 7f75bdd9b700 1 --2- 172.21.15.81:0/4228431705 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f75ac02ff60 0x7f75ac032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f75b00096a0 tx=0x7f75b0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:29:34.566 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:34.556+0000 7f75b6ffd700 1 -- 172.21.15.81:0/4228431705 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f75a80095f0 con 0x7f75c00f6510 2023-11-15T06:29:34.760 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:34.758+0000 7f75c4c2f700 1 -- 172.21.15.81:0/4228431705 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f75c00fbf60 con 0x7f75c00f6510 2023-11-15T06:29:34.761 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:34.759+0000 7f75b6ffd700 1 -- 172.21.15.81:0/4228431705 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f75a801db70 con 0x7f75c00f6510 2023-11-15T06:29:34.761 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:29:34.762 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:29:34.763 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:34.760+0000 7f75c4c2f700 1 -- 172.21.15.81:0/4228431705 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f75ac02ff60 msgr2=0x7f75ac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:34.763 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:34.760+0000 7f75c4c2f700 1 --2- 172.21.15.81:0/4228431705 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f75ac02ff60 0x7f75ac032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f75b00096a0 tx=0x7f75b0006b40).stop 2023-11-15T06:29:34.763 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:34.760+0000 7f75c4c2f700 1 -- 172.21.15.81:0/4228431705 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75c00f6510 msgr2=0x7f75c0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:34.763 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:34.760+0000 7f75c4c2f700 1 --2- 172.21.15.81:0/4228431705 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75c00f6510 0x7f75c0106730 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f75a8013510 tx=0x7f75a800b040).stop 2023-11-15T06:29:34.764 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:34.760+0000 7f75c4c2f700 1 -- 172.21.15.81:0/4228431705 shutdown_connections 2023-11-15T06:29:34.764 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:34.760+0000 7f75c4c2f700 1 --2- 172.21.15.81:0/4228431705 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f75ac02ff60 0x7f75ac032420 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:34.764 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:34.760+0000 7f75c4c2f700 1 --2- 172.21.15.81:0/4228431705 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75c00f6510 0x7f75c0106730 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:34.764 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:34.760+0000 7f75c4c2f700 1 -- 172.21.15.81:0/4228431705 >> 172.21.15.81:0/4228431705 conn(0x7f75c00f1c70 msgr2=0x7f75c00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:29:34.764 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:34.760+0000 7f75c4c2f700 1 -- 172.21.15.81:0/4228431705 shutdown_connections 2023-11-15T06:29:34.764 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:34.760+0000 7f75c4c2f700 1 -- 172.21.15.81:0/4228431705 wait complete. 2023-11-15T06:29:34.765 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:29:35.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:34 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:29:33.812733+0000 mgr.smithi062.rwurtk (mgr.14164) 111 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:29:35.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:34 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:29:34.759701+0000 mon.smithi062 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.81:0/4228431705' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:29:36.163 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:29:36.163 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:29:37.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:36 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:29:35.813079+0000 mgr.smithi062.rwurtk (mgr.14164) 112 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:29:37.372 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:37.368+0000 7f230f59e700 1 -- 172.21.15.81:0/3201153314 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2310060220 msgr2=0x7f23100fb280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:37.372 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:37.368+0000 7f230f59e700 1 --2- 172.21.15.81:0/3201153314 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2310060220 0x7f23100fb280 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f2300009fb0 tx=0x7f2300009b30).stop 2023-11-15T06:29:37.372 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:37.369+0000 7f230f59e700 1 -- 172.21.15.81:0/3201153314 shutdown_connections 2023-11-15T06:29:37.373 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:37.369+0000 7f230f59e700 1 --2- 172.21.15.81:0/3201153314 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2310060220 0x7f23100fb280 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:37.373 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:37.369+0000 7f230f59e700 1 -- 172.21.15.81:0/3201153314 >> 172.21.15.81:0/3201153314 conn(0x7f23100f08c0 msgr2=0x7f23100f2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:29:37.373 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:37.369+0000 7f230f59e700 1 -- 172.21.15.81:0/3201153314 shutdown_connections 2023-11-15T06:29:37.373 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:37.369+0000 7f230f59e700 1 -- 172.21.15.81:0/3201153314 wait complete. 2023-11-15T06:29:37.373 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:37.369+0000 7f230f59e700 1 Processor -- start 2023-11-15T06:29:37.374 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:37.369+0000 7f230f59e700 1 -- start start 2023-11-15T06:29:37.374 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:37.369+0000 7f230f59e700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2310060220 0x7f2310108650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:29:37.374 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:37.369+0000 7f230f59e700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2310108b40 con 0x7f2310060220 2023-11-15T06:29:37.374 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:37.369+0000 7f230e59c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2310060220 0x7f2310108650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:29:37.375 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:37.369+0000 7f230e59c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2310060220 0x7f2310108650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:41572/0 (socket says 172.21.15.81:41572) 2023-11-15T06:29:37.375 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:37.369+0000 7f230e59c700 1 -- 172.21.15.81:0/4051418034 learned_addr learned my addr 172.21.15.81:0/4051418034 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:29:37.376 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:37.370+0000 7f230e59c700 1 -- 172.21.15.81:0/4051418034 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2300005040 con 0x7f2310060220 2023-11-15T06:29:37.376 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:37.370+0000 7f230e59c700 1 --2- 172.21.15.81:0/4051418034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2310060220 0x7f2310108650 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f2300013510 tx=0x7f230000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:29:37.376 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:37.370+0000 7f22feffd700 1 -- 172.21.15.81:0/4051418034 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2300009320 con 0x7f2310060220 2023-11-15T06:29:37.376 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:37.370+0000 7f230f59e700 1 -- 172.21.15.81:0/4051418034 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2310108d40 con 0x7f2310060220 2023-11-15T06:29:37.377 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:37.370+0000 7f22feffd700 1 -- 172.21.15.81:0/4051418034 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2300013f00 con 0x7f2310060220 2023-11-15T06:29:37.377 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:37.370+0000 7f22feffd700 1 -- 172.21.15.81:0/4051418034 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2300014dc0 con 0x7f2310060220 2023-11-15T06:29:37.377 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:37.370+0000 7f230f59e700 1 -- 172.21.15.81:0/4051418034 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23101091d0 con 0x7f2310060220 2023-11-15T06:29:37.382 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:37.371+0000 7f22feffd700 1 -- 172.21.15.81:0/4051418034 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f23000263a0 con 0x7f2310060220 2023-11-15T06:29:37.383 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:37.371+0000 7f230f59e700 1 -- 172.21.15.81:0/4051418034 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2310045ad0 con 0x7f2310060220 2023-11-15T06:29:37.383 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:37.371+0000 7f22feffd700 1 --2- 172.21.15.81:0/4051418034 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f22f802ffb0 0x7f22f8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:29:37.383 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:37.371+0000 7f22feffd700 1 -- 172.21.15.81:0/4051418034 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2300041330 con 0x7f2310060220 2023-11-15T06:29:37.384 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:37.371+0000 7f230dd9b700 1 --2- 172.21.15.81:0/4051418034 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f22f802ffb0 0x7f22f8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:29:37.384 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:37.372+0000 7f230dd9b700 1 --2- 172.21.15.81:0/4051418034 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f22f802ffb0 0x7f22f8032470 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f23040096a0 tx=0x7f2304006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:29:37.384 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:37.376+0000 7f22feffd700 1 -- 172.21.15.81:0/4051418034 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2300012330 con 0x7f2310060220 2023-11-15T06:29:37.585 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:37.583+0000 7f230f59e700 1 -- 172.21.15.81:0/4051418034 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f231010a080 con 0x7f2310060220 2023-11-15T06:29:37.589 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:37.583+0000 7f22feffd700 1 -- 172.21.15.81:0/4051418034 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f230000f340 con 0x7f2310060220 2023-11-15T06:29:37.590 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:29:37.590 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:29:37.592 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:37.585+0000 7f230f59e700 1 -- 172.21.15.81:0/4051418034 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f22f802ffb0 msgr2=0x7f22f8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:37.592 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:37.585+0000 7f230f59e700 1 --2- 172.21.15.81:0/4051418034 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f22f802ffb0 0x7f22f8032470 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f23040096a0 tx=0x7f2304006b40).stop 2023-11-15T06:29:37.593 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:37.585+0000 7f230f59e700 1 -- 172.21.15.81:0/4051418034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2310060220 msgr2=0x7f2310108650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:37.593 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:37.585+0000 7f230f59e700 1 --2- 172.21.15.81:0/4051418034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2310060220 0x7f2310108650 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f2300013510 tx=0x7f230000b040).stop 2023-11-15T06:29:37.596 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:37.585+0000 7f230f59e700 1 -- 172.21.15.81:0/4051418034 shutdown_connections 2023-11-15T06:29:37.596 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:37.585+0000 7f230f59e700 1 --2- 172.21.15.81:0/4051418034 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f22f802ffb0 0x7f22f8032470 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:37.596 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:37.585+0000 7f230f59e700 1 --2- 172.21.15.81:0/4051418034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2310060220 0x7f2310108650 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:37.597 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:37.585+0000 7f230f59e700 1 -- 172.21.15.81:0/4051418034 >> 172.21.15.81:0/4051418034 conn(0x7f23100f08c0 msgr2=0x7f23100f1370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:29:37.597 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:37.585+0000 7f230f59e700 1 -- 172.21.15.81:0/4051418034 shutdown_connections 2023-11-15T06:29:37.597 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:37.585+0000 7f230f59e700 1 -- 172.21.15.81:0/4051418034 wait complete. 2023-11-15T06:29:37.598 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:29:38.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:37 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:29:37.584215+0000 mon.smithi062 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.81:0/4051418034' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:29:39.021 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:29:39.021 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:29:39.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:38 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:29:37.813493+0000 mgr.smithi062.rwurtk (mgr.14164) 113 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:29:40.263 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:40.258+0000 7f8a7cf35700 1 -- 172.21.15.81:0/22252777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a780f6510 msgr2=0x7f8a780f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:40.263 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:40.258+0000 7f8a7cf35700 1 --2- 172.21.15.81:0/22252777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a780f6510 0x7f8a780f6930 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f8a6c009fb0 tx=0x7f8a6c009b30).stop 2023-11-15T06:29:40.263 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:40.258+0000 7f8a7cf35700 1 -- 172.21.15.81:0/22252777 shutdown_connections 2023-11-15T06:29:40.264 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:40.258+0000 7f8a7cf35700 1 --2- 172.21.15.81:0/22252777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a780f6510 0x7f8a780f6930 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:40.264 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:40.258+0000 7f8a7cf35700 1 -- 172.21.15.81:0/22252777 >> 172.21.15.81:0/22252777 conn(0x7f8a780f1c70 msgr2=0x7f8a780f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:29:40.264 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:40.258+0000 7f8a7cf35700 1 -- 172.21.15.81:0/22252777 shutdown_connections 2023-11-15T06:29:40.264 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:40.258+0000 7f8a7cf35700 1 -- 172.21.15.81:0/22252777 wait complete. 2023-11-15T06:29:40.264 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:40.259+0000 7f8a7cf35700 1 Processor -- start 2023-11-15T06:29:40.265 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:40.259+0000 7f8a7cf35700 1 -- start start 2023-11-15T06:29:40.265 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:40.259+0000 7f8a7cf35700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a780f6510 0x7f8a78106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:29:40.265 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:40.259+0000 7f8a7cf35700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a78106c20 con 0x7f8a780f6510 2023-11-15T06:29:40.265 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:40.259+0000 7f8a7659c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a780f6510 0x7f8a78106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:29:40.265 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:40.259+0000 7f8a7659c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a780f6510 0x7f8a78106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:41584/0 (socket says 172.21.15.81:41584) 2023-11-15T06:29:40.266 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:40.259+0000 7f8a7659c700 1 -- 172.21.15.81:0/1492601899 learned_addr learned my addr 172.21.15.81:0/1492601899 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:29:40.266 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:40.260+0000 7f8a7659c700 1 -- 172.21.15.81:0/1492601899 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a6c005040 con 0x7f8a780f6510 2023-11-15T06:29:40.266 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:40.260+0000 7f8a7659c700 1 --2- 172.21.15.81:0/1492601899 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a780f6510 0x7f8a78106730 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f8a6c014000 tx=0x7f8a6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:29:40.266 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:40.260+0000 7f8a66ffd700 1 -- 172.21.15.81:0/1492601899 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a6c0095f0 con 0x7f8a780f6510 2023-11-15T06:29:40.267 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:40.260+0000 7f8a7cf35700 1 -- 172.21.15.81:0/1492601899 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a78106e20 con 0x7f8a780f6510 2023-11-15T06:29:40.267 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:40.260+0000 7f8a66ffd700 1 -- 172.21.15.81:0/1492601899 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8a6c007e70 con 0x7f8a780f6510 2023-11-15T06:29:40.267 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:40.260+0000 7f8a66ffd700 1 -- 172.21.15.81:0/1492601899 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a6c014ea0 con 0x7f8a780f6510 2023-11-15T06:29:40.267 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:40.260+0000 7f8a7cf35700 1 -- 172.21.15.81:0/1492601899 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a781072b0 con 0x7f8a780f6510 2023-11-15T06:29:40.269 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:40.261+0000 7f8a66ffd700 1 -- 172.21.15.81:0/1492601899 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8a6c025070 con 0x7f8a780f6510 2023-11-15T06:29:40.269 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:40.261+0000 7f8a7cf35700 1 -- 172.21.15.81:0/1492601899 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a78045ad0 con 0x7f8a780f6510 2023-11-15T06:29:40.269 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:40.261+0000 7f8a66ffd700 1 --2- 172.21.15.81:0/1492601899 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f8a6002ff60 0x7f8a60032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:29:40.269 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:40.261+0000 7f8a66ffd700 1 -- 172.21.15.81:0/1492601899 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8a6c040a30 con 0x7f8a780f6510 2023-11-15T06:29:40.270 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:40.261+0000 7f8a75d9b700 1 --2- 172.21.15.81:0/1492601899 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f8a6002ff60 0x7f8a60032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:29:40.270 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:40.261+0000 7f8a75d9b700 1 --2- 172.21.15.81:0/1492601899 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f8a6002ff60 0x7f8a60032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f8a680096a0 tx=0x7f8a68006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:29:40.270 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:40.266+0000 7f8a66ffd700 1 -- 172.21.15.81:0/1492601899 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8a6c0079c0 con 0x7f8a780f6510 2023-11-15T06:29:40.471 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:40.468+0000 7f8a7cf35700 1 -- 172.21.15.81:0/1492601899 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8a780fbf60 con 0x7f8a780f6510 2023-11-15T06:29:40.471 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:40.468+0000 7f8a66ffd700 1 -- 172.21.15.81:0/1492601899 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8a6c00f300 con 0x7f8a780f6510 2023-11-15T06:29:40.472 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:29:40.472 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:29:40.473 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:40.470+0000 7f8a7cf35700 1 -- 172.21.15.81:0/1492601899 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f8a6002ff60 msgr2=0x7f8a60032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:40.474 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:40.470+0000 7f8a7cf35700 1 --2- 172.21.15.81:0/1492601899 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f8a6002ff60 0x7f8a60032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f8a680096a0 tx=0x7f8a68006b40).stop 2023-11-15T06:29:40.474 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:40.470+0000 7f8a7cf35700 1 -- 172.21.15.81:0/1492601899 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a780f6510 msgr2=0x7f8a78106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:40.474 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:40.470+0000 7f8a7cf35700 1 --2- 172.21.15.81:0/1492601899 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a780f6510 0x7f8a78106730 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f8a6c014000 tx=0x7f8a6c00b040).stop 2023-11-15T06:29:40.474 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:40.470+0000 7f8a7cf35700 1 -- 172.21.15.81:0/1492601899 shutdown_connections 2023-11-15T06:29:40.475 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:40.470+0000 7f8a7cf35700 1 --2- 172.21.15.81:0/1492601899 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f8a6002ff60 0x7f8a60032420 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:40.475 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:40.470+0000 7f8a7cf35700 1 --2- 172.21.15.81:0/1492601899 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a780f6510 0x7f8a78106730 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:40.475 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:40.470+0000 7f8a7cf35700 1 -- 172.21.15.81:0/1492601899 >> 172.21.15.81:0/1492601899 conn(0x7f8a780f1c70 msgr2=0x7f8a780f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:29:40.475 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:40.470+0000 7f8a7cf35700 1 -- 172.21.15.81:0/1492601899 shutdown_connections 2023-11-15T06:29:40.475 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:40.470+0000 7f8a7cf35700 1 -- 172.21.15.81:0/1492601899 wait complete. 2023-11-15T06:29:40.475 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:29:41.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:40 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:29:39.813801+0000 mgr.smithi062.rwurtk (mgr.14164) 114 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:29:41.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:40 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:29:40.470298+0000 mon.smithi062 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.81:0/1492601899' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:29:41.866 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:29:41.867 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:29:43.077 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:43.074+0000 7f64b4a3f700 1 -- 172.21.15.81:0/1396580190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64b00f5410 msgr2=0x7f64b00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:43.077 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:43.074+0000 7f64b4a3f700 1 --2- 172.21.15.81:0/1396580190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64b00f5410 0x7f64b00f5830 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f64a4009fb0 tx=0x7f64a4009b30).stop 2023-11-15T06:29:43.078 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:43.074+0000 7f64b4a3f700 1 -- 172.21.15.81:0/1396580190 shutdown_connections 2023-11-15T06:29:43.078 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:43.074+0000 7f64b4a3f700 1 --2- 172.21.15.81:0/1396580190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64b00f5410 0x7f64b00f5830 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:43.078 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:43.074+0000 7f64b4a3f700 1 -- 172.21.15.81:0/1396580190 >> 172.21.15.81:0/1396580190 conn(0x7f64b00f0b70 msgr2=0x7f64b00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:29:43.078 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:43.074+0000 7f64b4a3f700 1 -- 172.21.15.81:0/1396580190 shutdown_connections 2023-11-15T06:29:43.078 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:43.074+0000 7f64b4a3f700 1 -- 172.21.15.81:0/1396580190 wait complete. 2023-11-15T06:29:43.079 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:43.074+0000 7f64b4a3f700 1 Processor -- start 2023-11-15T06:29:43.079 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:43.074+0000 7f64b4a3f700 1 -- start start 2023-11-15T06:29:43.080 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:43.075+0000 7f64b4a3f700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64b00f5410 0x7f64b00fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:29:43.081 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:43.075+0000 7f64b4a3f700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b00fb820 con 0x7f64b00f5410 2023-11-15T06:29:43.081 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:43.075+0000 7f64ae59c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64b00f5410 0x7f64b00fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:29:43.081 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:43.075+0000 7f64ae59c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64b00f5410 0x7f64b00fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:47996/0 (socket says 172.21.15.81:47996) 2023-11-15T06:29:43.081 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:43.075+0000 7f64ae59c700 1 -- 172.21.15.81:0/1841845644 learned_addr learned my addr 172.21.15.81:0/1841845644 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:29:43.082 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:43.075+0000 7f64ae59c700 1 -- 172.21.15.81:0/1841845644 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a4005040 con 0x7f64b00f5410 2023-11-15T06:29:43.082 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:43.075+0000 7f64ae59c700 1 --2- 172.21.15.81:0/1841845644 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64b00f5410 0x7f64b00fd3a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f64a4013510 tx=0x7f64a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:29:43.085 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:43.075+0000 7f649effd700 1 -- 172.21.15.81:0/1841845644 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64a4009320 con 0x7f64b00f5410 2023-11-15T06:29:43.085 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:43.076+0000 7f64b4a3f700 1 -- 172.21.15.81:0/1841845644 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64b00fb9c0 con 0x7f64b00f5410 2023-11-15T06:29:43.085 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:43.076+0000 7f649effd700 1 -- 172.21.15.81:0/1841845644 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f64a4013f00 con 0x7f64b00f5410 2023-11-15T06:29:43.085 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:43.076+0000 7f649effd700 1 -- 172.21.15.81:0/1841845644 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64a4014d90 con 0x7f64b00f5410 2023-11-15T06:29:43.086 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:43.076+0000 7f64b4a3f700 1 -- 172.21.15.81:0/1841845644 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64b00fbe50 con 0x7f64b00f5410 2023-11-15T06:29:43.086 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:43.076+0000 7f649effd700 1 -- 172.21.15.81:0/1841845644 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f64a40095f0 con 0x7f64b00f5410 2023-11-15T06:29:43.086 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:43.076+0000 7f64b4a3f700 1 -- 172.21.15.81:0/1841845644 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64b0059100 con 0x7f64b00f5410 2023-11-15T06:29:43.086 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:43.077+0000 7f649effd700 1 --2- 172.21.15.81:0/1841845644 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f649802ff60 0x7f6498032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:29:43.086 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:43.077+0000 7f649effd700 1 -- 172.21.15.81:0/1841845644 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f64a4041000 con 0x7f64b00f5410 2023-11-15T06:29:43.087 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:43.077+0000 7f64add9b700 1 --2- 172.21.15.81:0/1841845644 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f649802ff60 0x7f6498032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:29:43.087 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:43.077+0000 7f64add9b700 1 --2- 172.21.15.81:0/1841845644 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f649802ff60 0x7f6498032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f64a00096a0 tx=0x7f64a0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:29:43.087 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:43.081+0000 7f649effd700 1 -- 172.21.15.81:0/1841845644 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f64a4014420 con 0x7f64b00f5410 2023-11-15T06:29:43.169 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:42 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:29:41.814120+0000 mgr.smithi062.rwurtk (mgr.14164) 115 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:29:43.288 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:43.284+0000 7f64b4a3f700 1 -- 172.21.15.81:0/1841845644 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f64b00fae60 con 0x7f64b00f5410 2023-11-15T06:29:43.288 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:43.285+0000 7f649effd700 1 -- 172.21.15.81:0/1841845644 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f64a4010230 con 0x7f64b00f5410 2023-11-15T06:29:43.289 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:29:43.289 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:29:43.290 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:43.286+0000 7f64b4a3f700 1 -- 172.21.15.81:0/1841845644 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f649802ff60 msgr2=0x7f6498032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:43.290 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:43.286+0000 7f64b4a3f700 1 --2- 172.21.15.81:0/1841845644 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f649802ff60 0x7f6498032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f64a00096a0 tx=0x7f64a0006b40).stop 2023-11-15T06:29:43.291 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:43.286+0000 7f64b4a3f700 1 -- 172.21.15.81:0/1841845644 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64b00f5410 msgr2=0x7f64b00fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:43.291 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:43.286+0000 7f64b4a3f700 1 --2- 172.21.15.81:0/1841845644 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64b00f5410 0x7f64b00fd3a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f64a4013510 tx=0x7f64a400b040).stop 2023-11-15T06:29:43.291 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:43.286+0000 7f64b4a3f700 1 -- 172.21.15.81:0/1841845644 shutdown_connections 2023-11-15T06:29:43.291 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:43.286+0000 7f64b4a3f700 1 --2- 172.21.15.81:0/1841845644 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f649802ff60 0x7f6498032420 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:43.291 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:43.286+0000 7f64b4a3f700 1 --2- 172.21.15.81:0/1841845644 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64b00f5410 0x7f64b00fd3a0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:43.292 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:43.286+0000 7f64b4a3f700 1 -- 172.21.15.81:0/1841845644 >> 172.21.15.81:0/1841845644 conn(0x7f64b00f0b70 msgr2=0x7f64b005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:29:43.292 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:43.287+0000 7f64b4a3f700 1 -- 172.21.15.81:0/1841845644 shutdown_connections 2023-11-15T06:29:43.292 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:43.287+0000 7f64b4a3f700 1 -- 172.21.15.81:0/1841845644 wait complete. 2023-11-15T06:29:43.292 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:29:44.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:43 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:29:43.286890+0000 mon.smithi062 (mon.0) 193 : audit [DBG] from='client.? 172.21.15.81:0/1841845644' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:29:44.682 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:29:44.683 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:29:45.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:44 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:29:43.814388+0000 mgr.smithi062.rwurtk (mgr.14164) 116 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:29:45.860 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:45.854+0000 7f5500130700 1 -- 172.21.15.81:0/3018033419 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54f80f5410 msgr2=0x7f54f80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:45.860 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:45.854+0000 7f5500130700 1 --2- 172.21.15.81:0/3018033419 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54f80f5410 0x7f54f80f5830 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f54e8009fb0 tx=0x7f54e8009b30).stop 2023-11-15T06:29:45.860 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:45.854+0000 7f5500130700 1 -- 172.21.15.81:0/3018033419 shutdown_connections 2023-11-15T06:29:45.860 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:45.854+0000 7f5500130700 1 --2- 172.21.15.81:0/3018033419 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54f80f5410 0x7f54f80f5830 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:45.861 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:45.854+0000 7f5500130700 1 -- 172.21.15.81:0/3018033419 >> 172.21.15.81:0/3018033419 conn(0x7f54f80f0b70 msgr2=0x7f54f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:29:45.861 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:45.854+0000 7f5500130700 1 -- 172.21.15.81:0/3018033419 shutdown_connections 2023-11-15T06:29:45.861 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:45.854+0000 7f5500130700 1 -- 172.21.15.81:0/3018033419 wait complete. 2023-11-15T06:29:45.861 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:45.854+0000 7f5500130700 1 Processor -- start 2023-11-15T06:29:45.861 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:45.855+0000 7f5500130700 1 -- start start 2023-11-15T06:29:45.862 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:45.855+0000 7f5500130700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54f80f5410 0x7f54f8106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:29:45.862 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:45.855+0000 7f5500130700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54f8106c10 con 0x7f54f80f5410 2023-11-15T06:29:45.862 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:45.855+0000 7f54fdecc700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54f80f5410 0x7f54f8106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:29:45.862 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:45.855+0000 7f54fdecc700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54f80f5410 0x7f54f8106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:48016/0 (socket says 172.21.15.81:48016) 2023-11-15T06:29:45.862 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:45.855+0000 7f54fdecc700 1 -- 172.21.15.81:0/3633664206 learned_addr learned my addr 172.21.15.81:0/3633664206 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:29:45.863 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:45.855+0000 7f54fdecc700 1 -- 172.21.15.81:0/3633664206 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e8005040 con 0x7f54f80f5410 2023-11-15T06:29:45.863 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:45.855+0000 7f54fdecc700 1 --2- 172.21.15.81:0/3633664206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54f80f5410 0x7f54f8106720 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f54e8013510 tx=0x7f54e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:29:45.863 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:45.855+0000 7f54ee7fc700 1 -- 172.21.15.81:0/3633664206 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54e8009320 con 0x7f54f80f5410 2023-11-15T06:29:45.863 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:45.855+0000 7f5500130700 1 -- 172.21.15.81:0/3633664206 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54f8106e10 con 0x7f54f80f5410 2023-11-15T06:29:45.863 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:45.855+0000 7f54ee7fc700 1 -- 172.21.15.81:0/3633664206 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f54e8013f00 con 0x7f54f80f5410 2023-11-15T06:29:45.864 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:45.856+0000 7f54ee7fc700 1 -- 172.21.15.81:0/3633664206 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54e8014d90 con 0x7f54f80f5410 2023-11-15T06:29:45.864 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:45.856+0000 7f5500130700 1 -- 172.21.15.81:0/3633664206 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54f81072a0 con 0x7f54f80f5410 2023-11-15T06:29:45.864 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:45.856+0000 7f54ee7fc700 1 -- 172.21.15.81:0/3633664206 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f54e80095f0 con 0x7f54f80f5410 2023-11-15T06:29:45.864 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:45.856+0000 7f5500130700 1 -- 172.21.15.81:0/3633664206 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54f8045ad0 con 0x7f54f80f5410 2023-11-15T06:29:45.864 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:45.856+0000 7f54ee7fc700 1 --2- 172.21.15.81:0/3633664206 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f54e402ff60 0x7f54e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:29:45.865 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:45.856+0000 7f54ee7fc700 1 -- 172.21.15.81:0/3633664206 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f54e80410a0 con 0x7f54f80f5410 2023-11-15T06:29:45.865 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:45.857+0000 7f54fd6cb700 1 --2- 172.21.15.81:0/3633664206 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f54e402ff60 0x7f54e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:29:45.865 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:45.857+0000 7f54fd6cb700 1 --2- 172.21.15.81:0/3633664206 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f54e402ff60 0x7f54e4032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f54f40096a0 tx=0x7f54f4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:29:45.866 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:45.861+0000 7f54ee7fc700 1 -- 172.21.15.81:0/3633664206 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f54e801d7f0 con 0x7f54f80f5410 2023-11-15T06:29:46.069 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:46.066+0000 7f5500130700 1 -- 172.21.15.81:0/3633664206 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f54f8108150 con 0x7f54f80f5410 2023-11-15T06:29:46.070 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:46.066+0000 7f54ee7fc700 1 -- 172.21.15.81:0/3633664206 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f54e8010230 con 0x7f54f80f5410 2023-11-15T06:29:46.070 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:29:46.070 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:29:46.071 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:46.068+0000 7f5500130700 1 -- 172.21.15.81:0/3633664206 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f54e402ff60 msgr2=0x7f54e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:46.071 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:46.068+0000 7f5500130700 1 --2- 172.21.15.81:0/3633664206 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f54e402ff60 0x7f54e4032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f54f40096a0 tx=0x7f54f4006b40).stop 2023-11-15T06:29:46.072 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:46.068+0000 7f5500130700 1 -- 172.21.15.81:0/3633664206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54f80f5410 msgr2=0x7f54f8106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:46.072 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:46.068+0000 7f5500130700 1 --2- 172.21.15.81:0/3633664206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54f80f5410 0x7f54f8106720 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f54e8013510 tx=0x7f54e800b040).stop 2023-11-15T06:29:46.072 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:46.068+0000 7f5500130700 1 -- 172.21.15.81:0/3633664206 shutdown_connections 2023-11-15T06:29:46.072 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:46.068+0000 7f5500130700 1 --2- 172.21.15.81:0/3633664206 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f54e402ff60 0x7f54e4032420 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:46.072 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:46.068+0000 7f5500130700 1 --2- 172.21.15.81:0/3633664206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54f80f5410 0x7f54f8106720 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:46.073 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:46.068+0000 7f5500130700 1 -- 172.21.15.81:0/3633664206 >> 172.21.15.81:0/3633664206 conn(0x7f54f80f0b70 msgr2=0x7f54f805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:29:46.073 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:46.068+0000 7f5500130700 1 -- 172.21.15.81:0/3633664206 shutdown_connections 2023-11-15T06:29:46.073 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:46.068+0000 7f5500130700 1 -- 172.21.15.81:0/3633664206 wait complete. 2023-11-15T06:29:46.073 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:29:47.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:46 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:29:45.814784+0000 mgr.smithi062.rwurtk (mgr.14164) 117 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:29:47.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:46 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:29:46.068360+0000 mon.smithi062 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.81:0/3633664206' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:29:47.494 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:29:47.494 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:29:48.759 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:48.756+0000 7f90f4ea2700 1 -- 172.21.15.81:0/3876612496 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90f00f78e0 msgr2=0x7f90f00f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:48.760 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:48.756+0000 7f90f4ea2700 1 --2- 172.21.15.81:0/3876612496 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90f00f78e0 0x7f90f00f7d00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f90e0009fb0 tx=0x7f90e0009b30).stop 2023-11-15T06:29:48.760 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:48.756+0000 7f90f4ea2700 1 -- 172.21.15.81:0/3876612496 shutdown_connections 2023-11-15T06:29:48.760 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:48.756+0000 7f90f4ea2700 1 --2- 172.21.15.81:0/3876612496 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90f00f78e0 0x7f90f00f7d00 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:48.760 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:48.756+0000 7f90f4ea2700 1 -- 172.21.15.81:0/3876612496 >> 172.21.15.81:0/3876612496 conn(0x7f90f00f0120 msgr2=0x7f90f00f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:29:48.761 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:48.756+0000 7f90f4ea2700 1 -- 172.21.15.81:0/3876612496 shutdown_connections 2023-11-15T06:29:48.761 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:48.756+0000 7f90f4ea2700 1 -- 172.21.15.81:0/3876612496 wait complete. 2023-11-15T06:29:48.761 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:48.756+0000 7f90f4ea2700 1 Processor -- start 2023-11-15T06:29:48.761 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:48.757+0000 7f90f4ea2700 1 -- start start 2023-11-15T06:29:48.762 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:48.757+0000 7f90f4ea2700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90f00f78e0 0x7f90f01064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:29:48.762 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:48.757+0000 7f90f4ea2700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f01069d0 con 0x7f90f00f78e0 2023-11-15T06:29:48.764 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:48.757+0000 7f90ef7fe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90f00f78e0 0x7f90f01064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:29:48.764 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:48.757+0000 7f90ef7fe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90f00f78e0 0x7f90f01064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:48022/0 (socket says 172.21.15.81:48022) 2023-11-15T06:29:48.764 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:48.757+0000 7f90ef7fe700 1 -- 172.21.15.81:0/3766070881 learned_addr learned my addr 172.21.15.81:0/3766070881 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:29:48.764 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:48.757+0000 7f90ef7fe700 1 -- 172.21.15.81:0/3766070881 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90e0005040 con 0x7f90f00f78e0 2023-11-15T06:29:48.764 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:48.758+0000 7f90ef7fe700 1 --2- 172.21.15.81:0/3766070881 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90f00f78e0 0x7f90f01064e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f90e0013510 tx=0x7f90e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:29:48.765 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:48.758+0000 7f90d7fff700 1 -- 172.21.15.81:0/3766070881 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90e0009320 con 0x7f90f00f78e0 2023-11-15T06:29:48.765 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:48.758+0000 7f90f4ea2700 1 -- 172.21.15.81:0/3766070881 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90f0106bd0 con 0x7f90f00f78e0 2023-11-15T06:29:48.765 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:48.758+0000 7f90d7fff700 1 -- 172.21.15.81:0/3766070881 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f90e0013f00 con 0x7f90f00f78e0 2023-11-15T06:29:48.765 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:48.758+0000 7f90d7fff700 1 -- 172.21.15.81:0/3766070881 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90e0014dc0 con 0x7f90f00f78e0 2023-11-15T06:29:48.765 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:48.758+0000 7f90f4ea2700 1 -- 172.21.15.81:0/3766070881 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90f0107060 con 0x7f90f00f78e0 2023-11-15T06:29:48.766 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:48.759+0000 7f90d7fff700 1 -- 172.21.15.81:0/3766070881 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f90e0014420 con 0x7f90f00f78e0 2023-11-15T06:29:48.766 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:48.759+0000 7f90f4ea2700 1 -- 172.21.15.81:0/3766070881 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90f0045ad0 con 0x7f90f00f78e0 2023-11-15T06:29:48.766 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:48.759+0000 7f90d7fff700 1 --2- 172.21.15.81:0/3766070881 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f90d802ff60 0x7f90d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:29:48.766 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:48.759+0000 7f90d7fff700 1 -- 172.21.15.81:0/3766070881 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f90e0041100 con 0x7f90f00f78e0 2023-11-15T06:29:48.767 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:48.759+0000 7f90eeffd700 1 --2- 172.21.15.81:0/3766070881 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f90d802ff60 0x7f90d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:29:48.767 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:48.759+0000 7f90eeffd700 1 --2- 172.21.15.81:0/3766070881 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f90d802ff60 0x7f90d8032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f90e40096a0 tx=0x7f90e4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:29:48.768 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:48.764+0000 7f90d7fff700 1 -- 172.21.15.81:0/3766070881 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f90e00095f0 con 0x7f90f00f78e0 2023-11-15T06:29:48.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:48 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:29:47.815169+0000 mgr.smithi062.rwurtk (mgr.14164) 118 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:29:48.970 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:48.967+0000 7f90f4ea2700 1 -- 172.21.15.81:0/3766070881 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f90f0107eb0 con 0x7f90f00f78e0 2023-11-15T06:29:48.971 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:48.969+0000 7f90d7fff700 1 -- 172.21.15.81:0/3766070881 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f90e001cb10 con 0x7f90f00f78e0 2023-11-15T06:29:48.972 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:29:48.972 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:29:48.977 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:48.969+0000 7f90f4ea2700 1 -- 172.21.15.81:0/3766070881 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f90d802ff60 msgr2=0x7f90d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:48.977 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:48.969+0000 7f90f4ea2700 1 --2- 172.21.15.81:0/3766070881 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f90d802ff60 0x7f90d8032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f90e40096a0 tx=0x7f90e4006b40).stop 2023-11-15T06:29:48.977 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:48.969+0000 7f90f4ea2700 1 -- 172.21.15.81:0/3766070881 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90f00f78e0 msgr2=0x7f90f01064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:48.978 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:48.969+0000 7f90f4ea2700 1 --2- 172.21.15.81:0/3766070881 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90f00f78e0 0x7f90f01064e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f90e0013510 tx=0x7f90e000b040).stop 2023-11-15T06:29:48.978 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:48.969+0000 7f90f4ea2700 1 -- 172.21.15.81:0/3766070881 shutdown_connections 2023-11-15T06:29:48.979 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:48.969+0000 7f90f4ea2700 1 --2- 172.21.15.81:0/3766070881 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f90d802ff60 0x7f90d8032420 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:48.979 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:48.969+0000 7f90f4ea2700 1 --2- 172.21.15.81:0/3766070881 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90f00f78e0 0x7f90f01064e0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:48.979 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:48.969+0000 7f90f4ea2700 1 -- 172.21.15.81:0/3766070881 >> 172.21.15.81:0/3766070881 conn(0x7f90f00f0120 msgr2=0x7f90f00f0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:29:48.980 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:48.969+0000 7f90f4ea2700 1 -- 172.21.15.81:0/3766070881 shutdown_connections 2023-11-15T06:29:48.980 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:48.969+0000 7f90f4ea2700 1 -- 172.21.15.81:0/3766070881 wait complete. 2023-11-15T06:29:48.981 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:29:49.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:49 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:29:48.969539+0000 mon.smithi062 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.81:0/3766070881' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:29:50.366 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:29:50.367 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:29:50.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:50 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:29:49.815646+0000 mgr.smithi062.rwurtk (mgr.14164) 119 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:29:51.543 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:51.536+0000 7f6118186700 1 -- 172.21.15.81:0/2193980477 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61100f6510 msgr2=0x7f61100f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:51.543 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:51.536+0000 7f6118186700 1 --2- 172.21.15.81:0/2193980477 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61100f6510 0x7f61100f6930 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f6100009fb0 tx=0x7f6100009b30).stop 2023-11-15T06:29:51.544 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:51.536+0000 7f6118186700 1 -- 172.21.15.81:0/2193980477 shutdown_connections 2023-11-15T06:29:51.544 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:51.536+0000 7f6118186700 1 --2- 172.21.15.81:0/2193980477 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61100f6510 0x7f61100f6930 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:51.544 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:51.536+0000 7f6118186700 1 -- 172.21.15.81:0/2193980477 >> 172.21.15.81:0/2193980477 conn(0x7f61100f1c70 msgr2=0x7f61100f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:29:51.545 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:51.536+0000 7f6118186700 1 -- 172.21.15.81:0/2193980477 shutdown_connections 2023-11-15T06:29:51.545 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:51.536+0000 7f6118186700 1 -- 172.21.15.81:0/2193980477 wait complete. 2023-11-15T06:29:51.545 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:51.537+0000 7f6118186700 1 Processor -- start 2023-11-15T06:29:51.546 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:51.537+0000 7f6118186700 1 -- start start 2023-11-15T06:29:51.547 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:51.537+0000 7f6118186700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61100f6510 0x7f6110106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:29:51.547 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:51.537+0000 7f6118186700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6110106c20 con 0x7f61100f6510 2023-11-15T06:29:51.548 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:51.537+0000 7f6115f22700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61100f6510 0x7f6110106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:29:51.548 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:51.537+0000 7f6115f22700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61100f6510 0x7f6110106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:48050/0 (socket says 172.21.15.81:48050) 2023-11-15T06:29:51.549 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:51.537+0000 7f6115f22700 1 -- 172.21.15.81:0/1936040569 learned_addr learned my addr 172.21.15.81:0/1936040569 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:29:51.549 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:51.537+0000 7f6115f22700 1 -- 172.21.15.81:0/1936040569 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6100005040 con 0x7f61100f6510 2023-11-15T06:29:51.550 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:51.538+0000 7f6115f22700 1 --2- 172.21.15.81:0/1936040569 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61100f6510 0x7f6110106730 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f6100014000 tx=0x7f610000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:29:51.550 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:51.538+0000 7f61067fc700 1 -- 172.21.15.81:0/1936040569 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61000095f0 con 0x7f61100f6510 2023-11-15T06:29:51.550 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:51.538+0000 7f61067fc700 1 -- 172.21.15.81:0/1936040569 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6100007e70 con 0x7f61100f6510 2023-11-15T06:29:51.551 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:51.538+0000 7f6118186700 1 -- 172.21.15.81:0/1936040569 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6110106e20 con 0x7f61100f6510 2023-11-15T06:29:51.551 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:51.538+0000 7f61067fc700 1 -- 172.21.15.81:0/1936040569 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6100014ea0 con 0x7f61100f6510 2023-11-15T06:29:51.551 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:51.538+0000 7f6118186700 1 -- 172.21.15.81:0/1936040569 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61101072b0 con 0x7f61100f6510 2023-11-15T06:29:51.552 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:51.539+0000 7f61067fc700 1 -- 172.21.15.81:0/1936040569 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6100025070 con 0x7f61100f6510 2023-11-15T06:29:51.552 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:51.539+0000 7f6118186700 1 -- 172.21.15.81:0/1936040569 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6110045ad0 con 0x7f61100f6510 2023-11-15T06:29:51.552 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:51.539+0000 7f61067fc700 1 --2- 172.21.15.81:0/1936040569 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f60fc02ff60 0x7f60fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:29:51.553 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:51.539+0000 7f61067fc700 1 -- 172.21.15.81:0/1936040569 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6100040970 con 0x7f61100f6510 2023-11-15T06:29:51.553 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:51.539+0000 7f6115721700 1 --2- 172.21.15.81:0/1936040569 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f60fc02ff60 0x7f60fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:29:51.553 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:51.539+0000 7f6115721700 1 --2- 172.21.15.81:0/1936040569 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f60fc02ff60 0x7f60fc032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f610c0096a0 tx=0x7f610c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:29:51.554 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:51.543+0000 7f61067fc700 1 -- 172.21.15.81:0/1936040569 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f61000079c0 con 0x7f61100f6510 2023-11-15T06:29:51.750 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:51.747+0000 7f6118186700 1 -- 172.21.15.81:0/1936040569 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f61100fbf60 con 0x7f61100f6510 2023-11-15T06:29:51.751 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:51.748+0000 7f61067fc700 1 -- 172.21.15.81:0/1936040569 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f610000f300 con 0x7f61100f6510 2023-11-15T06:29:51.751 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:29:51.752 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:29:51.753 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:51.750+0000 7f6118186700 1 -- 172.21.15.81:0/1936040569 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f60fc02ff60 msgr2=0x7f60fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:51.753 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:51.750+0000 7f6118186700 1 --2- 172.21.15.81:0/1936040569 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f60fc02ff60 0x7f60fc032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f610c0096a0 tx=0x7f610c006b40).stop 2023-11-15T06:29:51.753 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:51.750+0000 7f6118186700 1 -- 172.21.15.81:0/1936040569 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61100f6510 msgr2=0x7f6110106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:51.753 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:51.750+0000 7f6118186700 1 --2- 172.21.15.81:0/1936040569 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61100f6510 0x7f6110106730 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f6100014000 tx=0x7f610000b040).stop 2023-11-15T06:29:51.754 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:51.750+0000 7f6118186700 1 -- 172.21.15.81:0/1936040569 shutdown_connections 2023-11-15T06:29:51.754 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:51.750+0000 7f6118186700 1 --2- 172.21.15.81:0/1936040569 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f60fc02ff60 0x7f60fc032420 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:51.754 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:51.750+0000 7f6118186700 1 --2- 172.21.15.81:0/1936040569 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61100f6510 0x7f6110106730 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:51.754 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:51.750+0000 7f6118186700 1 -- 172.21.15.81:0/1936040569 >> 172.21.15.81:0/1936040569 conn(0x7f61100f1c70 msgr2=0x7f61100f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:29:51.754 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:51.750+0000 7f6118186700 1 -- 172.21.15.81:0/1936040569 shutdown_connections 2023-11-15T06:29:51.755 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:51.750+0000 7f6118186700 1 -- 172.21.15.81:0/1936040569 wait complete. 2023-11-15T06:29:51.755 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:29:52.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:51 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:29:51.750010+0000 mon.smithi062 (mon.0) 196 : audit [DBG] from='client.? 172.21.15.81:0/1936040569' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:29:53.147 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:29:53.148 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:29:53.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:52 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:29:51.816126+0000 mgr.smithi062.rwurtk (mgr.14164) 120 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:29:54.335 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:54.332+0000 7ff81184e700 1 -- 172.21.15.81:0/1644980733 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff80c0f78e0 msgr2=0x7ff80c0f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:54.335 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:54.332+0000 7ff81184e700 1 --2- 172.21.15.81:0/1644980733 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff80c0f78e0 0x7ff80c0f7d00 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7ff800009fb0 tx=0x7ff800009b30).stop 2023-11-15T06:29:54.335 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:54.332+0000 7ff81184e700 1 -- 172.21.15.81:0/1644980733 shutdown_connections 2023-11-15T06:29:54.336 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:54.332+0000 7ff81184e700 1 --2- 172.21.15.81:0/1644980733 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff80c0f78e0 0x7ff80c0f7d00 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:54.336 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:54.332+0000 7ff81184e700 1 -- 172.21.15.81:0/1644980733 >> 172.21.15.81:0/1644980733 conn(0x7ff80c0f0120 msgr2=0x7ff80c0f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:29:54.336 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:54.332+0000 7ff81184e700 1 -- 172.21.15.81:0/1644980733 shutdown_connections 2023-11-15T06:29:54.336 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:54.332+0000 7ff81184e700 1 -- 172.21.15.81:0/1644980733 wait complete. 2023-11-15T06:29:54.338 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:54.332+0000 7ff81184e700 1 Processor -- start 2023-11-15T06:29:54.338 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:54.332+0000 7ff81184e700 1 -- start start 2023-11-15T06:29:54.338 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:54.333+0000 7ff81184e700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff80c0f78e0 0x7ff80c0f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:29:54.339 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:54.333+0000 7ff81184e700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80c0f5630 con 0x7ff80c0f78e0 2023-11-15T06:29:54.339 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:54.333+0000 7ff81084c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff80c0f78e0 0x7ff80c0f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:29:54.340 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:54.333+0000 7ff81084c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff80c0f78e0 0x7ff80c0f71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:37508/0 (socket says 172.21.15.81:37508) 2023-11-15T06:29:54.340 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:54.333+0000 7ff81084c700 1 -- 172.21.15.81:0/2640234497 learned_addr learned my addr 172.21.15.81:0/2640234497 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:29:54.340 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:54.333+0000 7ff81084c700 1 -- 172.21.15.81:0/2640234497 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff800005040 con 0x7ff80c0f78e0 2023-11-15T06:29:54.340 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:54.334+0000 7ff81084c700 1 --2- 172.21.15.81:0/2640234497 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff80c0f78e0 0x7ff80c0f71b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7ff800013510 tx=0x7ff80000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:29:54.340 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:54.334+0000 7ff8097fa700 1 -- 172.21.15.81:0/2640234497 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff800009320 con 0x7ff80c0f78e0 2023-11-15T06:29:54.341 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:54.334+0000 7ff81184e700 1 -- 172.21.15.81:0/2640234497 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff80c0f57d0 con 0x7ff80c0f78e0 2023-11-15T06:29:54.341 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:54.334+0000 7ff8097fa700 1 -- 172.21.15.81:0/2640234497 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff800013f00 con 0x7ff80c0f78e0 2023-11-15T06:29:54.341 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:54.334+0000 7ff8097fa700 1 -- 172.21.15.81:0/2640234497 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff800014dc0 con 0x7ff80c0f78e0 2023-11-15T06:29:54.341 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:54.334+0000 7ff81184e700 1 -- 172.21.15.81:0/2640234497 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff80c0f5c60 con 0x7ff80c0f78e0 2023-11-15T06:29:54.341 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:54.335+0000 7ff8097fa700 1 -- 172.21.15.81:0/2640234497 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff8000095f0 con 0x7ff80c0f78e0 2023-11-15T06:29:54.341 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:54.335+0000 7ff81184e700 1 -- 172.21.15.81:0/2640234497 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff80c059100 con 0x7ff80c0f78e0 2023-11-15T06:29:54.342 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:54.335+0000 7ff8097fa700 1 --2- 172.21.15.81:0/2640234497 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff7fc02ff60 0x7ff7fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:29:54.342 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:54.335+0000 7ff8097fa700 1 -- 172.21.15.81:0/2640234497 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff800041040 con 0x7ff80c0f78e0 2023-11-15T06:29:54.342 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:54.335+0000 7ff80bfff700 1 --2- 172.21.15.81:0/2640234497 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff7fc02ff60 0x7ff7fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:29:54.342 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:54.335+0000 7ff80bfff700 1 --2- 172.21.15.81:0/2640234497 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff7fc02ff60 0x7ff7fc032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7ff7f80096a0 tx=0x7ff7f8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:29:54.343 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:54.339+0000 7ff8097fa700 1 -- 172.21.15.81:0/2640234497 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff800014420 con 0x7ff80c0f78e0 2023-11-15T06:29:54.550 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:54.547+0000 7ff81184e700 1 -- 172.21.15.81:0/2640234497 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff80c0fbce0 con 0x7ff80c0f78e0 2023-11-15T06:29:54.551 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:54.548+0000 7ff8097fa700 1 -- 172.21.15.81:0/2640234497 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff800010230 con 0x7ff80c0f78e0 2023-11-15T06:29:54.551 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:29:54.552 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:29:54.553 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:54.550+0000 7ff81184e700 1 -- 172.21.15.81:0/2640234497 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff7fc02ff60 msgr2=0x7ff7fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:54.553 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:54.550+0000 7ff81184e700 1 --2- 172.21.15.81:0/2640234497 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff7fc02ff60 0x7ff7fc032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7ff7f80096a0 tx=0x7ff7f8006b40).stop 2023-11-15T06:29:54.553 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:54.550+0000 7ff81184e700 1 -- 172.21.15.81:0/2640234497 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff80c0f78e0 msgr2=0x7ff80c0f71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:54.553 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:54.550+0000 7ff81184e700 1 --2- 172.21.15.81:0/2640234497 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff80c0f78e0 0x7ff80c0f71b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7ff800013510 tx=0x7ff80000b040).stop 2023-11-15T06:29:54.554 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:54.550+0000 7ff81184e700 1 -- 172.21.15.81:0/2640234497 shutdown_connections 2023-11-15T06:29:54.554 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:54.550+0000 7ff81184e700 1 --2- 172.21.15.81:0/2640234497 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff7fc02ff60 0x7ff7fc032420 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:54.554 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:54.550+0000 7ff81184e700 1 --2- 172.21.15.81:0/2640234497 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff80c0f78e0 0x7ff80c0f71b0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:54.554 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:54.550+0000 7ff81184e700 1 -- 172.21.15.81:0/2640234497 >> 172.21.15.81:0/2640234497 conn(0x7ff80c0f0120 msgr2=0x7ff80c0f0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:29:54.554 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:54.550+0000 7ff81184e700 1 -- 172.21.15.81:0/2640234497 shutdown_connections 2023-11-15T06:29:54.555 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:54.550+0000 7ff81184e700 1 -- 172.21.15.81:0/2640234497 wait complete. 2023-11-15T06:29:54.555 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:29:55.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:54 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:29:53.816469+0000 mgr.smithi062.rwurtk (mgr.14164) 121 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:29:55.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:54 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:29:53.917480+0000 mon.smithi062 (mon.0) 197 : audit [INF] from='mgr.14164 172.21.15.62:0/147479783' entity='mgr.smithi062.rwurtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi062.rwurtk/mirror_snapshot_schedule"}]: dispatch 2023-11-15T06:29:55.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:54 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:29:53.918991+0000 mon.smithi062 (mon.0) 198 : audit [INF] from='mgr.14164 172.21.15.62:0/147479783' entity='mgr.smithi062.rwurtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi062.rwurtk/trash_purge_schedule"}]: dispatch 2023-11-15T06:29:55.169 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:54 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:29:54.549973+0000 mon.smithi062 (mon.0) 199 : audit [DBG] from='client.? 172.21.15.81:0/2640234497' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:29:55.987 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:29:55.988 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:29:57.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:56 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:29:55.816763+0000 mgr.smithi062.rwurtk (mgr.14164) 122 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:29:57.253 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:57.250+0000 7f2abf59e700 1 -- 172.21.15.81:0/2167638128 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ac00f51a0 msgr2=0x7f2ac00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:57.254 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:57.250+0000 7f2abf59e700 1 --2- 172.21.15.81:0/2167638128 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ac00f51a0 0x7f2ac00f55c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f2ab0009fb0 tx=0x7f2ab0009b30).stop 2023-11-15T06:29:57.254 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:57.250+0000 7f2abf59e700 1 -- 172.21.15.81:0/2167638128 shutdown_connections 2023-11-15T06:29:57.254 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:57.250+0000 7f2abf59e700 1 --2- 172.21.15.81:0/2167638128 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ac00f51a0 0x7f2ac00f55c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:57.254 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:57.250+0000 7f2abf59e700 1 -- 172.21.15.81:0/2167638128 >> 172.21.15.81:0/2167638128 conn(0x7f2ac00f09a0 msgr2=0x7f2ac00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:29:57.254 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:57.250+0000 7f2abf59e700 1 -- 172.21.15.81:0/2167638128 shutdown_connections 2023-11-15T06:29:57.255 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:57.250+0000 7f2abf59e700 1 -- 172.21.15.81:0/2167638128 wait complete. 2023-11-15T06:29:57.255 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:57.250+0000 7f2abf59e700 1 Processor -- start 2023-11-15T06:29:57.255 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:57.250+0000 7f2abf59e700 1 -- start start 2023-11-15T06:29:57.255 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:57.251+0000 7f2abf59e700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ac00f51a0 0x7f2ac01053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:29:57.256 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:57.251+0000 7f2abf59e700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac01058a0 con 0x7f2ac00f51a0 2023-11-15T06:29:57.257 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:57.251+0000 7f2abe59c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ac00f51a0 0x7f2ac01053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:29:57.258 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:57.251+0000 7f2abe59c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ac00f51a0 0x7f2ac01053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:37526/0 (socket says 172.21.15.81:37526) 2023-11-15T06:29:57.258 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:57.251+0000 7f2abe59c700 1 -- 172.21.15.81:0/1140752264 learned_addr learned my addr 172.21.15.81:0/1140752264 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:29:57.258 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:57.251+0000 7f2abe59c700 1 -- 172.21.15.81:0/1140752264 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ab0005040 con 0x7f2ac00f51a0 2023-11-15T06:29:57.258 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:57.251+0000 7f2abe59c700 1 --2- 172.21.15.81:0/1140752264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ac00f51a0 0x7f2ac01053b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f2ab0013510 tx=0x7f2ab000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:29:57.258 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:57.252+0000 7f2aaeffd700 1 -- 172.21.15.81:0/1140752264 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ab0009320 con 0x7f2ac00f51a0 2023-11-15T06:29:57.259 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:57.252+0000 7f2abf59e700 1 -- 172.21.15.81:0/1140752264 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ac0105aa0 con 0x7f2ac00f51a0 2023-11-15T06:29:57.259 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:57.252+0000 7f2aaeffd700 1 -- 172.21.15.81:0/1140752264 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2ab0013f00 con 0x7f2ac00f51a0 2023-11-15T06:29:57.259 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:57.252+0000 7f2aaeffd700 1 -- 172.21.15.81:0/1140752264 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ab0014dc0 con 0x7f2ac00f51a0 2023-11-15T06:29:57.259 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:57.252+0000 7f2abf59e700 1 -- 172.21.15.81:0/1140752264 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ac0105f30 con 0x7f2ac00f51a0 2023-11-15T06:29:57.259 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:57.252+0000 7f2aaeffd700 1 -- 172.21.15.81:0/1140752264 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2ab0014420 con 0x7f2ac00f51a0 2023-11-15T06:29:57.260 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:57.252+0000 7f2abf59e700 1 -- 172.21.15.81:0/1140752264 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ac0045ad0 con 0x7f2ac00f51a0 2023-11-15T06:29:57.260 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:57.252+0000 7f2aaeffd700 1 --2- 172.21.15.81:0/1140752264 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f2aa802ff60 0x7f2aa8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:29:57.260 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:57.253+0000 7f2aaeffd700 1 -- 172.21.15.81:0/1140752264 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2ab0041110 con 0x7f2ac00f51a0 2023-11-15T06:29:57.260 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:57.253+0000 7f2abdd9b700 1 --2- 172.21.15.81:0/1140752264 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f2aa802ff60 0x7f2aa8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:29:57.260 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:57.253+0000 7f2abdd9b700 1 --2- 172.21.15.81:0/1140752264 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f2aa802ff60 0x7f2aa8032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f2ab40096a0 tx=0x7f2ab4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:29:57.261 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:57.258+0000 7f2aaeffd700 1 -- 172.21.15.81:0/1140752264 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2ab00095f0 con 0x7f2ac00f51a0 2023-11-15T06:29:57.463 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:57.460+0000 7f2abf59e700 1 -- 172.21.15.81:0/1140752264 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2ac0106de0 con 0x7f2ac00f51a0 2023-11-15T06:29:57.464 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:57.462+0000 7f2aaeffd700 1 -- 172.21.15.81:0/1140752264 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2ab001cb10 con 0x7f2ac00f51a0 2023-11-15T06:29:57.464 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:29:57.464 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:29:57.466 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:57.463+0000 7f2abf59e700 1 -- 172.21.15.81:0/1140752264 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f2aa802ff60 msgr2=0x7f2aa8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:57.466 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:57.463+0000 7f2abf59e700 1 --2- 172.21.15.81:0/1140752264 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f2aa802ff60 0x7f2aa8032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f2ab40096a0 tx=0x7f2ab4006b40).stop 2023-11-15T06:29:57.466 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:57.463+0000 7f2abf59e700 1 -- 172.21.15.81:0/1140752264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ac00f51a0 msgr2=0x7f2ac01053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:29:57.467 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:57.463+0000 7f2abf59e700 1 --2- 172.21.15.81:0/1140752264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ac00f51a0 0x7f2ac01053b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f2ab0013510 tx=0x7f2ab000b040).stop 2023-11-15T06:29:57.467 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:57.463+0000 7f2abf59e700 1 -- 172.21.15.81:0/1140752264 shutdown_connections 2023-11-15T06:29:57.467 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:57.463+0000 7f2abf59e700 1 --2- 172.21.15.81:0/1140752264 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f2aa802ff60 0x7f2aa8032420 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:57.467 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:57.463+0000 7f2abf59e700 1 --2- 172.21.15.81:0/1140752264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ac00f51a0 0x7f2ac01053b0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:29:57.468 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:57.463+0000 7f2abf59e700 1 -- 172.21.15.81:0/1140752264 >> 172.21.15.81:0/1140752264 conn(0x7f2ac00f09a0 msgr2=0x7f2ac00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:29:57.468 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:57.463+0000 7f2abf59e700 1 -- 172.21.15.81:0/1140752264 shutdown_connections 2023-11-15T06:29:57.468 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:29:57.463+0000 7f2abf59e700 1 -- 172.21.15.81:0/1140752264 wait complete. 2023-11-15T06:29:57.468 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:29:58.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:57 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:29:57.462897+0000 mon.smithi062 (mon.0) 200 : audit [DBG] from='client.? 172.21.15.81:0/1140752264' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:29:58.832 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:29:58.833 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:29:59.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:29:58 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:29:57.817069+0000 mgr.smithi062.rwurtk (mgr.14164) 123 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:30:00.006 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:00.003+0000 7f799fd07700 1 -- 172.21.15.81:0/1667039196 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79980f5410 msgr2=0x7f79980f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:00.006 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:00.003+0000 7f799fd07700 1 --2- 172.21.15.81:0/1667039196 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79980f5410 0x7f79980f5830 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f7988009fb0 tx=0x7f7988009b30).stop 2023-11-15T06:30:00.006 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:00.003+0000 7f799fd07700 1 -- 172.21.15.81:0/1667039196 shutdown_connections 2023-11-15T06:30:00.007 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:00.003+0000 7f799fd07700 1 --2- 172.21.15.81:0/1667039196 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79980f5410 0x7f79980f5830 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:00.007 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:00.003+0000 7f799fd07700 1 -- 172.21.15.81:0/1667039196 >> 172.21.15.81:0/1667039196 conn(0x7f79980f0b70 msgr2=0x7f79980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:30:00.007 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:00.003+0000 7f799fd07700 1 -- 172.21.15.81:0/1667039196 shutdown_connections 2023-11-15T06:30:00.007 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:00.003+0000 7f799fd07700 1 -- 172.21.15.81:0/1667039196 wait complete. 2023-11-15T06:30:00.007 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:00.003+0000 7f799fd07700 1 Processor -- start 2023-11-15T06:30:00.008 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:00.003+0000 7f799fd07700 1 -- start start 2023-11-15T06:30:00.008 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:00.003+0000 7f799fd07700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79980f5410 0x7f7998109420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:30:00.008 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:00.003+0000 7f799fd07700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7998060220 con 0x7f79980f5410 2023-11-15T06:30:00.008 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:00.004+0000 7f799daa3700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79980f5410 0x7f7998109420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:30:00.008 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:00.004+0000 7f799daa3700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79980f5410 0x7f7998109420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:37548/0 (socket says 172.21.15.81:37548) 2023-11-15T06:30:00.009 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:00.004+0000 7f799daa3700 1 -- 172.21.15.81:0/2254264756 learned_addr learned my addr 172.21.15.81:0/2254264756 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:30:00.009 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:00.004+0000 7f799daa3700 1 -- 172.21.15.81:0/2254264756 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7988005040 con 0x7f79980f5410 2023-11-15T06:30:00.009 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:00.004+0000 7f799daa3700 1 --2- 172.21.15.81:0/2254264756 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79980f5410 0x7f7998109420 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f7988013510 tx=0x7f798800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:30:00.011 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:00.004+0000 7f798e7fc700 1 -- 172.21.15.81:0/2254264756 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7988009320 con 0x7f79980f5410 2023-11-15T06:30:00.011 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:00.004+0000 7f798e7fc700 1 -- 172.21.15.81:0/2254264756 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7988013f00 con 0x7f79980f5410 2023-11-15T06:30:00.011 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:00.004+0000 7f798e7fc700 1 -- 172.21.15.81:0/2254264756 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7988014d90 con 0x7f79980f5410 2023-11-15T06:30:00.011 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:00.004+0000 7f799fd07700 1 -- 172.21.15.81:0/2254264756 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7998060420 con 0x7f79980f5410 2023-11-15T06:30:00.012 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:00.005+0000 7f799fd07700 1 -- 172.21.15.81:0/2254264756 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7998109ad0 con 0x7f79980f5410 2023-11-15T06:30:00.012 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:00.005+0000 7f798e7fc700 1 -- 172.21.15.81:0/2254264756 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f79880263a0 con 0x7f79980f5410 2023-11-15T06:30:00.012 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:00.005+0000 7f798e7fc700 1 --2- 172.21.15.81:0/2254264756 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f798402ff60 0x7f7984032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:30:00.012 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:00.005+0000 7f798e7fc700 1 -- 172.21.15.81:0/2254264756 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f79880412d0 con 0x7f79980f5410 2023-11-15T06:30:00.012 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:00.005+0000 7f799fd07700 1 -- 172.21.15.81:0/2254264756 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7998059100 con 0x7f79980f5410 2023-11-15T06:30:00.013 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:00.006+0000 7f799d2a2700 1 --2- 172.21.15.81:0/2254264756 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f798402ff60 0x7f7984032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:30:00.013 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:00.006+0000 7f799d2a2700 1 --2- 172.21.15.81:0/2254264756 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f798402ff60 0x7f7984032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f79940096a0 tx=0x7f7994006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:30:00.014 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:00.010+0000 7f798e7fc700 1 -- 172.21.15.81:0/2254264756 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7988012920 con 0x7f79980f5410 2023-11-15T06:30:00.220 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:00.218+0000 7f799fd07700 1 -- 172.21.15.81:0/2254264756 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f799810a180 con 0x7f79980f5410 2023-11-15T06:30:00.220 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:00.218+0000 7f798e7fc700 1 -- 172.21.15.81:0/2254264756 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f798802b030 con 0x7f79980f5410 2023-11-15T06:30:00.221 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:30:00.221 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:30:00.222 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:00.220+0000 7f799fd07700 1 -- 172.21.15.81:0/2254264756 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f798402ff60 msgr2=0x7f7984032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:00.222 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:00.220+0000 7f799fd07700 1 --2- 172.21.15.81:0/2254264756 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f798402ff60 0x7f7984032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f79940096a0 tx=0x7f7994006b40).stop 2023-11-15T06:30:00.222 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:00.220+0000 7f799fd07700 1 -- 172.21.15.81:0/2254264756 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79980f5410 msgr2=0x7f7998109420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:00.223 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:00.220+0000 7f799fd07700 1 --2- 172.21.15.81:0/2254264756 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79980f5410 0x7f7998109420 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f7988013510 tx=0x7f798800b040).stop 2023-11-15T06:30:00.223 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:00.220+0000 7f799fd07700 1 -- 172.21.15.81:0/2254264756 shutdown_connections 2023-11-15T06:30:00.223 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:00.220+0000 7f799fd07700 1 --2- 172.21.15.81:0/2254264756 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f798402ff60 0x7f7984032420 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:00.223 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:00.220+0000 7f799fd07700 1 --2- 172.21.15.81:0/2254264756 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79980f5410 0x7f7998109420 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:00.224 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:00.220+0000 7f799fd07700 1 -- 172.21.15.81:0/2254264756 >> 172.21.15.81:0/2254264756 conn(0x7f79980f0b70 msgr2=0x7f79980f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:30:00.224 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:00.220+0000 7f799fd07700 1 -- 172.21.15.81:0/2254264756 shutdown_connections 2023-11-15T06:30:00.224 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:00.220+0000 7f799fd07700 1 -- 172.21.15.81:0/2254264756 wait complete. 2023-11-15T06:30:00.224 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:30:01.171 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:00 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:29:59.817495+0000 mgr.smithi062.rwurtk (mgr.14164) 124 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:30:01.171 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:00 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:30:00.219162+0000 mon.smithi062 (mon.0) 201 : audit [DBG] from='client.? 172.21.15.81:0/2254264756' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:30:01.608 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:30:01.609 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:30:02.796 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:02.791+0000 7fa017e50700 1 -- 172.21.15.81:0/3684206744 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0100f5410 msgr2=0x7fa0100f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:02.796 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:02.791+0000 7fa017e50700 1 --2- 172.21.15.81:0/3684206744 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0100f5410 0x7fa0100f5830 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7fa000009fb0 tx=0x7fa000009b30).stop 2023-11-15T06:30:02.796 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:02.791+0000 7fa017e50700 1 -- 172.21.15.81:0/3684206744 shutdown_connections 2023-11-15T06:30:02.797 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:02.791+0000 7fa017e50700 1 --2- 172.21.15.81:0/3684206744 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0100f5410 0x7fa0100f5830 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:02.797 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:02.791+0000 7fa017e50700 1 -- 172.21.15.81:0/3684206744 >> 172.21.15.81:0/3684206744 conn(0x7fa0100f0b70 msgr2=0x7fa0100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:30:02.797 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:02.792+0000 7fa017e50700 1 -- 172.21.15.81:0/3684206744 shutdown_connections 2023-11-15T06:30:02.797 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:02.792+0000 7fa017e50700 1 -- 172.21.15.81:0/3684206744 wait complete. 2023-11-15T06:30:02.798 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:02.792+0000 7fa017e50700 1 Processor -- start 2023-11-15T06:30:02.798 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:02.792+0000 7fa017e50700 1 -- start start 2023-11-15T06:30:02.798 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:02.792+0000 7fa017e50700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0100fd360 0x7fa0100fb820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:30:02.798 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:02.792+0000 7fa017e50700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0100fd780 con 0x7fa0100fd360 2023-11-15T06:30:02.799 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:02.792+0000 7fa015bec700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0100fd360 0x7fa0100fb820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:30:02.799 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:02.792+0000 7fa015bec700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0100fd360 0x7fa0100fb820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:39192/0 (socket says 172.21.15.81:39192) 2023-11-15T06:30:02.799 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:02.792+0000 7fa015bec700 1 -- 172.21.15.81:0/2567796331 learned_addr learned my addr 172.21.15.81:0/2567796331 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:30:02.799 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:02.793+0000 7fa015bec700 1 -- 172.21.15.81:0/2567796331 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa000005040 con 0x7fa0100fd360 2023-11-15T06:30:02.800 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:02.793+0000 7fa015bec700 1 --2- 172.21.15.81:0/2567796331 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0100fd360 0x7fa0100fb820 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fa000009f80 tx=0x7fa000009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:30:02.800 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:02.793+0000 7fa0067fc700 1 -- 172.21.15.81:0/2567796331 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0000136c0 con 0x7fa0100fd360 2023-11-15T06:30:02.800 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:02.793+0000 7fa0067fc700 1 -- 172.21.15.81:0/2567796331 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa000013e70 con 0x7fa0100fd360 2023-11-15T06:30:02.800 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:02.793+0000 7fa017e50700 1 -- 172.21.15.81:0/2567796331 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0100fbd10 con 0x7fa0100fd360 2023-11-15T06:30:02.800 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:02.793+0000 7fa0067fc700 1 -- 172.21.15.81:0/2567796331 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa000014420 con 0x7fa0100fd360 2023-11-15T06:30:02.801 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:02.793+0000 7fa017e50700 1 -- 172.21.15.81:0/2567796331 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0100fc200 con 0x7fa0100fd360 2023-11-15T06:30:02.801 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:02.794+0000 7fa0067fc700 1 -- 172.21.15.81:0/2567796331 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa000025060 con 0x7fa0100fd360 2023-11-15T06:30:02.801 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:02.794+0000 7fa017e50700 1 -- 172.21.15.81:0/2567796331 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa010045ad0 con 0x7fa0100fd360 2023-11-15T06:30:02.801 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:02.794+0000 7fa0067fc700 1 --2- 172.21.15.81:0/2567796331 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f9ffc02ffb0 0x7f9ffc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:30:02.801 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:02.794+0000 7fa0067fc700 1 -- 172.21.15.81:0/2567796331 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa000040e80 con 0x7fa0100fd360 2023-11-15T06:30:02.802 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:02.794+0000 7fa0153eb700 1 --2- 172.21.15.81:0/2567796331 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f9ffc02ffb0 0x7f9ffc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:30:02.802 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:02.795+0000 7fa0153eb700 1 --2- 172.21.15.81:0/2567796331 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f9ffc02ffb0 0x7f9ffc032470 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fa00c0096a0 tx=0x7fa00c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:30:02.803 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:02.799+0000 7fa0067fc700 1 -- 172.21.15.81:0/2567796331 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa00001cb30 con 0x7fa0100fd360 2023-11-15T06:30:03.012 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:03.010+0000 7fa017e50700 1 -- 172.21.15.81:0/2567796331 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa01010ca20 con 0x7fa0100fd360 2023-11-15T06:30:03.012 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:03.010+0000 7fa0067fc700 1 -- 172.21.15.81:0/2567796331 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa00002d2d0 con 0x7fa0100fd360 2023-11-15T06:30:03.013 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:30:03.013 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:30:03.014 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:03.012+0000 7fa017e50700 1 -- 172.21.15.81:0/2567796331 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f9ffc02ffb0 msgr2=0x7f9ffc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:03.014 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:03.012+0000 7fa017e50700 1 --2- 172.21.15.81:0/2567796331 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f9ffc02ffb0 0x7f9ffc032470 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fa00c0096a0 tx=0x7fa00c006b40).stop 2023-11-15T06:30:03.015 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:03.012+0000 7fa017e50700 1 -- 172.21.15.81:0/2567796331 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0100fd360 msgr2=0x7fa0100fb820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:03.015 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:03.012+0000 7fa017e50700 1 --2- 172.21.15.81:0/2567796331 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0100fd360 0x7fa0100fb820 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fa000009f80 tx=0x7fa000009b30).stop 2023-11-15T06:30:03.015 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:03.012+0000 7fa017e50700 1 -- 172.21.15.81:0/2567796331 shutdown_connections 2023-11-15T06:30:03.015 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:03.012+0000 7fa017e50700 1 --2- 172.21.15.81:0/2567796331 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f9ffc02ffb0 0x7f9ffc032470 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:03.016 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:03.012+0000 7fa017e50700 1 --2- 172.21.15.81:0/2567796331 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0100fd360 0x7fa0100fb820 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:03.016 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:03.012+0000 7fa017e50700 1 -- 172.21.15.81:0/2567796331 >> 172.21.15.81:0/2567796331 conn(0x7fa0100f0b70 msgr2=0x7fa01005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:30:03.016 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:03.012+0000 7fa017e50700 1 -- 172.21.15.81:0/2567796331 shutdown_connections 2023-11-15T06:30:03.016 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:03.012+0000 7fa017e50700 1 -- 172.21.15.81:0/2567796331 wait complete. 2023-11-15T06:30:03.017 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:30:03.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:30:01.817807+0000 mgr.smithi062.rwurtk (mgr.14164) 125 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:30:04.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:03 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:30:03.011189+0000 mon.smithi062 (mon.0) 202 : audit [DBG] from='client.? 172.21.15.81:0/2567796331' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:30:04.435 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:30:04.435 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:30:05.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:04 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:30:03.818123+0000 mgr.smithi062.rwurtk (mgr.14164) 126 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:30:05.687 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:05.684+0000 7fa556b87700 1 -- 172.21.15.81:0/3925152445 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5500f51a0 msgr2=0x7fa5500f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:05.688 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:05.684+0000 7fa556b87700 1 --2- 172.21.15.81:0/3925152445 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5500f51a0 0x7fa5500f55c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7fa54c009fb0 tx=0x7fa54c009b30).stop 2023-11-15T06:30:05.688 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:05.684+0000 7fa556b87700 1 -- 172.21.15.81:0/3925152445 shutdown_connections 2023-11-15T06:30:05.688 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:05.684+0000 7fa556b87700 1 --2- 172.21.15.81:0/3925152445 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5500f51a0 0x7fa5500f55c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:05.688 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:05.684+0000 7fa556b87700 1 -- 172.21.15.81:0/3925152445 >> 172.21.15.81:0/3925152445 conn(0x7fa5500f09a0 msgr2=0x7fa5500f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:30:05.688 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:05.684+0000 7fa556b87700 1 -- 172.21.15.81:0/3925152445 shutdown_connections 2023-11-15T06:30:05.689 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:05.684+0000 7fa556b87700 1 -- 172.21.15.81:0/3925152445 wait complete. 2023-11-15T06:30:05.689 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:05.684+0000 7fa556b87700 1 Processor -- start 2023-11-15T06:30:05.689 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:05.684+0000 7fa556b87700 1 -- start start 2023-11-15T06:30:05.689 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:05.685+0000 7fa556b87700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5500f51a0 0x7fa5501091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:30:05.689 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:05.685+0000 7fa556b87700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa550060240 con 0x7fa5500f51a0 2023-11-15T06:30:05.690 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:05.685+0000 7fa555b85700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5500f51a0 0x7fa5501091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:30:05.690 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:05.685+0000 7fa555b85700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5500f51a0 0x7fa5501091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:39212/0 (socket says 172.21.15.81:39212) 2023-11-15T06:30:05.690 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:05.685+0000 7fa555b85700 1 -- 172.21.15.81:0/3392187049 learned_addr learned my addr 172.21.15.81:0/3392187049 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:30:05.690 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:05.685+0000 7fa555b85700 1 -- 172.21.15.81:0/3392187049 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa54c005040 con 0x7fa5500f51a0 2023-11-15T06:30:05.690 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:05.685+0000 7fa555b85700 1 --2- 172.21.15.81:0/3392187049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5500f51a0 0x7fa5501091b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fa54c013510 tx=0x7fa54c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:30:05.691 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:05.685+0000 7fa5467fc700 1 -- 172.21.15.81:0/3392187049 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa54c009320 con 0x7fa5500f51a0 2023-11-15T06:30:05.691 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:05.685+0000 7fa556b87700 1 -- 172.21.15.81:0/3392187049 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa550060440 con 0x7fa5500f51a0 2023-11-15T06:30:05.691 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:05.685+0000 7fa5467fc700 1 -- 172.21.15.81:0/3392187049 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa54c013f00 con 0x7fa5500f51a0 2023-11-15T06:30:05.691 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:05.685+0000 7fa5467fc700 1 -- 172.21.15.81:0/3392187049 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa54c014dc0 con 0x7fa5500f51a0 2023-11-15T06:30:05.691 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:05.685+0000 7fa556b87700 1 -- 172.21.15.81:0/3392187049 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa550109860 con 0x7fa5500f51a0 2023-11-15T06:30:05.692 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:05.686+0000 7fa5467fc700 1 -- 172.21.15.81:0/3392187049 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa54c014420 con 0x7fa5500f51a0 2023-11-15T06:30:05.693 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:05.686+0000 7fa556b87700 1 -- 172.21.15.81:0/3392187049 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa550045ad0 con 0x7fa5500f51a0 2023-11-15T06:30:05.693 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:05.686+0000 7fa5467fc700 1 --2- 172.21.15.81:0/3392187049 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fa53c02ffb0 0x7fa53c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:30:05.694 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:05.686+0000 7fa5467fc700 1 -- 172.21.15.81:0/3392187049 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa54c041110 con 0x7fa5500f51a0 2023-11-15T06:30:05.694 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:05.687+0000 7fa555384700 1 --2- 172.21.15.81:0/3392187049 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fa53c02ffb0 0x7fa53c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:30:05.694 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:05.687+0000 7fa555384700 1 --2- 172.21.15.81:0/3392187049 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fa53c02ffb0 0x7fa53c032470 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fa5400096a0 tx=0x7fa540006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:30:05.695 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:05.691+0000 7fa5467fc700 1 -- 172.21.15.81:0/3392187049 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa54c0095f0 con 0x7fa5500f51a0 2023-11-15T06:30:05.897 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:05.895+0000 7fa556b87700 1 -- 172.21.15.81:0/3392187049 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa55010a080 con 0x7fa5500f51a0 2023-11-15T06:30:05.897 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:05.896+0000 7fa5467fc700 1 -- 172.21.15.81:0/3392187049 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa54c01cb10 con 0x7fa5500f51a0 2023-11-15T06:30:05.898 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:30:05.898 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:30:05.899 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:05.897+0000 7fa556b87700 1 -- 172.21.15.81:0/3392187049 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fa53c02ffb0 msgr2=0x7fa53c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:05.899 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:05.897+0000 7fa556b87700 1 --2- 172.21.15.81:0/3392187049 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fa53c02ffb0 0x7fa53c032470 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fa5400096a0 tx=0x7fa540006b40).stop 2023-11-15T06:30:05.900 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:05.897+0000 7fa556b87700 1 -- 172.21.15.81:0/3392187049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5500f51a0 msgr2=0x7fa5501091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:05.900 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:05.897+0000 7fa556b87700 1 --2- 172.21.15.81:0/3392187049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5500f51a0 0x7fa5501091b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fa54c013510 tx=0x7fa54c00b040).stop 2023-11-15T06:30:05.900 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:05.897+0000 7fa556b87700 1 -- 172.21.15.81:0/3392187049 shutdown_connections 2023-11-15T06:30:05.900 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:05.897+0000 7fa556b87700 1 --2- 172.21.15.81:0/3392187049 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fa53c02ffb0 0x7fa53c032470 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:05.901 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:05.897+0000 7fa556b87700 1 --2- 172.21.15.81:0/3392187049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5500f51a0 0x7fa5501091b0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:05.901 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:05.897+0000 7fa556b87700 1 -- 172.21.15.81:0/3392187049 >> 172.21.15.81:0/3392187049 conn(0x7fa5500f09a0 msgr2=0x7fa5500f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:30:05.901 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:05.897+0000 7fa556b87700 1 -- 172.21.15.81:0/3392187049 shutdown_connections 2023-11-15T06:30:05.902 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:05.897+0000 7fa556b87700 1 -- 172.21.15.81:0/3392187049 wait complete. 2023-11-15T06:30:05.902 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:30:07.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:06 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:30:05.818434+0000 mgr.smithi062.rwurtk (mgr.14164) 127 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:30:07.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:06 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:30:05.896466+0000 mon.smithi062 (mon.0) 203 : audit [DBG] from='client.? 172.21.15.81:0/3392187049' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:30:07.281 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:30:07.282 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:30:08.520 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:08.517+0000 7f6ba933a700 1 -- 172.21.15.81:0/3567835055 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ba40f6510 msgr2=0x7f6ba40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:08.520 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:08.517+0000 7f6ba933a700 1 --2- 172.21.15.81:0/3567835055 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ba40f6510 0x7f6ba40f6930 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f6b94009fb0 tx=0x7f6b94009b30).stop 2023-11-15T06:30:08.521 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:08.517+0000 7f6ba933a700 1 -- 172.21.15.81:0/3567835055 shutdown_connections 2023-11-15T06:30:08.521 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:08.517+0000 7f6ba933a700 1 --2- 172.21.15.81:0/3567835055 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ba40f6510 0x7f6ba40f6930 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:08.521 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:08.517+0000 7f6ba933a700 1 -- 172.21.15.81:0/3567835055 >> 172.21.15.81:0/3567835055 conn(0x7f6ba40f1c70 msgr2=0x7f6ba40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:30:08.521 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:08.518+0000 7f6ba933a700 1 -- 172.21.15.81:0/3567835055 shutdown_connections 2023-11-15T06:30:08.521 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:08.518+0000 7f6ba933a700 1 -- 172.21.15.81:0/3567835055 wait complete. 2023-11-15T06:30:08.522 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:08.518+0000 7f6ba933a700 1 Processor -- start 2023-11-15T06:30:08.522 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:08.518+0000 7f6ba933a700 1 -- start start 2023-11-15T06:30:08.522 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:08.518+0000 7f6ba933a700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ba40f6510 0x7f6ba4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:30:08.522 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:08.518+0000 7f6ba933a700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba4106c20 con 0x7f6ba40f6510 2023-11-15T06:30:08.524 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:08.519+0000 7f6ba2ffd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ba40f6510 0x7f6ba4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:30:08.524 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:08.519+0000 7f6ba2ffd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ba40f6510 0x7f6ba4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:39228/0 (socket says 172.21.15.81:39228) 2023-11-15T06:30:08.524 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:08.519+0000 7f6ba2ffd700 1 -- 172.21.15.81:0/2287658859 learned_addr learned my addr 172.21.15.81:0/2287658859 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:30:08.525 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:08.519+0000 7f6ba2ffd700 1 -- 172.21.15.81:0/2287658859 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b94005040 con 0x7f6ba40f6510 2023-11-15T06:30:08.525 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:08.519+0000 7f6ba2ffd700 1 --2- 172.21.15.81:0/2287658859 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ba40f6510 0x7f6ba4106730 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f6b94013510 tx=0x7f6b9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:30:08.525 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:08.519+0000 7f6b8b7fe700 1 -- 172.21.15.81:0/2287658859 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b94009320 con 0x7f6ba40f6510 2023-11-15T06:30:08.525 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:08.519+0000 7f6ba933a700 1 -- 172.21.15.81:0/2287658859 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ba4106e20 con 0x7f6ba40f6510 2023-11-15T06:30:08.525 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:08.519+0000 7f6b8b7fe700 1 -- 172.21.15.81:0/2287658859 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6b94013f00 con 0x7f6ba40f6510 2023-11-15T06:30:08.526 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:08.519+0000 7f6b8b7fe700 1 -- 172.21.15.81:0/2287658859 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b94014d90 con 0x7f6ba40f6510 2023-11-15T06:30:08.526 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:08.520+0000 7f6ba933a700 1 -- 172.21.15.81:0/2287658859 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ba41072b0 con 0x7f6ba40f6510 2023-11-15T06:30:08.526 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:08.520+0000 7f6b8b7fe700 1 -- 172.21.15.81:0/2287658859 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6b940263a0 con 0x7f6ba40f6510 2023-11-15T06:30:08.526 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:08.520+0000 7f6b8b7fe700 1 --2- 172.21.15.81:0/2287658859 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f6b8c02ff60 0x7f6b8c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:30:08.526 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:08.520+0000 7f6b8b7fe700 1 -- 172.21.15.81:0/2287658859 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6b94041020 con 0x7f6ba40f6510 2023-11-15T06:30:08.527 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:08.520+0000 7f6ba933a700 1 -- 172.21.15.81:0/2287658859 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ba4045ad0 con 0x7f6ba40f6510 2023-11-15T06:30:08.527 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:08.521+0000 7f6ba27fc700 1 --2- 172.21.15.81:0/2287658859 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f6b8c02ff60 0x7f6b8c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:30:08.527 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:08.521+0000 7f6ba27fc700 1 --2- 172.21.15.81:0/2287658859 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f6b8c02ff60 0x7f6b8c032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f6b980096a0 tx=0x7f6b98006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:30:08.530 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:08.525+0000 7f6b8b7fe700 1 -- 172.21.15.81:0/2287658859 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6b94012920 con 0x7f6ba40f6510 2023-11-15T06:30:08.730 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:08.728+0000 7f6ba933a700 1 -- 172.21.15.81:0/2287658859 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6ba40fc050 con 0x7f6ba40f6510 2023-11-15T06:30:08.731 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:08.729+0000 7f6b8b7fe700 1 -- 172.21.15.81:0/2287658859 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6b9402b020 con 0x7f6ba40f6510 2023-11-15T06:30:08.731 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:30:08.731 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:30:08.732 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:08.731+0000 7f6ba933a700 1 -- 172.21.15.81:0/2287658859 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f6b8c02ff60 msgr2=0x7f6b8c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:08.733 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:08.731+0000 7f6ba933a700 1 --2- 172.21.15.81:0/2287658859 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f6b8c02ff60 0x7f6b8c032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f6b980096a0 tx=0x7f6b98006b40).stop 2023-11-15T06:30:08.733 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:08.731+0000 7f6ba933a700 1 -- 172.21.15.81:0/2287658859 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ba40f6510 msgr2=0x7f6ba4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:08.733 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:08.731+0000 7f6ba933a700 1 --2- 172.21.15.81:0/2287658859 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ba40f6510 0x7f6ba4106730 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f6b94013510 tx=0x7f6b9400b040).stop 2023-11-15T06:30:08.733 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:08.731+0000 7f6ba933a700 1 -- 172.21.15.81:0/2287658859 shutdown_connections 2023-11-15T06:30:08.733 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:08.731+0000 7f6ba933a700 1 --2- 172.21.15.81:0/2287658859 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f6b8c02ff60 0x7f6b8c032420 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:08.734 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:08.731+0000 7f6ba933a700 1 --2- 172.21.15.81:0/2287658859 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ba40f6510 0x7f6ba4106730 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:08.734 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:08.731+0000 7f6ba933a700 1 -- 172.21.15.81:0/2287658859 >> 172.21.15.81:0/2287658859 conn(0x7f6ba40f1c70 msgr2=0x7f6ba40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:30:08.734 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:08.731+0000 7f6ba933a700 1 -- 172.21.15.81:0/2287658859 shutdown_connections 2023-11-15T06:30:08.734 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:08.731+0000 7f6ba933a700 1 -- 172.21.15.81:0/2287658859 wait complete. 2023-11-15T06:30:08.735 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:30:08.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:08 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:30:07.818743+0000 mgr.smithi062.rwurtk (mgr.14164) 128 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:30:09.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:09 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:30:08.729712+0000 mon.smithi062 (mon.0) 204 : audit [DBG] from='client.? 172.21.15.81:0/2287658859' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:30:10.115 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:30:10.116 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:30:10.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:10 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:30:09.819014+0000 mgr.smithi062.rwurtk (mgr.14164) 129 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:30:11.303 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:11.301+0000 7f2deda78700 1 -- 172.21.15.81:0/1521433945 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2de80f62a0 msgr2=0x7f2de80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:11.304 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:11.301+0000 7f2deda78700 1 --2- 172.21.15.81:0/1521433945 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2de80f62a0 0x7f2de80f66c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f2ddc009fb0 tx=0x7f2ddc009b30).stop 2023-11-15T06:30:11.304 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:11.301+0000 7f2deda78700 1 -- 172.21.15.81:0/1521433945 shutdown_connections 2023-11-15T06:30:11.304 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:11.301+0000 7f2deda78700 1 --2- 172.21.15.81:0/1521433945 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2de80f62a0 0x7f2de80f66c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:11.304 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:11.301+0000 7f2deda78700 1 -- 172.21.15.81:0/1521433945 >> 172.21.15.81:0/1521433945 conn(0x7f2de80f1aa0 msgr2=0x7f2de80f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:30:11.305 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:11.301+0000 7f2deda78700 1 -- 172.21.15.81:0/1521433945 shutdown_connections 2023-11-15T06:30:11.305 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:11.301+0000 7f2deda78700 1 -- 172.21.15.81:0/1521433945 wait complete. 2023-11-15T06:30:11.305 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:11.302+0000 7f2deda78700 1 Processor -- start 2023-11-15T06:30:11.305 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:11.302+0000 7f2deda78700 1 -- start start 2023-11-15T06:30:11.307 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:11.302+0000 7f2deda78700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2de80f62a0 0x7f2de81064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:30:11.307 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:11.302+0000 7f2deda78700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de81069a0 con 0x7f2de80f62a0 2023-11-15T06:30:11.307 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:11.302+0000 7f2deca76700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2de80f62a0 0x7f2de81064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:30:11.308 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:11.302+0000 7f2deca76700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2de80f62a0 0x7f2de81064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:39254/0 (socket says 172.21.15.81:39254) 2023-11-15T06:30:11.308 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:11.302+0000 7f2deca76700 1 -- 172.21.15.81:0/114627413 learned_addr learned my addr 172.21.15.81:0/114627413 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:30:11.308 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:11.303+0000 7f2deca76700 1 -- 172.21.15.81:0/114627413 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ddc005040 con 0x7f2de80f62a0 2023-11-15T06:30:11.308 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:11.303+0000 7f2deca76700 1 --2- 172.21.15.81:0/114627413 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2de80f62a0 0x7f2de81064b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f2ddc013510 tx=0x7f2ddc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:30:11.308 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:11.303+0000 7f2de57fa700 1 -- 172.21.15.81:0/114627413 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ddc009320 con 0x7f2de80f62a0 2023-11-15T06:30:11.309 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:11.303+0000 7f2deda78700 1 -- 172.21.15.81:0/114627413 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2de8106ba0 con 0x7f2de80f62a0 2023-11-15T06:30:11.309 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:11.303+0000 7f2de57fa700 1 -- 172.21.15.81:0/114627413 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2ddc013f00 con 0x7f2de80f62a0 2023-11-15T06:30:11.309 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:11.303+0000 7f2de57fa700 1 -- 172.21.15.81:0/114627413 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ddc014dc0 con 0x7f2de80f62a0 2023-11-15T06:30:11.309 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:11.303+0000 7f2deda78700 1 -- 172.21.15.81:0/114627413 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2de8107030 con 0x7f2de80f62a0 2023-11-15T06:30:11.309 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:11.304+0000 7f2de57fa700 1 -- 172.21.15.81:0/114627413 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2ddc01d3d0 con 0x7f2de80f62a0 2023-11-15T06:30:11.310 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:11.304+0000 7f2deda78700 1 -- 172.21.15.81:0/114627413 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2de8045ad0 con 0x7f2de80f62a0 2023-11-15T06:30:11.310 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:11.304+0000 7f2de57fa700 1 --2- 172.21.15.81:0/114627413 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f2dd802ff60 0x7f2dd8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:30:11.310 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:11.304+0000 7f2de57fa700 1 -- 172.21.15.81:0/114627413 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2ddc041000 con 0x7f2de80f62a0 2023-11-15T06:30:11.310 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:11.304+0000 7f2de7fff700 1 --2- 172.21.15.81:0/114627413 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f2dd802ff60 0x7f2dd8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:30:11.310 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:11.304+0000 7f2de7fff700 1 --2- 172.21.15.81:0/114627413 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f2dd802ff60 0x7f2dd8032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f2dd40096a0 tx=0x7f2dd4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:30:11.311 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:11.309+0000 7f2de57fa700 1 -- 172.21.15.81:0/114627413 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2ddc01cb10 con 0x7f2de80f62a0 2023-11-15T06:30:11.516 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:11.514+0000 7f2deda78700 1 -- 172.21.15.81:0/114627413 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2de8107ee0 con 0x7f2de80f62a0 2023-11-15T06:30:11.516 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:11.515+0000 7f2de57fa700 1 -- 172.21.15.81:0/114627413 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2ddc01cb10 con 0x7f2de80f62a0 2023-11-15T06:30:11.517 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:30:11.517 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:30:11.519 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:11.516+0000 7f2deda78700 1 -- 172.21.15.81:0/114627413 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f2dd802ff60 msgr2=0x7f2dd8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:11.519 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:11.516+0000 7f2deda78700 1 --2- 172.21.15.81:0/114627413 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f2dd802ff60 0x7f2dd8032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f2dd40096a0 tx=0x7f2dd4006b40).stop 2023-11-15T06:30:11.520 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:11.517+0000 7f2deda78700 1 -- 172.21.15.81:0/114627413 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2de80f62a0 msgr2=0x7f2de81064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:11.520 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:11.517+0000 7f2deda78700 1 --2- 172.21.15.81:0/114627413 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2de80f62a0 0x7f2de81064b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f2ddc013510 tx=0x7f2ddc00b040).stop 2023-11-15T06:30:11.520 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:11.517+0000 7f2deda78700 1 -- 172.21.15.81:0/114627413 shutdown_connections 2023-11-15T06:30:11.520 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:11.517+0000 7f2deda78700 1 --2- 172.21.15.81:0/114627413 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f2dd802ff60 0x7f2dd8032420 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:11.520 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:11.517+0000 7f2deda78700 1 --2- 172.21.15.81:0/114627413 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2de80f62a0 0x7f2de81064b0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:11.521 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:11.517+0000 7f2deda78700 1 -- 172.21.15.81:0/114627413 >> 172.21.15.81:0/114627413 conn(0x7f2de80f1aa0 msgr2=0x7f2de80f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:30:11.521 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:11.517+0000 7f2deda78700 1 -- 172.21.15.81:0/114627413 shutdown_connections 2023-11-15T06:30:11.521 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:11.517+0000 7f2deda78700 1 -- 172.21.15.81:0/114627413 wait complete. 2023-11-15T06:30:11.521 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:30:11.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:11 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:30:11.515461+0000 mon.smithi062 (mon.0) 205 : audit [DBG] from='client.? 172.21.15.81:0/114627413' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:30:12.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:12 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:30:11.819233+0000 mgr.smithi062.rwurtk (mgr.14164) 130 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:30:12.941 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:30:12.942 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:30:14.224 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:14.221+0000 7f3950bf6700 1 -- 172.21.15.81:0/4220047490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f394c060200 msgr2=0x7f394c0fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:14.225 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:14.221+0000 7f3950bf6700 1 --2- 172.21.15.81:0/4220047490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f394c060200 0x7f394c0fb570 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f3940009fb0 tx=0x7f3940009b30).stop 2023-11-15T06:30:14.225 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:14.221+0000 7f3950bf6700 1 -- 172.21.15.81:0/4220047490 shutdown_connections 2023-11-15T06:30:14.225 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:14.221+0000 7f3950bf6700 1 --2- 172.21.15.81:0/4220047490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f394c060200 0x7f394c0fb570 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:14.226 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:14.221+0000 7f3950bf6700 1 -- 172.21.15.81:0/4220047490 >> 172.21.15.81:0/4220047490 conn(0x7f394c0f0b70 msgr2=0x7f394c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:30:14.226 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:14.221+0000 7f3950bf6700 1 -- 172.21.15.81:0/4220047490 shutdown_connections 2023-11-15T06:30:14.226 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:14.221+0000 7f3950bf6700 1 -- 172.21.15.81:0/4220047490 wait complete. 2023-11-15T06:30:14.226 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:14.222+0000 7f3950bf6700 1 Processor -- start 2023-11-15T06:30:14.226 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:14.222+0000 7f3950bf6700 1 -- start start 2023-11-15T06:30:14.227 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:14.222+0000 7f3950bf6700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f394c060200 0x7f394c1088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:30:14.227 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:14.222+0000 7f3950bf6700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394c108dc0 con 0x7f394c060200 2023-11-15T06:30:14.227 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:14.222+0000 7f394a59c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f394c060200 0x7f394c1088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:30:14.227 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:14.222+0000 7f394a59c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f394c060200 0x7f394c1088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:45156/0 (socket says 172.21.15.81:45156) 2023-11-15T06:30:14.227 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:14.222+0000 7f394a59c700 1 -- 172.21.15.81:0/2823830800 learned_addr learned my addr 172.21.15.81:0/2823830800 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:30:14.228 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:14.222+0000 7f394a59c700 1 -- 172.21.15.81:0/2823830800 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3940005040 con 0x7f394c060200 2023-11-15T06:30:14.228 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:14.222+0000 7f394a59c700 1 --2- 172.21.15.81:0/2823830800 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f394c060200 0x7f394c1088d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f3940009830 tx=0x7f394000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:30:14.228 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:14.223+0000 7f393affd700 1 -- 172.21.15.81:0/2823830800 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3940009320 con 0x7f394c060200 2023-11-15T06:30:14.229 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:14.223+0000 7f3950bf6700 1 -- 172.21.15.81:0/2823830800 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f394c108fc0 con 0x7f394c060200 2023-11-15T06:30:14.229 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:14.223+0000 7f393affd700 1 -- 172.21.15.81:0/2823830800 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f39400136c0 con 0x7f394c060200 2023-11-15T06:30:14.229 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:14.223+0000 7f393affd700 1 -- 172.21.15.81:0/2823830800 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f394001ee90 con 0x7f394c060200 2023-11-15T06:30:14.229 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:14.223+0000 7f3950bf6700 1 -- 172.21.15.81:0/2823830800 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f394c109450 con 0x7f394c060200 2023-11-15T06:30:14.230 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:14.223+0000 7f393affd700 1 -- 172.21.15.81:0/2823830800 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f394001e420 con 0x7f394c060200 2023-11-15T06:30:14.231 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:14.223+0000 7f3950bf6700 1 -- 172.21.15.81:0/2823830800 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f394c045ad0 con 0x7f394c060200 2023-11-15T06:30:14.231 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:14.223+0000 7f393affd700 1 --2- 172.21.15.81:0/2823830800 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f393402ff10 0x7f39340323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:30:14.231 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:14.224+0000 7f393affd700 1 -- 172.21.15.81:0/2823830800 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3940041120 con 0x7f394c060200 2023-11-15T06:30:14.231 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:14.224+0000 7f3949d9b700 1 --2- 172.21.15.81:0/2823830800 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f393402ff10 0x7f39340323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:30:14.232 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:14.224+0000 7f3949d9b700 1 --2- 172.21.15.81:0/2823830800 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f393402ff10 0x7f39340323d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f393c0096a0 tx=0x7f393c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:30:14.232 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:14.229+0000 7f393affd700 1 -- 172.21.15.81:0/2823830800 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f39400095f0 con 0x7f394c060200 2023-11-15T06:30:14.436 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:14.434+0000 7f3950bf6700 1 -- 172.21.15.81:0/2823830800 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f394c10a300 con 0x7f394c060200 2023-11-15T06:30:14.437 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:14.435+0000 7f393affd700 1 -- 172.21.15.81:0/2823830800 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f394001cb10 con 0x7f394c060200 2023-11-15T06:30:14.438 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:30:14.439 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:30:14.440 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:14.437+0000 7f3950bf6700 1 -- 172.21.15.81:0/2823830800 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f393402ff10 msgr2=0x7f39340323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:14.441 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:14.437+0000 7f3950bf6700 1 --2- 172.21.15.81:0/2823830800 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f393402ff10 0x7f39340323d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f393c0096a0 tx=0x7f393c006b40).stop 2023-11-15T06:30:14.441 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:14.437+0000 7f3950bf6700 1 -- 172.21.15.81:0/2823830800 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f394c060200 msgr2=0x7f394c1088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:14.442 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:14.437+0000 7f3950bf6700 1 --2- 172.21.15.81:0/2823830800 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f394c060200 0x7f394c1088d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f3940009830 tx=0x7f394000b040).stop 2023-11-15T06:30:14.443 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:14.437+0000 7f3950bf6700 1 -- 172.21.15.81:0/2823830800 shutdown_connections 2023-11-15T06:30:14.443 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:14.437+0000 7f3950bf6700 1 --2- 172.21.15.81:0/2823830800 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f393402ff10 0x7f39340323d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:14.443 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:14.437+0000 7f3950bf6700 1 --2- 172.21.15.81:0/2823830800 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f394c060200 0x7f394c1088d0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:14.445 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:14.437+0000 7f3950bf6700 1 -- 172.21.15.81:0/2823830800 >> 172.21.15.81:0/2823830800 conn(0x7f394c0f0b70 msgr2=0x7f394c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:30:14.445 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:14.437+0000 7f3950bf6700 1 -- 172.21.15.81:0/2823830800 shutdown_connections 2023-11-15T06:30:14.445 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:14.437+0000 7f3950bf6700 1 -- 172.21.15.81:0/2823830800 wait complete. 2023-11-15T06:30:14.446 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:30:15.169 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:14 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:30:13.819567+0000 mgr.smithi062.rwurtk (mgr.14164) 131 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:30:15.169 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:14 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:30:14.435774+0000 mon.smithi062 (mon.0) 206 : audit [DBG] from='client.? 172.21.15.81:0/2823830800' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:30:15.828 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:30:15.828 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:30:17.052 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:17.048+0000 7fbef80e8700 1 -- 172.21.15.81:0/3284619179 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbef00f6510 msgr2=0x7fbef00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:17.052 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:17.048+0000 7fbef80e8700 1 --2- 172.21.15.81:0/3284619179 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbef00f6510 0x7fbef00f6930 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7fbeec009fb0 tx=0x7fbeec009b30).stop 2023-11-15T06:30:17.052 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:17.048+0000 7fbef80e8700 1 -- 172.21.15.81:0/3284619179 shutdown_connections 2023-11-15T06:30:17.053 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:17.048+0000 7fbef80e8700 1 --2- 172.21.15.81:0/3284619179 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbef00f6510 0x7fbef00f6930 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:17.053 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:17.048+0000 7fbef80e8700 1 -- 172.21.15.81:0/3284619179 >> 172.21.15.81:0/3284619179 conn(0x7fbef00f1c70 msgr2=0x7fbef00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:30:17.053 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:17.048+0000 7fbef80e8700 1 -- 172.21.15.81:0/3284619179 shutdown_connections 2023-11-15T06:30:17.053 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:17.048+0000 7fbef80e8700 1 -- 172.21.15.81:0/3284619179 wait complete. 2023-11-15T06:30:17.053 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:17.049+0000 7fbef80e8700 1 Processor -- start 2023-11-15T06:30:17.054 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:17.049+0000 7fbef80e8700 1 -- start start 2023-11-15T06:30:17.054 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:17.049+0000 7fbef80e8700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbef00f6510 0x7fbef0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:30:17.054 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:17.049+0000 7fbef80e8700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef0106c20 con 0x7fbef00f6510 2023-11-15T06:30:17.054 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:17.049+0000 7fbef5e84700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbef00f6510 0x7fbef0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:30:17.054 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:17.049+0000 7fbef5e84700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbef00f6510 0x7fbef0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:45176/0 (socket says 172.21.15.81:45176) 2023-11-15T06:30:17.055 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:17.049+0000 7fbef5e84700 1 -- 172.21.15.81:0/2614532767 learned_addr learned my addr 172.21.15.81:0/2614532767 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:30:17.055 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:17.049+0000 7fbef5e84700 1 -- 172.21.15.81:0/2614532767 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeec005040 con 0x7fbef00f6510 2023-11-15T06:30:17.055 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:17.050+0000 7fbef5e84700 1 --2- 172.21.15.81:0/2614532767 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbef00f6510 0x7fbef0106730 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7fbeec013510 tx=0x7fbeec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:30:17.055 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:17.050+0000 7fbee67fc700 1 -- 172.21.15.81:0/2614532767 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbeec009320 con 0x7fbef00f6510 2023-11-15T06:30:17.056 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:17.050+0000 7fbef80e8700 1 -- 172.21.15.81:0/2614532767 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbef0106e20 con 0x7fbef00f6510 2023-11-15T06:30:17.056 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:17.050+0000 7fbee67fc700 1 -- 172.21.15.81:0/2614532767 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbeec013f00 con 0x7fbef00f6510 2023-11-15T06:30:17.056 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:17.050+0000 7fbee67fc700 1 -- 172.21.15.81:0/2614532767 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbeec014d90 con 0x7fbef00f6510 2023-11-15T06:30:17.056 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:17.050+0000 7fbef80e8700 1 -- 172.21.15.81:0/2614532767 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbef01072b0 con 0x7fbef00f6510 2023-11-15T06:30:17.057 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:17.051+0000 7fbef80e8700 1 -- 172.21.15.81:0/2614532767 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbef0045ad0 con 0x7fbef00f6510 2023-11-15T06:30:17.057 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:17.051+0000 7fbee67fc700 1 -- 172.21.15.81:0/2614532767 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbeec014420 con 0x7fbef00f6510 2023-11-15T06:30:17.058 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:17.051+0000 7fbee67fc700 1 --2- 172.21.15.81:0/2614532767 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fbedc02ff60 0x7fbedc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:30:17.058 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:17.051+0000 7fbee67fc700 1 -- 172.21.15.81:0/2614532767 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbeec040fb0 con 0x7fbef00f6510 2023-11-15T06:30:17.058 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:17.051+0000 7fbef5683700 1 --2- 172.21.15.81:0/2614532767 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fbedc02ff60 0x7fbedc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:30:17.058 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:17.051+0000 7fbef5683700 1 --2- 172.21.15.81:0/2614532767 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fbedc02ff60 0x7fbedc032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fbee00096a0 tx=0x7fbee0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:30:17.058 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:17.055+0000 7fbee67fc700 1 -- 172.21.15.81:0/2614532767 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbeec0095f0 con 0x7fbef00f6510 2023-11-15T06:30:17.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:16 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:30:15.819853+0000 mgr.smithi062.rwurtk (mgr.14164) 132 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:30:17.262 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:17.260+0000 7fbef80e8700 1 -- 172.21.15.81:0/2614532767 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbef00fbf60 con 0x7fbef00f6510 2023-11-15T06:30:17.263 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:17.261+0000 7fbee67fc700 1 -- 172.21.15.81:0/2614532767 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbeec01db70 con 0x7fbef00f6510 2023-11-15T06:30:17.263 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:30:17.264 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:30:17.265 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:17.263+0000 7fbef80e8700 1 -- 172.21.15.81:0/2614532767 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fbedc02ff60 msgr2=0x7fbedc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:17.265 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:17.263+0000 7fbef80e8700 1 --2- 172.21.15.81:0/2614532767 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fbedc02ff60 0x7fbedc032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fbee00096a0 tx=0x7fbee0006b40).stop 2023-11-15T06:30:17.265 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:17.263+0000 7fbef80e8700 1 -- 172.21.15.81:0/2614532767 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbef00f6510 msgr2=0x7fbef0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:17.265 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:17.263+0000 7fbef80e8700 1 --2- 172.21.15.81:0/2614532767 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbef00f6510 0x7fbef0106730 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7fbeec013510 tx=0x7fbeec00b040).stop 2023-11-15T06:30:17.266 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:17.263+0000 7fbef80e8700 1 -- 172.21.15.81:0/2614532767 shutdown_connections 2023-11-15T06:30:17.266 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:17.263+0000 7fbef80e8700 1 --2- 172.21.15.81:0/2614532767 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fbedc02ff60 0x7fbedc032420 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:17.266 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:17.263+0000 7fbef80e8700 1 --2- 172.21.15.81:0/2614532767 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbef00f6510 0x7fbef0106730 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:17.266 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:17.263+0000 7fbef80e8700 1 -- 172.21.15.81:0/2614532767 >> 172.21.15.81:0/2614532767 conn(0x7fbef00f1c70 msgr2=0x7fbef00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:30:17.266 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:17.263+0000 7fbef80e8700 1 -- 172.21.15.81:0/2614532767 shutdown_connections 2023-11-15T06:30:17.267 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:17.263+0000 7fbef80e8700 1 -- 172.21.15.81:0/2614532767 wait complete. 2023-11-15T06:30:17.267 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:30:18.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:17 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:30:17.261986+0000 mon.smithi062 (mon.0) 207 : audit [DBG] from='client.? 172.21.15.81:0/2614532767' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:30:18.683 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:30:18.683 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:30:19.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:30:17.820175+0000 mgr.smithi062.rwurtk (mgr.14164) 133 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:30:19.843 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:19.841+0000 7ffa5199f700 1 -- 172.21.15.81:0/2639156629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa4c0f51a0 msgr2=0x7ffa4c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:19.844 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:19.841+0000 7ffa5199f700 1 --2- 172.21.15.81:0/2639156629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa4c0f51a0 0x7ffa4c0f55c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7ffa40009fb0 tx=0x7ffa40009b30).stop 2023-11-15T06:30:19.844 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:19.841+0000 7ffa5199f700 1 -- 172.21.15.81:0/2639156629 shutdown_connections 2023-11-15T06:30:19.844 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:19.841+0000 7ffa5199f700 1 --2- 172.21.15.81:0/2639156629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa4c0f51a0 0x7ffa4c0f55c0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:19.844 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:19.841+0000 7ffa5199f700 1 -- 172.21.15.81:0/2639156629 >> 172.21.15.81:0/2639156629 conn(0x7ffa4c0f09a0 msgr2=0x7ffa4c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:30:19.845 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:19.841+0000 7ffa5199f700 1 -- 172.21.15.81:0/2639156629 shutdown_connections 2023-11-15T06:30:19.845 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:19.841+0000 7ffa5199f700 1 -- 172.21.15.81:0/2639156629 wait complete. 2023-11-15T06:30:19.845 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:19.841+0000 7ffa5199f700 1 Processor -- start 2023-11-15T06:30:19.845 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:19.841+0000 7ffa5199f700 1 -- start start 2023-11-15T06:30:19.845 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:19.842+0000 7ffa5199f700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa4c0f51a0 0x7ffa4c1091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:30:19.846 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:19.842+0000 7ffa5199f700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4c060220 con 0x7ffa4c0f51a0 2023-11-15T06:30:19.848 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:19.842+0000 7ffa5099d700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa4c0f51a0 0x7ffa4c1091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:30:19.849 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:19.842+0000 7ffa5099d700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa4c0f51a0 0x7ffa4c1091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:45194/0 (socket says 172.21.15.81:45194) 2023-11-15T06:30:19.849 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:19.842+0000 7ffa5099d700 1 -- 172.21.15.81:0/3006967305 learned_addr learned my addr 172.21.15.81:0/3006967305 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:30:19.849 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:19.842+0000 7ffa5099d700 1 -- 172.21.15.81:0/3006967305 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa40005040 con 0x7ffa4c0f51a0 2023-11-15T06:30:19.849 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:19.842+0000 7ffa5099d700 1 --2- 172.21.15.81:0/3006967305 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa4c0f51a0 0x7ffa4c1091d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7ffa40013510 tx=0x7ffa4000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:30:19.849 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:19.843+0000 7ffa497fa700 1 -- 172.21.15.81:0/3006967305 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa40009320 con 0x7ffa4c0f51a0 2023-11-15T06:30:19.850 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:19.843+0000 7ffa5199f700 1 -- 172.21.15.81:0/3006967305 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa4c060420 con 0x7ffa4c0f51a0 2023-11-15T06:30:19.850 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:19.843+0000 7ffa497fa700 1 -- 172.21.15.81:0/3006967305 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffa40013f00 con 0x7ffa4c0f51a0 2023-11-15T06:30:19.850 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:19.843+0000 7ffa497fa700 1 -- 172.21.15.81:0/3006967305 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa40014dc0 con 0x7ffa4c0f51a0 2023-11-15T06:30:19.850 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:19.843+0000 7ffa5199f700 1 -- 172.21.15.81:0/3006967305 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa4c109880 con 0x7ffa4c0f51a0 2023-11-15T06:30:19.850 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:19.843+0000 7ffa497fa700 1 -- 172.21.15.81:0/3006967305 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffa400095f0 con 0x7ffa4c0f51a0 2023-11-15T06:30:19.851 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:19.843+0000 7ffa5199f700 1 -- 172.21.15.81:0/3006967305 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa4c059100 con 0x7ffa4c0f51a0 2023-11-15T06:30:19.851 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:19.844+0000 7ffa497fa700 1 --2- 172.21.15.81:0/3006967305 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ffa3c02ff60 0x7ffa3c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:30:19.851 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:19.844+0000 7ffa497fa700 1 -- 172.21.15.81:0/3006967305 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffa400410c0 con 0x7ffa4c0f51a0 2023-11-15T06:30:19.851 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:19.844+0000 7ffa4bfff700 1 --2- 172.21.15.81:0/3006967305 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ffa3c02ff60 0x7ffa3c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:30:19.851 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:19.844+0000 7ffa4bfff700 1 --2- 172.21.15.81:0/3006967305 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ffa3c02ff60 0x7ffa3c032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7ffa380096a0 tx=0x7ffa38006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:30:19.852 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:19.848+0000 7ffa497fa700 1 -- 172.21.15.81:0/3006967305 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffa40014420 con 0x7ffa4c0f51a0 2023-11-15T06:30:20.055 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:20.053+0000 7ffa5199f700 1 -- 172.21.15.81:0/3006967305 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffa4c045ad0 con 0x7ffa4c0f51a0 2023-11-15T06:30:20.056 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:20.054+0000 7ffa497fa700 1 -- 172.21.15.81:0/3006967305 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ffa40010230 con 0x7ffa4c0f51a0 2023-11-15T06:30:20.056 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:30:20.057 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:30:20.058 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:20.056+0000 7ffa5199f700 1 -- 172.21.15.81:0/3006967305 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ffa3c02ff60 msgr2=0x7ffa3c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:20.058 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:20.056+0000 7ffa5199f700 1 --2- 172.21.15.81:0/3006967305 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ffa3c02ff60 0x7ffa3c032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7ffa380096a0 tx=0x7ffa38006b40).stop 2023-11-15T06:30:20.058 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:20.056+0000 7ffa5199f700 1 -- 172.21.15.81:0/3006967305 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa4c0f51a0 msgr2=0x7ffa4c1091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:20.059 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:20.056+0000 7ffa5199f700 1 --2- 172.21.15.81:0/3006967305 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa4c0f51a0 0x7ffa4c1091d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7ffa40013510 tx=0x7ffa4000b040).stop 2023-11-15T06:30:20.059 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:20.056+0000 7ffa5199f700 1 -- 172.21.15.81:0/3006967305 shutdown_connections 2023-11-15T06:30:20.059 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:20.056+0000 7ffa5199f700 1 --2- 172.21.15.81:0/3006967305 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ffa3c02ff60 0x7ffa3c032420 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:20.059 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:20.056+0000 7ffa5199f700 1 --2- 172.21.15.81:0/3006967305 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa4c0f51a0 0x7ffa4c1091d0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:20.059 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:20.056+0000 7ffa5199f700 1 -- 172.21.15.81:0/3006967305 >> 172.21.15.81:0/3006967305 conn(0x7ffa4c0f09a0 msgr2=0x7ffa4c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:30:20.060 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:20.056+0000 7ffa5199f700 1 -- 172.21.15.81:0/3006967305 shutdown_connections 2023-11-15T06:30:20.060 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:20.056+0000 7ffa5199f700 1 -- 172.21.15.81:0/3006967305 wait complete. 2023-11-15T06:30:20.061 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:30:21.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:20 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:30:19.820608+0000 mgr.smithi062.rwurtk (mgr.14164) 134 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:30:21.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:20 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:30:20.055028+0000 mon.smithi062 (mon.0) 208 : audit [DBG] from='client.? 172.21.15.81:0/3006967305' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:30:21.476 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:30:21.476 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:30:22.746 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:22.744+0000 7f679a88c700 1 -- 172.21.15.81:0/3107961811 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67940f6510 msgr2=0x7f67940f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:22.747 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:22.744+0000 7f679a88c700 1 --2- 172.21.15.81:0/3107961811 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67940f6510 0x7f67940f6930 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f6788009fb0 tx=0x7f6788009b30).stop 2023-11-15T06:30:22.747 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:22.744+0000 7f679a88c700 1 -- 172.21.15.81:0/3107961811 shutdown_connections 2023-11-15T06:30:22.747 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:22.744+0000 7f679a88c700 1 --2- 172.21.15.81:0/3107961811 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67940f6510 0x7f67940f6930 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:22.747 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:22.744+0000 7f679a88c700 1 -- 172.21.15.81:0/3107961811 >> 172.21.15.81:0/3107961811 conn(0x7f67940f1c70 msgr2=0x7f67940f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:30:22.747 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:22.744+0000 7f679a88c700 1 -- 172.21.15.81:0/3107961811 shutdown_connections 2023-11-15T06:30:22.748 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:22.744+0000 7f679a88c700 1 -- 172.21.15.81:0/3107961811 wait complete. 2023-11-15T06:30:22.748 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:22.744+0000 7f679a88c700 1 Processor -- start 2023-11-15T06:30:22.748 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:22.745+0000 7f679a88c700 1 -- start start 2023-11-15T06:30:22.750 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:22.745+0000 7f679a88c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67940f6510 0x7f679406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:30:22.750 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:22.745+0000 7f679a88c700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6794069e20 con 0x7f67940f6510 2023-11-15T06:30:22.750 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:22.745+0000 7f6793fff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67940f6510 0x7f679406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:30:22.750 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:22.745+0000 7f6793fff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67940f6510 0x7f679406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:56960/0 (socket says 172.21.15.81:56960) 2023-11-15T06:30:22.751 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:22.745+0000 7f6793fff700 1 -- 172.21.15.81:0/2509448019 learned_addr learned my addr 172.21.15.81:0/2509448019 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:30:22.751 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:22.745+0000 7f6793fff700 1 -- 172.21.15.81:0/2509448019 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6788005040 con 0x7f67940f6510 2023-11-15T06:30:22.751 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:22.745+0000 7f6793fff700 1 --2- 172.21.15.81:0/2509448019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67940f6510 0x7f679406b9a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f6788013510 tx=0x7f678800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:30:22.751 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:22.746+0000 7f6790ff9700 1 -- 172.21.15.81:0/2509448019 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6788009320 con 0x7f67940f6510 2023-11-15T06:30:22.751 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:22.746+0000 7f679a88c700 1 -- 172.21.15.81:0/2509448019 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6794069fc0 con 0x7f67940f6510 2023-11-15T06:30:22.752 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:22.746+0000 7f6790ff9700 1 -- 172.21.15.81:0/2509448019 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6788013f00 con 0x7f67940f6510 2023-11-15T06:30:22.752 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:22.746+0000 7f6790ff9700 1 -- 172.21.15.81:0/2509448019 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6788014d90 con 0x7f67940f6510 2023-11-15T06:30:22.752 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:22.746+0000 7f679a88c700 1 -- 172.21.15.81:0/2509448019 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f679406a450 con 0x7f67940f6510 2023-11-15T06:30:22.752 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:22.746+0000 7f6790ff9700 1 -- 172.21.15.81:0/2509448019 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6788014420 con 0x7f67940f6510 2023-11-15T06:30:22.752 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:22.746+0000 7f679a88c700 1 -- 172.21.15.81:0/2509448019 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6794059100 con 0x7f67940f6510 2023-11-15T06:30:22.753 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:22.747+0000 7f6790ff9700 1 --2- 172.21.15.81:0/2509448019 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f677c02ff60 0x7f677c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:30:22.753 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:22.747+0000 7f6790ff9700 1 -- 172.21.15.81:0/2509448019 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f67880410c0 con 0x7f67940f6510 2023-11-15T06:30:22.753 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:22.747+0000 7f67937fe700 1 --2- 172.21.15.81:0/2509448019 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f677c02ff60 0x7f677c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:30:22.753 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:22.747+0000 7f67937fe700 1 --2- 172.21.15.81:0/2509448019 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f677c02ff60 0x7f677c032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f67840096a0 tx=0x7f6784006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:30:22.754 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:22.752+0000 7f6790ff9700 1 -- 172.21.15.81:0/2509448019 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f67880095f0 con 0x7f67940f6510 2023-11-15T06:30:22.959 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:22.957+0000 7f679a88c700 1 -- 172.21.15.81:0/2509448019 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f67940fbf60 con 0x7f67940f6510 2023-11-15T06:30:22.960 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:22.958+0000 7f6790ff9700 1 -- 172.21.15.81:0/2509448019 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f678801db70 con 0x7f67940f6510 2023-11-15T06:30:22.960 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:30:22.960 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:30:22.961 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:22.959+0000 7f679a88c700 1 -- 172.21.15.81:0/2509448019 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f677c02ff60 msgr2=0x7f677c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:22.962 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:22.959+0000 7f679a88c700 1 --2- 172.21.15.81:0/2509448019 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f677c02ff60 0x7f677c032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f67840096a0 tx=0x7f6784006b40).stop 2023-11-15T06:30:22.962 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:22.959+0000 7f679a88c700 1 -- 172.21.15.81:0/2509448019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67940f6510 msgr2=0x7f679406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:22.963 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:22.959+0000 7f679a88c700 1 --2- 172.21.15.81:0/2509448019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67940f6510 0x7f679406b9a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f6788013510 tx=0x7f678800b040).stop 2023-11-15T06:30:22.963 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:22.959+0000 7f679a88c700 1 -- 172.21.15.81:0/2509448019 shutdown_connections 2023-11-15T06:30:22.964 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:22.959+0000 7f679a88c700 1 --2- 172.21.15.81:0/2509448019 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f677c02ff60 0x7f677c032420 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:22.964 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:22.959+0000 7f679a88c700 1 --2- 172.21.15.81:0/2509448019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67940f6510 0x7f679406b9a0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:22.964 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:22.959+0000 7f679a88c700 1 -- 172.21.15.81:0/2509448019 >> 172.21.15.81:0/2509448019 conn(0x7f67940f1c70 msgr2=0x7f67940f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:30:22.964 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:22.959+0000 7f679a88c700 1 -- 172.21.15.81:0/2509448019 shutdown_connections 2023-11-15T06:30:22.965 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:22.959+0000 7f679a88c700 1 -- 172.21.15.81:0/2509448019 wait complete. 2023-11-15T06:30:22.966 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:30:23.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:22 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:30:21.820907+0000 mgr.smithi062.rwurtk (mgr.14164) 135 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:30:24.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:23 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:30:22.958517+0000 mon.smithi062 (mon.0) 209 : audit [DBG] from='client.? 172.21.15.81:0/2509448019' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:30:24.342 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:30:24.343 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:30:25.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:24 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:30:23.821208+0000 mgr.smithi062.rwurtk (mgr.14164) 136 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:30:25.545 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:25.541+0000 7f4ac14e9700 1 -- 172.21.15.81:0/3546511587 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4abc0f5410 msgr2=0x7f4abc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:25.546 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:25.541+0000 7f4ac14e9700 1 --2- 172.21.15.81:0/3546511587 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4abc0f5410 0x7f4abc0f5830 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f4aac009fb0 tx=0x7f4aac009b30).stop 2023-11-15T06:30:25.546 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:25.541+0000 7f4ac14e9700 1 -- 172.21.15.81:0/3546511587 shutdown_connections 2023-11-15T06:30:25.546 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:25.541+0000 7f4ac14e9700 1 --2- 172.21.15.81:0/3546511587 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4abc0f5410 0x7f4abc0f5830 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:25.546 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:25.541+0000 7f4ac14e9700 1 -- 172.21.15.81:0/3546511587 >> 172.21.15.81:0/3546511587 conn(0x7f4abc0f0b70 msgr2=0x7f4abc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:30:25.547 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:25.541+0000 7f4ac14e9700 1 -- 172.21.15.81:0/3546511587 shutdown_connections 2023-11-15T06:30:25.547 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:25.541+0000 7f4ac14e9700 1 -- 172.21.15.81:0/3546511587 wait complete. 2023-11-15T06:30:25.547 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:25.542+0000 7f4ac14e9700 1 Processor -- start 2023-11-15T06:30:25.547 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:25.542+0000 7f4ac14e9700 1 -- start start 2023-11-15T06:30:25.547 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:25.542+0000 7f4ac14e9700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4abc0f5410 0x7f4abc106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:30:25.548 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:25.542+0000 7f4ac14e9700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4abc106c10 con 0x7f4abc0f5410 2023-11-15T06:30:25.548 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:25.542+0000 7f4abaffd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4abc0f5410 0x7f4abc106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:30:25.548 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:25.542+0000 7f4abaffd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4abc0f5410 0x7f4abc106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:56984/0 (socket says 172.21.15.81:56984) 2023-11-15T06:30:25.548 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:25.542+0000 7f4abaffd700 1 -- 172.21.15.81:0/2735611691 learned_addr learned my addr 172.21.15.81:0/2735611691 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:30:25.549 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:25.542+0000 7f4abaffd700 1 -- 172.21.15.81:0/2735611691 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aac005040 con 0x7f4abc0f5410 2023-11-15T06:30:25.549 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:25.542+0000 7f4abaffd700 1 --2- 172.21.15.81:0/2735611691 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4abc0f5410 0x7f4abc106720 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f4aac013510 tx=0x7f4aac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:30:25.549 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:25.543+0000 7f4aa37fe700 1 -- 172.21.15.81:0/2735611691 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4aac009320 con 0x7f4abc0f5410 2023-11-15T06:30:25.549 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:25.543+0000 7f4aa37fe700 1 -- 172.21.15.81:0/2735611691 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4aac013f00 con 0x7f4abc0f5410 2023-11-15T06:30:25.549 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:25.543+0000 7f4aa37fe700 1 -- 172.21.15.81:0/2735611691 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4aac014d90 con 0x7f4abc0f5410 2023-11-15T06:30:25.550 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:25.543+0000 7f4ac14e9700 1 -- 172.21.15.81:0/2735611691 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4abc106e10 con 0x7f4abc0f5410 2023-11-15T06:30:25.550 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:25.543+0000 7f4ac14e9700 1 -- 172.21.15.81:0/2735611691 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4abc1072a0 con 0x7f4abc0f5410 2023-11-15T06:30:25.550 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:25.544+0000 7f4aa37fe700 1 -- 172.21.15.81:0/2735611691 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4aac014420 con 0x7f4abc0f5410 2023-11-15T06:30:25.551 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:25.544+0000 7f4aa37fe700 1 --2- 172.21.15.81:0/2735611691 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f4aa402ffb0 0x7f4aa4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:30:25.551 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:25.544+0000 7f4aa37fe700 1 -- 172.21.15.81:0/2735611691 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4aac040fe0 con 0x7f4abc0f5410 2023-11-15T06:30:25.551 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:25.544+0000 7f4ac14e9700 1 -- 172.21.15.81:0/2735611691 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4abc045ad0 con 0x7f4abc0f5410 2023-11-15T06:30:25.551 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:25.544+0000 7f4aba7fc700 1 --2- 172.21.15.81:0/2735611691 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f4aa402ffb0 0x7f4aa4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:30:25.552 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:25.545+0000 7f4aba7fc700 1 --2- 172.21.15.81:0/2735611691 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f4aa402ffb0 0x7f4aa4032470 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f4ab00096a0 tx=0x7f4ab0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:30:25.552 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:25.549+0000 7f4aa37fe700 1 -- 172.21.15.81:0/2735611691 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4aac0095f0 con 0x7f4abc0f5410 2023-11-15T06:30:25.755 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:25.753+0000 7f4ac14e9700 1 -- 172.21.15.81:0/2735611691 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4abc107fc0 con 0x7f4abc0f5410 2023-11-15T06:30:25.755 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:25.754+0000 7f4aa37fe700 1 -- 172.21.15.81:0/2735611691 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4aac01db70 con 0x7f4abc0f5410 2023-11-15T06:30:25.756 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:30:25.756 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:30:25.757 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:25.755+0000 7f4ac14e9700 1 -- 172.21.15.81:0/2735611691 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f4aa402ffb0 msgr2=0x7f4aa4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:25.757 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:25.755+0000 7f4ac14e9700 1 --2- 172.21.15.81:0/2735611691 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f4aa402ffb0 0x7f4aa4032470 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f4ab00096a0 tx=0x7f4ab0006b40).stop 2023-11-15T06:30:25.757 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:25.755+0000 7f4ac14e9700 1 -- 172.21.15.81:0/2735611691 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4abc0f5410 msgr2=0x7f4abc106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:25.757 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:25.755+0000 7f4ac14e9700 1 --2- 172.21.15.81:0/2735611691 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4abc0f5410 0x7f4abc106720 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f4aac013510 tx=0x7f4aac00b040).stop 2023-11-15T06:30:25.758 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:25.755+0000 7f4ac14e9700 1 -- 172.21.15.81:0/2735611691 shutdown_connections 2023-11-15T06:30:25.758 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:25.755+0000 7f4ac14e9700 1 --2- 172.21.15.81:0/2735611691 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f4aa402ffb0 0x7f4aa4032470 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:25.758 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:25.755+0000 7f4ac14e9700 1 --2- 172.21.15.81:0/2735611691 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4abc0f5410 0x7f4abc106720 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:25.758 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:25.755+0000 7f4ac14e9700 1 -- 172.21.15.81:0/2735611691 >> 172.21.15.81:0/2735611691 conn(0x7f4abc0f0b70 msgr2=0x7f4abc05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:30:25.759 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:25.755+0000 7f4ac14e9700 1 -- 172.21.15.81:0/2735611691 shutdown_connections 2023-11-15T06:30:25.759 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:25.755+0000 7f4ac14e9700 1 -- 172.21.15.81:0/2735611691 wait complete. 2023-11-15T06:30:25.759 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:30:26.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:25 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:30:25.754465+0000 mon.smithi062 (mon.0) 210 : audit [DBG] from='client.? 172.21.15.81:0/2735611691' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:30:27.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:26 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:30:25.821575+0000 mgr.smithi062.rwurtk (mgr.14164) 137 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:30:27.180 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:30:27.180 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:30:28.389 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:28.387+0000 7f72b4971700 1 -- 172.21.15.81:0/1414984091 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72b00f6510 msgr2=0x7f72b00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:28.390 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:28.387+0000 7f72b4971700 1 --2- 172.21.15.81:0/1414984091 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72b00f6510 0x7f72b00f6930 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f72a4009fb0 tx=0x7f72a4009b30).stop 2023-11-15T06:30:28.390 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:28.387+0000 7f72b4971700 1 -- 172.21.15.81:0/1414984091 shutdown_connections 2023-11-15T06:30:28.390 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:28.387+0000 7f72b4971700 1 --2- 172.21.15.81:0/1414984091 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72b00f6510 0x7f72b00f6930 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:28.390 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:28.387+0000 7f72b4971700 1 -- 172.21.15.81:0/1414984091 >> 172.21.15.81:0/1414984091 conn(0x7f72b00f1c70 msgr2=0x7f72b00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:30:28.390 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:28.387+0000 7f72b4971700 1 -- 172.21.15.81:0/1414984091 shutdown_connections 2023-11-15T06:30:28.391 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:28.387+0000 7f72b4971700 1 -- 172.21.15.81:0/1414984091 wait complete. 2023-11-15T06:30:28.393 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:28.388+0000 7f72b4971700 1 Processor -- start 2023-11-15T06:30:28.393 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:28.388+0000 7f72b4971700 1 -- start start 2023-11-15T06:30:28.393 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:28.388+0000 7f72b4971700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72b00f6510 0x7f72b0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:30:28.393 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:28.388+0000 7f72b4971700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b0106c20 con 0x7f72b00f6510 2023-11-15T06:30:28.393 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:28.388+0000 7f72ae59c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72b00f6510 0x7f72b0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:30:28.394 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:28.388+0000 7f72ae59c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72b00f6510 0x7f72b0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:56996/0 (socket says 172.21.15.81:56996) 2023-11-15T06:30:28.395 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:28.388+0000 7f72ae59c700 1 -- 172.21.15.81:0/3584671351 learned_addr learned my addr 172.21.15.81:0/3584671351 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:30:28.395 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:28.388+0000 7f72ae59c700 1 -- 172.21.15.81:0/3584671351 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a4005040 con 0x7f72b00f6510 2023-11-15T06:30:28.395 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:28.389+0000 7f72ae59c700 1 --2- 172.21.15.81:0/3584671351 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72b00f6510 0x7f72b0106730 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f72a4013510 tx=0x7f72a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:30:28.395 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:28.389+0000 7f729effd700 1 -- 172.21.15.81:0/3584671351 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72a4009320 con 0x7f72b00f6510 2023-11-15T06:30:28.396 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:28.389+0000 7f72b4971700 1 -- 172.21.15.81:0/3584671351 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72b0106e20 con 0x7f72b00f6510 2023-11-15T06:30:28.396 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:28.389+0000 7f729effd700 1 -- 172.21.15.81:0/3584671351 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f72a4013f00 con 0x7f72b00f6510 2023-11-15T06:30:28.396 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:28.389+0000 7f729effd700 1 -- 172.21.15.81:0/3584671351 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72a4014d90 con 0x7f72b00f6510 2023-11-15T06:30:28.396 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:28.389+0000 7f72b4971700 1 -- 172.21.15.81:0/3584671351 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72b01072b0 con 0x7f72b00f6510 2023-11-15T06:30:28.396 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:28.389+0000 7f729effd700 1 -- 172.21.15.81:0/3584671351 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f72a40095f0 con 0x7f72b00f6510 2023-11-15T06:30:28.397 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:28.390+0000 7f72b4971700 1 -- 172.21.15.81:0/3584671351 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72b0045ad0 con 0x7f72b00f6510 2023-11-15T06:30:28.397 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:28.390+0000 7f729effd700 1 --2- 172.21.15.81:0/3584671351 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f729802ff60 0x7f7298032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:30:28.397 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:28.390+0000 7f729effd700 1 -- 172.21.15.81:0/3584671351 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f72a4041000 con 0x7f72b00f6510 2023-11-15T06:30:28.397 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:28.390+0000 7f72add9b700 1 --2- 172.21.15.81:0/3584671351 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f729802ff60 0x7f7298032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:30:28.397 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:28.390+0000 7f72add9b700 1 --2- 172.21.15.81:0/3584671351 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f729802ff60 0x7f7298032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f72a00096a0 tx=0x7f72a0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:30:28.398 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:28.395+0000 7f729effd700 1 -- 172.21.15.81:0/3584671351 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f72a4014420 con 0x7f72b00f6510 2023-11-15T06:30:28.602 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:28.600+0000 7f72b4971700 1 -- 172.21.15.81:0/3584671351 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f72b00fbf60 con 0x7f72b00f6510 2023-11-15T06:30:28.602 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:28.601+0000 7f729effd700 1 -- 172.21.15.81:0/3584671351 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f72a4010230 con 0x7f72b00f6510 2023-11-15T06:30:28.603 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:30:28.603 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:30:28.604 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:28.602+0000 7f72b4971700 1 -- 172.21.15.81:0/3584671351 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f729802ff60 msgr2=0x7f7298032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:28.605 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:28.602+0000 7f72b4971700 1 --2- 172.21.15.81:0/3584671351 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f729802ff60 0x7f7298032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f72a00096a0 tx=0x7f72a0006b40).stop 2023-11-15T06:30:28.605 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:28.602+0000 7f72b4971700 1 -- 172.21.15.81:0/3584671351 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72b00f6510 msgr2=0x7f72b0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:28.605 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:28.602+0000 7f72b4971700 1 --2- 172.21.15.81:0/3584671351 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72b00f6510 0x7f72b0106730 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f72a4013510 tx=0x7f72a400b040).stop 2023-11-15T06:30:28.605 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:28.602+0000 7f72b4971700 1 -- 172.21.15.81:0/3584671351 shutdown_connections 2023-11-15T06:30:28.605 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:28.602+0000 7f72b4971700 1 --2- 172.21.15.81:0/3584671351 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f729802ff60 0x7f7298032420 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:28.606 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:28.602+0000 7f72b4971700 1 --2- 172.21.15.81:0/3584671351 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72b00f6510 0x7f72b0106730 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:28.606 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:28.602+0000 7f72b4971700 1 -- 172.21.15.81:0/3584671351 >> 172.21.15.81:0/3584671351 conn(0x7f72b00f1c70 msgr2=0x7f72b00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:30:28.606 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:28.602+0000 7f72b4971700 1 -- 172.21.15.81:0/3584671351 shutdown_connections 2023-11-15T06:30:28.606 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:28.602+0000 7f72b4971700 1 -- 172.21.15.81:0/3584671351 wait complete. 2023-11-15T06:30:28.607 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:30:28.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:28 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:30:27.821870+0000 mgr.smithi062.rwurtk (mgr.14164) 138 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:30:28.919 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:28 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:30:28.601369+0000 mon.smithi062 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.81:0/3584671351' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:30:29.993 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:30:29.993 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:30:31.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:30 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:30:29.822200+0000 mgr.smithi062.rwurtk (mgr.14164) 139 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:30:31.193 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:31.191+0000 7fc401926700 1 -- 172.21.15.81:0/333590053 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3fc060280 msgr2=0x7fc3fc0fb4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:31.193 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:31.191+0000 7fc401926700 1 --2- 172.21.15.81:0/333590053 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3fc060280 0x7fc3fc0fb4f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7fc3f0009fb0 tx=0x7fc3f0009b30).stop 2023-11-15T06:30:31.194 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:31.191+0000 7fc401926700 1 -- 172.21.15.81:0/333590053 shutdown_connections 2023-11-15T06:30:31.194 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:31.191+0000 7fc401926700 1 --2- 172.21.15.81:0/333590053 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3fc060280 0x7fc3fc0fb4f0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:31.194 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:31.191+0000 7fc401926700 1 -- 172.21.15.81:0/333590053 >> 172.21.15.81:0/333590053 conn(0x7fc3fc0f0b70 msgr2=0x7fc3fc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:30:31.194 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:31.191+0000 7fc401926700 1 -- 172.21.15.81:0/333590053 shutdown_connections 2023-11-15T06:30:31.195 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:31.191+0000 7fc401926700 1 -- 172.21.15.81:0/333590053 wait complete. 2023-11-15T06:30:31.196 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:31.192+0000 7fc401926700 1 Processor -- start 2023-11-15T06:30:31.196 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:31.192+0000 7fc401926700 1 -- start start 2023-11-15T06:30:31.196 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:31.192+0000 7fc401926700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3fc060280 0x7fc3fc1088e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:30:31.197 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:31.192+0000 7fc401926700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3fc108dd0 con 0x7fc3fc060280 2023-11-15T06:30:31.197 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:31.192+0000 7fc3faffd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3fc060280 0x7fc3fc1088e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:30:31.197 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:31.192+0000 7fc3faffd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3fc060280 0x7fc3fc1088e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:57014/0 (socket says 172.21.15.81:57014) 2023-11-15T06:30:31.197 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:31.192+0000 7fc3faffd700 1 -- 172.21.15.81:0/4127613872 learned_addr learned my addr 172.21.15.81:0/4127613872 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:30:31.197 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:31.193+0000 7fc3faffd700 1 -- 172.21.15.81:0/4127613872 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f0005040 con 0x7fc3fc060280 2023-11-15T06:30:31.198 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:31.193+0000 7fc3faffd700 1 --2- 172.21.15.81:0/4127613872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3fc060280 0x7fc3fc1088e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fc3f0013510 tx=0x7fc3f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:30:31.198 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:31.193+0000 7fc3e37fe700 1 -- 172.21.15.81:0/4127613872 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc3f0009320 con 0x7fc3fc060280 2023-11-15T06:30:31.198 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:31.193+0000 7fc401926700 1 -- 172.21.15.81:0/4127613872 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3fc108fd0 con 0x7fc3fc060280 2023-11-15T06:30:31.198 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:31.193+0000 7fc3e37fe700 1 -- 172.21.15.81:0/4127613872 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc3f0013f00 con 0x7fc3fc060280 2023-11-15T06:30:31.198 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:31.193+0000 7fc3e37fe700 1 -- 172.21.15.81:0/4127613872 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc3f0014dc0 con 0x7fc3fc060280 2023-11-15T06:30:31.199 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:31.193+0000 7fc401926700 1 -- 172.21.15.81:0/4127613872 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3fc109460 con 0x7fc3fc060280 2023-11-15T06:30:31.199 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:31.194+0000 7fc401926700 1 -- 172.21.15.81:0/4127613872 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3fc045ad0 con 0x7fc3fc060280 2023-11-15T06:30:31.200 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:31.195+0000 7fc3e37fe700 1 -- 172.21.15.81:0/4127613872 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc3f00095f0 con 0x7fc3fc060280 2023-11-15T06:30:31.200 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:31.195+0000 7fc3e37fe700 1 --2- 172.21.15.81:0/4127613872 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fc3e402ffb0 0x7fc3e4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:30:31.200 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:31.195+0000 7fc3e37fe700 1 -- 172.21.15.81:0/4127613872 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc3f0041110 con 0x7fc3fc060280 2023-11-15T06:30:31.200 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:31.195+0000 7fc3fa7fc700 1 --2- 172.21.15.81:0/4127613872 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fc3e402ffb0 0x7fc3e4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:30:31.200 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:31.196+0000 7fc3fa7fc700 1 --2- 172.21.15.81:0/4127613872 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fc3e402ffb0 0x7fc3e4032470 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fc3ec0096a0 tx=0x7fc3ec006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:30:31.201 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:31.199+0000 7fc3e37fe700 1 -- 172.21.15.81:0/4127613872 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc3f0014420 con 0x7fc3fc060280 2023-11-15T06:30:31.405 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:31.403+0000 7fc401926700 1 -- 172.21.15.81:0/4127613872 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc3fc0fe150 con 0x7fc3fc060280 2023-11-15T06:30:31.405 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:31.403+0000 7fc3e37fe700 1 -- 172.21.15.81:0/4127613872 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc3f000f760 con 0x7fc3fc060280 2023-11-15T06:30:31.406 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:30:31.406 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:30:31.407 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:31.405+0000 7fc401926700 1 -- 172.21.15.81:0/4127613872 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fc3e402ffb0 msgr2=0x7fc3e4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:31.407 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:31.405+0000 7fc401926700 1 --2- 172.21.15.81:0/4127613872 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fc3e402ffb0 0x7fc3e4032470 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fc3ec0096a0 tx=0x7fc3ec006b40).stop 2023-11-15T06:30:31.407 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:31.405+0000 7fc401926700 1 -- 172.21.15.81:0/4127613872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3fc060280 msgr2=0x7fc3fc1088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:31.408 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:31.405+0000 7fc401926700 1 --2- 172.21.15.81:0/4127613872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3fc060280 0x7fc3fc1088e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fc3f0013510 tx=0x7fc3f000b040).stop 2023-11-15T06:30:31.408 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:31.405+0000 7fc401926700 1 -- 172.21.15.81:0/4127613872 shutdown_connections 2023-11-15T06:30:31.408 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:31.405+0000 7fc401926700 1 --2- 172.21.15.81:0/4127613872 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fc3e402ffb0 0x7fc3e4032470 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:31.408 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:31.405+0000 7fc401926700 1 --2- 172.21.15.81:0/4127613872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3fc060280 0x7fc3fc1088e0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:31.409 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:31.405+0000 7fc401926700 1 -- 172.21.15.81:0/4127613872 >> 172.21.15.81:0/4127613872 conn(0x7fc3fc0f0b70 msgr2=0x7fc3fc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:30:31.409 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:31.405+0000 7fc401926700 1 -- 172.21.15.81:0/4127613872 shutdown_connections 2023-11-15T06:30:31.409 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:31.405+0000 7fc401926700 1 -- 172.21.15.81:0/4127613872 wait complete. 2023-11-15T06:30:31.414 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:30:32.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:31 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:30:31.404346+0000 mon.smithi062 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.81:0/4127613872' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:30:32.797 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:30:32.797 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:30:33.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:32 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:30:31.822548+0000 mgr.smithi062.rwurtk (mgr.14164) 140 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:30:34.010 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:34.005+0000 7f7f2fc93700 1 -- 172.21.15.81:0/2513933476 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f28060220 msgr2=0x7f7f280f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:34.011 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:34.005+0000 7f7f2fc93700 1 --2- 172.21.15.81:0/2513933476 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f28060220 0x7f7f280f9150 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f7f18009fb0 tx=0x7f7f18009b30).stop 2023-11-15T06:30:34.011 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:34.005+0000 7f7f2fc93700 1 -- 172.21.15.81:0/2513933476 shutdown_connections 2023-11-15T06:30:34.011 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:34.005+0000 7f7f2fc93700 1 --2- 172.21.15.81:0/2513933476 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f28060220 0x7f7f280f9150 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:34.011 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:34.005+0000 7f7f2fc93700 1 -- 172.21.15.81:0/2513933476 >> 172.21.15.81:0/2513933476 conn(0x7f7f280f0b70 msgr2=0x7f7f280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:30:34.011 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:34.006+0000 7f7f2fc93700 1 -- 172.21.15.81:0/2513933476 shutdown_connections 2023-11-15T06:30:34.012 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:34.006+0000 7f7f2fc93700 1 -- 172.21.15.81:0/2513933476 wait complete. 2023-11-15T06:30:34.012 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:34.006+0000 7f7f2fc93700 1 Processor -- start 2023-11-15T06:30:34.012 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:34.006+0000 7f7f2fc93700 1 -- start start 2023-11-15T06:30:34.012 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:34.006+0000 7f7f2fc93700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f28060220 0x7f7f280f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:30:34.012 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:34.006+0000 7f7f2fc93700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f280f5050 con 0x7f7f28060220 2023-11-15T06:30:34.013 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:34.006+0000 7f7f2da2f700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f28060220 0x7f7f280f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:30:34.013 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:34.006+0000 7f7f2da2f700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f28060220 0x7f7f280f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:56344/0 (socket says 172.21.15.81:56344) 2023-11-15T06:30:34.013 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:34.006+0000 7f7f2da2f700 1 -- 172.21.15.81:0/1810422918 learned_addr learned my addr 172.21.15.81:0/1810422918 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:30:34.013 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:34.007+0000 7f7f2da2f700 1 -- 172.21.15.81:0/1810422918 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f18005040 con 0x7f7f28060220 2023-11-15T06:30:34.013 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:34.007+0000 7f7f2da2f700 1 --2- 172.21.15.81:0/1810422918 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f28060220 0x7f7f280f6bd0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f7f18013510 tx=0x7f7f1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:30:34.014 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:34.007+0000 7f7f1e7fc700 1 -- 172.21.15.81:0/1810422918 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f18009320 con 0x7f7f28060220 2023-11-15T06:30:34.014 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:34.007+0000 7f7f1e7fc700 1 -- 172.21.15.81:0/1810422918 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7f18013f00 con 0x7f7f28060220 2023-11-15T06:30:34.014 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:34.007+0000 7f7f2fc93700 1 -- 172.21.15.81:0/1810422918 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f280f51f0 con 0x7f7f28060220 2023-11-15T06:30:34.014 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:34.007+0000 7f7f1e7fc700 1 -- 172.21.15.81:0/1810422918 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f18014dc0 con 0x7f7f28060220 2023-11-15T06:30:34.014 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:34.007+0000 7f7f2fc93700 1 -- 172.21.15.81:0/1810422918 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f280f5680 con 0x7f7f28060220 2023-11-15T06:30:34.015 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:34.008+0000 7f7f2fc93700 1 -- 172.21.15.81:0/1810422918 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f28059100 con 0x7f7f28060220 2023-11-15T06:30:34.016 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:34.008+0000 7f7f1e7fc700 1 -- 172.21.15.81:0/1810422918 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7f1801d3d0 con 0x7f7f28060220 2023-11-15T06:30:34.016 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:34.008+0000 7f7f1e7fc700 1 --2- 172.21.15.81:0/1810422918 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f7f1402ff60 0x7f7f14032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:30:34.016 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:34.008+0000 7f7f1e7fc700 1 -- 172.21.15.81:0/1810422918 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7f180413a0 con 0x7f7f28060220 2023-11-15T06:30:34.016 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:34.008+0000 7f7f2d22e700 1 --2- 172.21.15.81:0/1810422918 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f7f1402ff60 0x7f7f14032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:30:34.016 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:34.009+0000 7f7f2d22e700 1 --2- 172.21.15.81:0/1810422918 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f7f1402ff60 0x7f7f14032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f7f240096a0 tx=0x7f7f24006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:30:34.017 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:34.013+0000 7f7f1e7fc700 1 -- 172.21.15.81:0/1810422918 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7f1801cb10 con 0x7f7f28060220 2023-11-15T06:30:34.220 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:34.217+0000 7f7f2fc93700 1 -- 172.21.15.81:0/1810422918 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7f28045ad0 con 0x7f7f28060220 2023-11-15T06:30:34.220 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:34.217+0000 7f7f1e7fc700 1 -- 172.21.15.81:0/1810422918 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7f1801cb10 con 0x7f7f28060220 2023-11-15T06:30:34.221 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:30:34.221 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:30:34.222 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:34.219+0000 7f7f2fc93700 1 -- 172.21.15.81:0/1810422918 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f7f1402ff60 msgr2=0x7f7f14032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:34.222 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:34.219+0000 7f7f2fc93700 1 --2- 172.21.15.81:0/1810422918 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f7f1402ff60 0x7f7f14032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f7f240096a0 tx=0x7f7f24006b40).stop 2023-11-15T06:30:34.222 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:34.219+0000 7f7f2fc93700 1 -- 172.21.15.81:0/1810422918 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f28060220 msgr2=0x7f7f280f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:34.222 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:34.219+0000 7f7f2fc93700 1 --2- 172.21.15.81:0/1810422918 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f28060220 0x7f7f280f6bd0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f7f18013510 tx=0x7f7f1800b040).stop 2023-11-15T06:30:34.223 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:34.219+0000 7f7f2fc93700 1 -- 172.21.15.81:0/1810422918 shutdown_connections 2023-11-15T06:30:34.223 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:34.219+0000 7f7f2fc93700 1 --2- 172.21.15.81:0/1810422918 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f7f1402ff60 0x7f7f14032420 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:34.223 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:34.219+0000 7f7f2fc93700 1 --2- 172.21.15.81:0/1810422918 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f28060220 0x7f7f280f6bd0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:34.223 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:34.219+0000 7f7f2fc93700 1 -- 172.21.15.81:0/1810422918 >> 172.21.15.81:0/1810422918 conn(0x7f7f280f0b70 msgr2=0x7f7f280f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:30:34.223 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:34.219+0000 7f7f2fc93700 1 -- 172.21.15.81:0/1810422918 shutdown_connections 2023-11-15T06:30:34.224 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:34.219+0000 7f7f2fc93700 1 -- 172.21.15.81:0/1810422918 wait complete. 2023-11-15T06:30:34.224 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:30:35.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:34 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:30:33.822800+0000 mgr.smithi062.rwurtk (mgr.14164) 141 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:30:35.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:34 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:30:34.219313+0000 mon.smithi062 (mon.0) 213 : audit [DBG] from='client.? 172.21.15.81:0/1810422918' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:30:35.650 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:30:35.705 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:30:36.991 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:36.988+0000 7f82d63b5700 1 -- 172.21.15.81:0/1981854337 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82d00f5410 msgr2=0x7f82d00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:36.991 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:36.988+0000 7f82d63b5700 1 --2- 172.21.15.81:0/1981854337 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82d00f5410 0x7f82d00f5830 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f82b8009fb0 tx=0x7f82b8009b30).stop 2023-11-15T06:30:36.992 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:36.988+0000 7f82d63b5700 1 -- 172.21.15.81:0/1981854337 shutdown_connections 2023-11-15T06:30:36.992 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:36.988+0000 7f82d63b5700 1 --2- 172.21.15.81:0/1981854337 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82d00f5410 0x7f82d00f5830 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:36.992 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:36.988+0000 7f82d63b5700 1 -- 172.21.15.81:0/1981854337 >> 172.21.15.81:0/1981854337 conn(0x7f82d00f0b70 msgr2=0x7f82d00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:30:36.992 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:36.988+0000 7f82d63b5700 1 -- 172.21.15.81:0/1981854337 shutdown_connections 2023-11-15T06:30:36.993 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:36.988+0000 7f82d63b5700 1 -- 172.21.15.81:0/1981854337 wait complete. 2023-11-15T06:30:36.993 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:36.988+0000 7f82d63b5700 1 Processor -- start 2023-11-15T06:30:36.993 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:36.988+0000 7f82d63b5700 1 -- start start 2023-11-15T06:30:36.993 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:36.988+0000 7f82d63b5700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82d00f5410 0x7f82d0109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:30:36.993 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:36.988+0000 7f82d63b5700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d0060220 con 0x7f82d00f5410 2023-11-15T06:30:36.995 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:36.989+0000 7f82cffff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82d00f5410 0x7f82d0109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:30:36.995 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:36.989+0000 7f82cffff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82d00f5410 0x7f82d0109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:56354/0 (socket says 172.21.15.81:56354) 2023-11-15T06:30:36.995 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:36.989+0000 7f82cffff700 1 -- 172.21.15.81:0/1161467441 learned_addr learned my addr 172.21.15.81:0/1161467441 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:30:36.996 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:36.989+0000 7f82cffff700 1 -- 172.21.15.81:0/1161467441 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b8005040 con 0x7f82d00f5410 2023-11-15T06:30:36.996 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:36.989+0000 7f82cffff700 1 --2- 172.21.15.81:0/1161467441 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82d00f5410 0x7f82d0109450 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f82b8013510 tx=0x7f82b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:30:36.996 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:36.989+0000 7f82ccff9700 1 -- 172.21.15.81:0/1161467441 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82b8009320 con 0x7f82d00f5410 2023-11-15T06:30:36.996 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:36.989+0000 7f82d63b5700 1 -- 172.21.15.81:0/1161467441 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82d0060420 con 0x7f82d00f5410 2023-11-15T06:30:36.996 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:36.989+0000 7f82ccff9700 1 -- 172.21.15.81:0/1161467441 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f82b8013f00 con 0x7f82d00f5410 2023-11-15T06:30:36.997 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:36.989+0000 7f82ccff9700 1 -- 172.21.15.81:0/1161467441 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82b8014d90 con 0x7f82d00f5410 2023-11-15T06:30:36.997 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:36.989+0000 7f82d63b5700 1 -- 172.21.15.81:0/1161467441 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82d0109b00 con 0x7f82d00f5410 2023-11-15T06:30:36.997 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:36.990+0000 7f82ccff9700 1 -- 172.21.15.81:0/1161467441 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f82b80095f0 con 0x7f82d00f5410 2023-11-15T06:30:36.997 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:36.990+0000 7f82d63b5700 1 -- 172.21.15.81:0/1161467441 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82d0059100 con 0x7f82d00f5410 2023-11-15T06:30:36.997 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:36.990+0000 7f82ccff9700 1 --2- 172.21.15.81:0/1161467441 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f82bc02ff60 0x7f82bc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:30:36.998 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:36.990+0000 7f82ccff9700 1 -- 172.21.15.81:0/1161467441 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f82b8041000 con 0x7f82d00f5410 2023-11-15T06:30:36.998 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:36.990+0000 7f82cf7fe700 1 --2- 172.21.15.81:0/1161467441 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f82bc02ff60 0x7f82bc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:30:36.998 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:36.991+0000 7f82cf7fe700 1 --2- 172.21.15.81:0/1161467441 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f82bc02ff60 0x7f82bc032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f82c00096a0 tx=0x7f82c0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:30:36.998 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:36.995+0000 7f82ccff9700 1 -- 172.21.15.81:0/1161467441 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f82b8014420 con 0x7f82d00f5410 2023-11-15T06:30:37.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:36 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:30:35.823151+0000 mgr.smithi062.rwurtk (mgr.14164) 142 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:30:37.201 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:37.198+0000 7f82d63b5700 1 -- 172.21.15.81:0/1161467441 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f82d0045ad0 con 0x7f82d00f5410 2023-11-15T06:30:37.202 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:37.199+0000 7f82ccff9700 1 -- 172.21.15.81:0/1161467441 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f82b8010230 con 0x7f82d00f5410 2023-11-15T06:30:37.202 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:30:37.202 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:30:37.204 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:37.200+0000 7f82d63b5700 1 -- 172.21.15.81:0/1161467441 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f82bc02ff60 msgr2=0x7f82bc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:37.204 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:37.200+0000 7f82d63b5700 1 --2- 172.21.15.81:0/1161467441 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f82bc02ff60 0x7f82bc032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f82c00096a0 tx=0x7f82c0006b40).stop 2023-11-15T06:30:37.204 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:37.201+0000 7f82d63b5700 1 -- 172.21.15.81:0/1161467441 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82d00f5410 msgr2=0x7f82d0109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:37.204 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:37.201+0000 7f82d63b5700 1 --2- 172.21.15.81:0/1161467441 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82d00f5410 0x7f82d0109450 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f82b8013510 tx=0x7f82b800b040).stop 2023-11-15T06:30:37.204 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:37.201+0000 7f82d63b5700 1 -- 172.21.15.81:0/1161467441 shutdown_connections 2023-11-15T06:30:37.205 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:37.201+0000 7f82d63b5700 1 --2- 172.21.15.81:0/1161467441 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f82bc02ff60 0x7f82bc032420 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:37.205 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:37.201+0000 7f82d63b5700 1 --2- 172.21.15.81:0/1161467441 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82d00f5410 0x7f82d0109450 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:37.205 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:37.201+0000 7f82d63b5700 1 -- 172.21.15.81:0/1161467441 >> 172.21.15.81:0/1161467441 conn(0x7f82d00f0b70 msgr2=0x7f82d00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:30:37.205 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:37.201+0000 7f82d63b5700 1 -- 172.21.15.81:0/1161467441 shutdown_connections 2023-11-15T06:30:37.205 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:37.201+0000 7f82d63b5700 1 -- 172.21.15.81:0/1161467441 wait complete. 2023-11-15T06:30:37.206 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:30:38.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:37 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:30:37.200670+0000 mon.smithi062 (mon.0) 214 : audit [DBG] from='client.? 172.21.15.81:0/1161467441' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:30:38.624 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:30:38.625 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:30:39.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:38 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:30:37.823483+0000 mgr.smithi062.rwurtk (mgr.14164) 143 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:30:39.788 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:39.784+0000 7fadbbb8b700 1 -- 172.21.15.81:0/1162272312 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fadb40f6510 msgr2=0x7fadb40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:39.789 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:39.784+0000 7fadbbb8b700 1 --2- 172.21.15.81:0/1162272312 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fadb40f6510 0x7fadb40f6930 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7fada8009fb0 tx=0x7fada8009b30).stop 2023-11-15T06:30:39.789 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:39.785+0000 7fadbbb8b700 1 -- 172.21.15.81:0/1162272312 shutdown_connections 2023-11-15T06:30:39.789 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:39.785+0000 7fadbbb8b700 1 --2- 172.21.15.81:0/1162272312 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fadb40f6510 0x7fadb40f6930 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:39.790 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:39.785+0000 7fadbbb8b700 1 -- 172.21.15.81:0/1162272312 >> 172.21.15.81:0/1162272312 conn(0x7fadb40f1c70 msgr2=0x7fadb40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:30:39.790 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:39.785+0000 7fadbbb8b700 1 -- 172.21.15.81:0/1162272312 shutdown_connections 2023-11-15T06:30:39.790 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:39.785+0000 7fadbbb8b700 1 -- 172.21.15.81:0/1162272312 wait complete. 2023-11-15T06:30:39.790 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:39.785+0000 7fadbbb8b700 1 Processor -- start 2023-11-15T06:30:39.791 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:39.785+0000 7fadbbb8b700 1 -- start start 2023-11-15T06:30:39.791 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:39.785+0000 7fadbbb8b700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fadb40f6510 0x7fadb4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:30:39.791 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:39.785+0000 7fadbbb8b700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb4106c20 con 0x7fadb40f6510 2023-11-15T06:30:39.793 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:39.786+0000 7fadb9927700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fadb40f6510 0x7fadb4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:30:39.794 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:39.786+0000 7fadb9927700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fadb40f6510 0x7fadb4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:56376/0 (socket says 172.21.15.81:56376) 2023-11-15T06:30:39.794 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:39.786+0000 7fadb9927700 1 -- 172.21.15.81:0/544374763 learned_addr learned my addr 172.21.15.81:0/544374763 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:30:39.794 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:39.786+0000 7fadb9927700 1 -- 172.21.15.81:0/544374763 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada8005040 con 0x7fadb40f6510 2023-11-15T06:30:39.794 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:39.786+0000 7fadb9927700 1 --2- 172.21.15.81:0/544374763 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fadb40f6510 0x7fadb4106730 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fada8014000 tx=0x7fada800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:30:39.795 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:39.786+0000 7fada67fc700 1 -- 172.21.15.81:0/544374763 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fada80095f0 con 0x7fadb40f6510 2023-11-15T06:30:39.795 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:39.786+0000 7fadbbb8b700 1 -- 172.21.15.81:0/544374763 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadb4106e20 con 0x7fadb40f6510 2023-11-15T06:30:39.795 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:39.786+0000 7fada67fc700 1 -- 172.21.15.81:0/544374763 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fada8007e70 con 0x7fadb40f6510 2023-11-15T06:30:39.795 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:39.786+0000 7fada67fc700 1 -- 172.21.15.81:0/544374763 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fada8014ea0 con 0x7fadb40f6510 2023-11-15T06:30:39.796 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:39.786+0000 7fadbbb8b700 1 -- 172.21.15.81:0/544374763 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadb41072b0 con 0x7fadb40f6510 2023-11-15T06:30:39.796 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:39.787+0000 7fada67fc700 1 -- 172.21.15.81:0/544374763 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fada8025070 con 0x7fadb40f6510 2023-11-15T06:30:39.796 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:39.787+0000 7fadbbb8b700 1 -- 172.21.15.81:0/544374763 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadb4045ad0 con 0x7fadb40f6510 2023-11-15T06:30:39.796 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:39.787+0000 7fada67fc700 1 --2- 172.21.15.81:0/544374763 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fada002ffb0 0x7fada0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:30:39.796 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:39.787+0000 7fada67fc700 1 -- 172.21.15.81:0/544374763 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fada8040920 con 0x7fadb40f6510 2023-11-15T06:30:39.797 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:39.788+0000 7fadb9126700 1 --2- 172.21.15.81:0/544374763 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fada002ffb0 0x7fada0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:30:39.797 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:39.788+0000 7fadb9126700 1 --2- 172.21.15.81:0/544374763 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fada002ffb0 0x7fada0032470 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fadb00096a0 tx=0x7fadb0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:30:39.797 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:39.792+0000 7fada67fc700 1 -- 172.21.15.81:0/544374763 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fada800f4e0 con 0x7fadb40f6510 2023-11-15T06:30:40.001 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:39.998+0000 7fadbbb8b700 1 -- 172.21.15.81:0/544374763 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fadb40fbf60 con 0x7fadb40f6510 2023-11-15T06:30:40.001 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:39.998+0000 7fada67fc700 1 -- 172.21.15.81:0/544374763 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fada800f300 con 0x7fadb40f6510 2023-11-15T06:30:40.002 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:30:40.002 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:30:40.003 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:40.000+0000 7fadbbb8b700 1 -- 172.21.15.81:0/544374763 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fada002ffb0 msgr2=0x7fada0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:40.003 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:40.000+0000 7fadbbb8b700 1 --2- 172.21.15.81:0/544374763 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fada002ffb0 0x7fada0032470 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fadb00096a0 tx=0x7fadb0006b40).stop 2023-11-15T06:30:40.003 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:40.000+0000 7fadbbb8b700 1 -- 172.21.15.81:0/544374763 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fadb40f6510 msgr2=0x7fadb4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:40.004 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:40.000+0000 7fadbbb8b700 1 --2- 172.21.15.81:0/544374763 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fadb40f6510 0x7fadb4106730 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fada8014000 tx=0x7fada800b040).stop 2023-11-15T06:30:40.004 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:40.000+0000 7fadbbb8b700 1 -- 172.21.15.81:0/544374763 shutdown_connections 2023-11-15T06:30:40.004 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:40.000+0000 7fadbbb8b700 1 --2- 172.21.15.81:0/544374763 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fada002ffb0 0x7fada0032470 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:40.004 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:40.000+0000 7fadbbb8b700 1 --2- 172.21.15.81:0/544374763 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fadb40f6510 0x7fadb4106730 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:40.004 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:40.000+0000 7fadbbb8b700 1 -- 172.21.15.81:0/544374763 >> 172.21.15.81:0/544374763 conn(0x7fadb40f1c70 msgr2=0x7fadb40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:30:40.005 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:40.000+0000 7fadbbb8b700 1 -- 172.21.15.81:0/544374763 shutdown_connections 2023-11-15T06:30:40.005 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:40.000+0000 7fadbbb8b700 1 -- 172.21.15.81:0/544374763 wait complete. 2023-11-15T06:30:40.005 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:30:41.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:40 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:30:39.823818+0000 mgr.smithi062.rwurtk (mgr.14164) 144 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:30:41.169 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:40 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:30:40.000145+0000 mon.smithi062 (mon.0) 215 : audit [DBG] from='client.? 172.21.15.81:0/544374763' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:30:41.416 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:30:41.417 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:30:42.693 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:42.688+0000 7f57ba852700 1 -- 172.21.15.81:0/770832544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57b40f6510 msgr2=0x7f57b40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:42.693 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:42.688+0000 7f57ba852700 1 --2- 172.21.15.81:0/770832544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57b40f6510 0x7f57b40f6930 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f57a4009fb0 tx=0x7f57a4009b30).stop 2023-11-15T06:30:42.693 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:42.688+0000 7f57ba852700 1 -- 172.21.15.81:0/770832544 shutdown_connections 2023-11-15T06:30:42.694 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:42.688+0000 7f57ba852700 1 --2- 172.21.15.81:0/770832544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57b40f6510 0x7f57b40f6930 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:42.694 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:42.688+0000 7f57ba852700 1 -- 172.21.15.81:0/770832544 >> 172.21.15.81:0/770832544 conn(0x7f57b40f1c70 msgr2=0x7f57b40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:30:42.694 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:42.688+0000 7f57ba852700 1 -- 172.21.15.81:0/770832544 shutdown_connections 2023-11-15T06:30:42.694 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:42.688+0000 7f57ba852700 1 -- 172.21.15.81:0/770832544 wait complete. 2023-11-15T06:30:42.694 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:42.689+0000 7f57ba852700 1 Processor -- start 2023-11-15T06:30:42.695 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:42.689+0000 7f57ba852700 1 -- start start 2023-11-15T06:30:42.695 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:42.689+0000 7f57ba852700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57b40f6510 0x7f57b406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:30:42.695 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:42.689+0000 7f57ba852700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b4069e20 con 0x7f57b40f6510 2023-11-15T06:30:42.695 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:42.689+0000 7f57b3fff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57b40f6510 0x7f57b406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:30:42.696 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:42.689+0000 7f57b3fff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57b40f6510 0x7f57b406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:37336/0 (socket says 172.21.15.81:37336) 2023-11-15T06:30:42.696 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:42.689+0000 7f57b3fff700 1 -- 172.21.15.81:0/406008882 learned_addr learned my addr 172.21.15.81:0/406008882 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:30:42.696 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:42.690+0000 7f57b3fff700 1 -- 172.21.15.81:0/406008882 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a4005040 con 0x7f57b40f6510 2023-11-15T06:30:42.696 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:42.690+0000 7f57b3fff700 1 --2- 172.21.15.81:0/406008882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57b40f6510 0x7f57b406b9a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f57a4013510 tx=0x7f57a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:30:42.697 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:42.690+0000 7f57b17fa700 1 -- 172.21.15.81:0/406008882 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57a4009320 con 0x7f57b40f6510 2023-11-15T06:30:42.697 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:42.690+0000 7f57ba852700 1 -- 172.21.15.81:0/406008882 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57b4069fc0 con 0x7f57b40f6510 2023-11-15T06:30:42.697 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:42.690+0000 7f57b17fa700 1 -- 172.21.15.81:0/406008882 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f57a4013f00 con 0x7f57b40f6510 2023-11-15T06:30:42.697 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:42.690+0000 7f57b17fa700 1 -- 172.21.15.81:0/406008882 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57a4014d90 con 0x7f57b40f6510 2023-11-15T06:30:42.697 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:42.690+0000 7f57ba852700 1 -- 172.21.15.81:0/406008882 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57b406a450 con 0x7f57b40f6510 2023-11-15T06:30:42.699 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:42.691+0000 7f57b17fa700 1 -- 172.21.15.81:0/406008882 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f57a40095f0 con 0x7f57b40f6510 2023-11-15T06:30:42.699 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:42.691+0000 7f57ba852700 1 -- 172.21.15.81:0/406008882 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57b4059100 con 0x7f57b40f6510 2023-11-15T06:30:42.699 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:42.691+0000 7f57b17fa700 1 --2- 172.21.15.81:0/406008882 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f57a002ff60 0x7f57a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:30:42.699 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:42.691+0000 7f57b17fa700 1 -- 172.21.15.81:0/406008882 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f57a4041000 con 0x7f57b40f6510 2023-11-15T06:30:42.700 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:42.691+0000 7f57abfff700 1 --2- 172.21.15.81:0/406008882 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f57a002ff60 0x7f57a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:30:42.700 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:42.691+0000 7f57abfff700 1 --2- 172.21.15.81:0/406008882 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f57a002ff60 0x7f57a0032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f579c0096a0 tx=0x7f579c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:30:42.700 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:42.695+0000 7f57b17fa700 1 -- 172.21.15.81:0/406008882 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f57a4014420 con 0x7f57b40f6510 2023-11-15T06:30:42.901 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:42.898+0000 7f57ba852700 1 -- 172.21.15.81:0/406008882 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f57b40fbf60 con 0x7f57b40f6510 2023-11-15T06:30:42.902 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:42.899+0000 7f57b17fa700 1 -- 172.21.15.81:0/406008882 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f57a401d7f0 con 0x7f57b40f6510 2023-11-15T06:30:42.903 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:30:42.903 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:30:42.905 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:42.901+0000 7f57ba852700 1 -- 172.21.15.81:0/406008882 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f57a002ff60 msgr2=0x7f57a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:42.905 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:42.901+0000 7f57ba852700 1 --2- 172.21.15.81:0/406008882 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f57a002ff60 0x7f57a0032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f579c0096a0 tx=0x7f579c006b40).stop 2023-11-15T06:30:42.905 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:42.901+0000 7f57ba852700 1 -- 172.21.15.81:0/406008882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57b40f6510 msgr2=0x7f57b406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:42.906 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:42.901+0000 7f57ba852700 1 --2- 172.21.15.81:0/406008882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57b40f6510 0x7f57b406b9a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f57a4013510 tx=0x7f57a400b040).stop 2023-11-15T06:30:42.906 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:42.901+0000 7f57ba852700 1 -- 172.21.15.81:0/406008882 shutdown_connections 2023-11-15T06:30:42.906 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:42.901+0000 7f57ba852700 1 --2- 172.21.15.81:0/406008882 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f57a002ff60 0x7f57a0032420 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:42.907 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:42.901+0000 7f57ba852700 1 --2- 172.21.15.81:0/406008882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57b40f6510 0x7f57b406b9a0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:42.907 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:42.901+0000 7f57ba852700 1 -- 172.21.15.81:0/406008882 >> 172.21.15.81:0/406008882 conn(0x7f57b40f1c70 msgr2=0x7f57b40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:30:42.907 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:42.901+0000 7f57ba852700 1 -- 172.21.15.81:0/406008882 shutdown_connections 2023-11-15T06:30:42.908 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:42.901+0000 7f57ba852700 1 -- 172.21.15.81:0/406008882 wait complete. 2023-11-15T06:30:42.908 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:30:43.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:42 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:30:41.824162+0000 mgr.smithi062.rwurtk (mgr.14164) 145 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:30:44.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:43 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:30:42.901051+0000 mon.smithi062 (mon.0) 216 : audit [DBG] from='client.? 172.21.15.81:0/406008882' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:30:44.292 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:30:44.293 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:30:45.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:44 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:30:43.824496+0000 mgr.smithi062.rwurtk (mgr.14164) 146 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:30:45.492 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:45.489+0000 7f1a5d368700 1 -- 172.21.15.81:0/4252244725 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a580f51a0 msgr2=0x7f1a580f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:45.493 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:45.489+0000 7f1a5d368700 1 --2- 172.21.15.81:0/4252244725 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a580f51a0 0x7f1a580f55c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f1a48009fb0 tx=0x7f1a48009b30).stop 2023-11-15T06:30:45.493 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:45.489+0000 7f1a5d368700 1 -- 172.21.15.81:0/4252244725 shutdown_connections 2023-11-15T06:30:45.493 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:45.489+0000 7f1a5d368700 1 --2- 172.21.15.81:0/4252244725 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a580f51a0 0x7f1a580f55c0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:45.493 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:45.489+0000 7f1a5d368700 1 -- 172.21.15.81:0/4252244725 >> 172.21.15.81:0/4252244725 conn(0x7f1a580f09a0 msgr2=0x7f1a580f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:30:45.493 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:45.489+0000 7f1a5d368700 1 -- 172.21.15.81:0/4252244725 shutdown_connections 2023-11-15T06:30:45.494 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:45.489+0000 7f1a5d368700 1 -- 172.21.15.81:0/4252244725 wait complete. 2023-11-15T06:30:45.494 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:45.489+0000 7f1a5d368700 1 Processor -- start 2023-11-15T06:30:45.494 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:45.489+0000 7f1a5d368700 1 -- start start 2023-11-15T06:30:45.494 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:45.489+0000 7f1a5d368700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a580f51a0 0x7f1a581053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:30:45.494 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:45.489+0000 7f1a5d368700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a581058a0 con 0x7f1a580f51a0 2023-11-15T06:30:45.496 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:45.490+0000 7f1a57fff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a580f51a0 0x7f1a581053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:30:45.496 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:45.490+0000 7f1a57fff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a580f51a0 0x7f1a581053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:37354/0 (socket says 172.21.15.81:37354) 2023-11-15T06:30:45.497 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:45.490+0000 7f1a57fff700 1 -- 172.21.15.81:0/470168992 learned_addr learned my addr 172.21.15.81:0/470168992 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:30:45.497 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:45.490+0000 7f1a57fff700 1 -- 172.21.15.81:0/470168992 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a48005040 con 0x7f1a580f51a0 2023-11-15T06:30:45.497 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:45.490+0000 7f1a57fff700 1 --2- 172.21.15.81:0/470168992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a580f51a0 0x7f1a581053b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f1a48013510 tx=0x7f1a4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:30:45.497 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:45.490+0000 7f1a54ff9700 1 -- 172.21.15.81:0/470168992 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a48009320 con 0x7f1a580f51a0 2023-11-15T06:30:45.497 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:45.490+0000 7f1a54ff9700 1 -- 172.21.15.81:0/470168992 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1a48013f00 con 0x7f1a580f51a0 2023-11-15T06:30:45.498 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:45.490+0000 7f1a5d368700 1 -- 172.21.15.81:0/470168992 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a58105aa0 con 0x7f1a580f51a0 2023-11-15T06:30:45.498 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:45.490+0000 7f1a54ff9700 1 -- 172.21.15.81:0/470168992 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a48014dc0 con 0x7f1a580f51a0 2023-11-15T06:30:45.498 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:45.490+0000 7f1a5d368700 1 -- 172.21.15.81:0/470168992 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a58105f30 con 0x7f1a580f51a0 2023-11-15T06:30:45.498 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:45.491+0000 7f1a54ff9700 1 -- 172.21.15.81:0/470168992 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1a48014420 con 0x7f1a580f51a0 2023-11-15T06:30:45.498 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:45.491+0000 7f1a5d368700 1 -- 172.21.15.81:0/470168992 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a58045ad0 con 0x7f1a580f51a0 2023-11-15T06:30:45.499 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:45.491+0000 7f1a54ff9700 1 --2- 172.21.15.81:0/470168992 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f1a4002ff60 0x7f1a40032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:30:45.499 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:45.491+0000 7f1a54ff9700 1 -- 172.21.15.81:0/470168992 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1a48042ee0 con 0x7f1a580f51a0 2023-11-15T06:30:45.499 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:45.492+0000 7f1a577fe700 1 --2- 172.21.15.81:0/470168992 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f1a4002ff60 0x7f1a40032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:30:45.499 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:45.492+0000 7f1a577fe700 1 --2- 172.21.15.81:0/470168992 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f1a4002ff60 0x7f1a40032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f1a4c0096a0 tx=0x7f1a4c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:30:45.500 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:45.496+0000 7f1a54ff9700 1 -- 172.21.15.81:0/470168992 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1a480095f0 con 0x7f1a580f51a0 2023-11-15T06:30:45.702 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:45.699+0000 7f1a5d368700 1 -- 172.21.15.81:0/470168992 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1a58106de0 con 0x7f1a580f51a0 2023-11-15T06:30:45.703 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:45.700+0000 7f1a54ff9700 1 -- 172.21.15.81:0/470168992 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1a4801fb70 con 0x7f1a580f51a0 2023-11-15T06:30:45.703 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:30:45.703 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:30:45.704 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:45.701+0000 7f1a5d368700 1 -- 172.21.15.81:0/470168992 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f1a4002ff60 msgr2=0x7f1a40032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:45.705 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:45.701+0000 7f1a5d368700 1 --2- 172.21.15.81:0/470168992 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f1a4002ff60 0x7f1a40032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f1a4c0096a0 tx=0x7f1a4c006b40).stop 2023-11-15T06:30:45.705 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:45.701+0000 7f1a5d368700 1 -- 172.21.15.81:0/470168992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a580f51a0 msgr2=0x7f1a581053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:45.705 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:45.701+0000 7f1a5d368700 1 --2- 172.21.15.81:0/470168992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a580f51a0 0x7f1a581053b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f1a48013510 tx=0x7f1a4800b040).stop 2023-11-15T06:30:45.705 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:45.701+0000 7f1a5d368700 1 -- 172.21.15.81:0/470168992 shutdown_connections 2023-11-15T06:30:45.705 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:45.701+0000 7f1a5d368700 1 --2- 172.21.15.81:0/470168992 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f1a4002ff60 0x7f1a40032420 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:45.706 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:45.701+0000 7f1a5d368700 1 --2- 172.21.15.81:0/470168992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a580f51a0 0x7f1a581053b0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:45.706 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:45.701+0000 7f1a5d368700 1 -- 172.21.15.81:0/470168992 >> 172.21.15.81:0/470168992 conn(0x7f1a580f09a0 msgr2=0x7f1a580f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:30:45.706 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:45.701+0000 7f1a5d368700 1 -- 172.21.15.81:0/470168992 shutdown_connections 2023-11-15T06:30:45.707 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:45.701+0000 7f1a5d368700 1 -- 172.21.15.81:0/470168992 wait complete. 2023-11-15T06:30:45.707 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:30:46.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:45 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:30:45.701680+0000 mon.smithi062 (mon.0) 217 : audit [DBG] from='client.? 172.21.15.81:0/470168992' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:30:47.079 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:30:47.080 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:30:47.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:46 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:30:45.824746+0000 mgr.smithi062.rwurtk (mgr.14164) 147 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:30:48.236 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:48.233+0000 7f7d659c8700 1 -- 172.21.15.81:0/36445598 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d600f5410 msgr2=0x7f7d600f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:48.237 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:48.233+0000 7f7d659c8700 1 --2- 172.21.15.81:0/36445598 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d600f5410 0x7f7d600f5830 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f7d54009fb0 tx=0x7f7d54009b30).stop 2023-11-15T06:30:48.237 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:48.233+0000 7f7d659c8700 1 -- 172.21.15.81:0/36445598 shutdown_connections 2023-11-15T06:30:48.237 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:48.233+0000 7f7d659c8700 1 --2- 172.21.15.81:0/36445598 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d600f5410 0x7f7d600f5830 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:48.237 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:48.233+0000 7f7d659c8700 1 -- 172.21.15.81:0/36445598 >> 172.21.15.81:0/36445598 conn(0x7f7d600f0b70 msgr2=0x7f7d600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:30:48.238 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:48.233+0000 7f7d659c8700 1 -- 172.21.15.81:0/36445598 shutdown_connections 2023-11-15T06:30:48.238 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:48.233+0000 7f7d659c8700 1 -- 172.21.15.81:0/36445598 wait complete. 2023-11-15T06:30:48.240 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:48.234+0000 7f7d659c8700 1 Processor -- start 2023-11-15T06:30:48.240 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:48.234+0000 7f7d659c8700 1 -- start start 2023-11-15T06:30:48.240 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:48.234+0000 7f7d659c8700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d600f5410 0x7f7d60106760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:30:48.240 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:48.234+0000 7f7d659c8700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d60106c50 con 0x7f7d600f5410 2023-11-15T06:30:48.241 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:48.234+0000 7f7d5effd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d600f5410 0x7f7d60106760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:30:48.241 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:48.234+0000 7f7d5effd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d600f5410 0x7f7d60106760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:37374/0 (socket says 172.21.15.81:37374) 2023-11-15T06:30:48.242 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:48.234+0000 7f7d5effd700 1 -- 172.21.15.81:0/2664825670 learned_addr learned my addr 172.21.15.81:0/2664825670 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:30:48.242 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:48.235+0000 7f7d5effd700 1 -- 172.21.15.81:0/2664825670 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d54005040 con 0x7f7d600f5410 2023-11-15T06:30:48.242 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:48.235+0000 7f7d5effd700 1 --2- 172.21.15.81:0/2664825670 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d600f5410 0x7f7d60106760 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f7d54009f80 tx=0x7f7d5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:30:48.242 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:48.235+0000 7f7d477fe700 1 -- 172.21.15.81:0/2664825670 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d54009320 con 0x7f7d600f5410 2023-11-15T06:30:48.242 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:48.235+0000 7f7d659c8700 1 -- 172.21.15.81:0/2664825670 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d60106e50 con 0x7f7d600f5410 2023-11-15T06:30:48.243 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:48.235+0000 7f7d477fe700 1 -- 172.21.15.81:0/2664825670 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7d54013ec0 con 0x7f7d600f5410 2023-11-15T06:30:48.243 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:48.235+0000 7f7d477fe700 1 -- 172.21.15.81:0/2664825670 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d5401ece0 con 0x7f7d600f5410 2023-11-15T06:30:48.243 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:48.235+0000 7f7d659c8700 1 -- 172.21.15.81:0/2664825670 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d601072e0 con 0x7f7d600f5410 2023-11-15T06:30:48.244 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:48.236+0000 7f7d477fe700 1 -- 172.21.15.81:0/2664825670 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7d54014420 con 0x7f7d600f5410 2023-11-15T06:30:48.244 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:48.236+0000 7f7d477fe700 1 --2- 172.21.15.81:0/2664825670 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f7d4802ff60 0x7f7d48032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:30:48.244 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:48.236+0000 7f7d477fe700 1 -- 172.21.15.81:0/2664825670 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7d54040d00 con 0x7f7d600f5410 2023-11-15T06:30:48.244 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:48.236+0000 7f7d659c8700 1 -- 172.21.15.81:0/2664825670 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d60045ad0 con 0x7f7d600f5410 2023-11-15T06:30:48.246 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:48.236+0000 7f7d5e7fc700 1 --2- 172.21.15.81:0/2664825670 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f7d4802ff60 0x7f7d48032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:30:48.246 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:48.237+0000 7f7d5e7fc700 1 --2- 172.21.15.81:0/2664825670 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f7d4802ff60 0x7f7d48032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f7d500096a0 tx=0x7f7d50006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:30:48.247 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:48.241+0000 7f7d477fe700 1 -- 172.21.15.81:0/2664825670 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7d5401c910 con 0x7f7d600f5410 2023-11-15T06:30:48.456 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:48.453+0000 7f7d659c8700 1 -- 172.21.15.81:0/2664825670 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7d60108020 con 0x7f7d600f5410 2023-11-15T06:30:48.457 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:48.454+0000 7f7d477fe700 1 -- 172.21.15.81:0/2664825670 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7d5401c320 con 0x7f7d600f5410 2023-11-15T06:30:48.457 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:30:48.457 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:30:48.458 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:48.455+0000 7f7d659c8700 1 -- 172.21.15.81:0/2664825670 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f7d4802ff60 msgr2=0x7f7d48032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:48.458 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:48.455+0000 7f7d659c8700 1 --2- 172.21.15.81:0/2664825670 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f7d4802ff60 0x7f7d48032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f7d500096a0 tx=0x7f7d50006b40).stop 2023-11-15T06:30:48.459 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:48.455+0000 7f7d659c8700 1 -- 172.21.15.81:0/2664825670 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d600f5410 msgr2=0x7f7d60106760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:48.459 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:48.455+0000 7f7d659c8700 1 --2- 172.21.15.81:0/2664825670 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d600f5410 0x7f7d60106760 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f7d54009f80 tx=0x7f7d5400b040).stop 2023-11-15T06:30:48.459 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:48.455+0000 7f7d659c8700 1 -- 172.21.15.81:0/2664825670 shutdown_connections 2023-11-15T06:30:48.459 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:48.455+0000 7f7d659c8700 1 --2- 172.21.15.81:0/2664825670 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f7d4802ff60 0x7f7d48032420 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:48.459 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:48.455+0000 7f7d659c8700 1 --2- 172.21.15.81:0/2664825670 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d600f5410 0x7f7d60106760 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:48.460 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:48.455+0000 7f7d659c8700 1 -- 172.21.15.81:0/2664825670 >> 172.21.15.81:0/2664825670 conn(0x7f7d600f0b70 msgr2=0x7f7d6005fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:30:48.460 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:48.455+0000 7f7d659c8700 1 -- 172.21.15.81:0/2664825670 shutdown_connections 2023-11-15T06:30:48.460 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:48.455+0000 7f7d659c8700 1 -- 172.21.15.81:0/2664825670 wait complete. 2023-11-15T06:30:48.460 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:30:48.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:48 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:30:47.825069+0000 mgr.smithi062.rwurtk (mgr.14164) 148 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:30:48.919 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:48 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:30:48.455520+0000 mon.smithi062 (mon.0) 218 : audit [DBG] from='client.? 172.21.15.81:0/2664825670' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:30:49.948 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:30:49.949 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:30:51.167 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:51.161+0000 7fa923646700 1 -- 172.21.15.81:0/3979457416 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa91c0f6510 msgr2=0x7fa91c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:51.168 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:51.161+0000 7fa923646700 1 --2- 172.21.15.81:0/3979457416 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa91c0f6510 0x7fa91c0f6930 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7fa90c009fb0 tx=0x7fa90c009b30).stop 2023-11-15T06:30:51.168 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:51.161+0000 7fa923646700 1 -- 172.21.15.81:0/3979457416 shutdown_connections 2023-11-15T06:30:51.168 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:51.161+0000 7fa923646700 1 --2- 172.21.15.81:0/3979457416 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa91c0f6510 0x7fa91c0f6930 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:51.168 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:51.161+0000 7fa923646700 1 -- 172.21.15.81:0/3979457416 >> 172.21.15.81:0/3979457416 conn(0x7fa91c0f1c70 msgr2=0x7fa91c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:30:51.169 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:51.161+0000 7fa923646700 1 -- 172.21.15.81:0/3979457416 shutdown_connections 2023-11-15T06:30:51.169 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:51.161+0000 7fa923646700 1 -- 172.21.15.81:0/3979457416 wait complete. 2023-11-15T06:30:51.169 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:51.162+0000 7fa923646700 1 Processor -- start 2023-11-15T06:30:51.170 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:51.162+0000 7fa923646700 1 -- start start 2023-11-15T06:30:51.170 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:51.162+0000 7fa923646700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa91c0f6510 0x7fa91c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:30:51.170 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:51.162+0000 7fa923646700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa91c106c20 con 0x7fa91c0f6510 2023-11-15T06:30:51.170 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:51.162+0000 7fa9213e2700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa91c0f6510 0x7fa91c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:30:51.171 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:51.162+0000 7fa9213e2700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa91c0f6510 0x7fa91c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:37386/0 (socket says 172.21.15.81:37386) 2023-11-15T06:30:51.171 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:51.162+0000 7fa9213e2700 1 -- 172.21.15.81:0/3150108837 learned_addr learned my addr 172.21.15.81:0/3150108837 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:30:51.171 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:51.162+0000 7fa9213e2700 1 -- 172.21.15.81:0/3150108837 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa90c005040 con 0x7fa91c0f6510 2023-11-15T06:30:51.171 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:51.163+0000 7fa9213e2700 1 --2- 172.21.15.81:0/3150108837 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa91c0f6510 0x7fa91c106730 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7fa90c013510 tx=0x7fa90c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:30:51.171 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:51.163+0000 7fa911ffb700 1 -- 172.21.15.81:0/3150108837 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa90c009320 con 0x7fa91c0f6510 2023-11-15T06:30:51.172 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:51.163+0000 7fa923646700 1 -- 172.21.15.81:0/3150108837 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa91c106e20 con 0x7fa91c0f6510 2023-11-15T06:30:51.172 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:51.163+0000 7fa911ffb700 1 -- 172.21.15.81:0/3150108837 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa90c013f00 con 0x7fa91c0f6510 2023-11-15T06:30:51.172 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:51.163+0000 7fa911ffb700 1 -- 172.21.15.81:0/3150108837 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa90c014d90 con 0x7fa91c0f6510 2023-11-15T06:30:51.172 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:51.163+0000 7fa923646700 1 -- 172.21.15.81:0/3150108837 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa91c1072b0 con 0x7fa91c0f6510 2023-11-15T06:30:51.172 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:51.164+0000 7fa911ffb700 1 -- 172.21.15.81:0/3150108837 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa90c0095f0 con 0x7fa91c0f6510 2023-11-15T06:30:51.172 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:51.164+0000 7fa923646700 1 -- 172.21.15.81:0/3150108837 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa91c045ad0 con 0x7fa91c0f6510 2023-11-15T06:30:51.173 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:51.164+0000 7fa911ffb700 1 --2- 172.21.15.81:0/3150108837 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fa90802ff60 0x7fa908032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:30:51.173 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:51.164+0000 7fa911ffb700 1 -- 172.21.15.81:0/3150108837 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa90c041000 con 0x7fa91c0f6510 2023-11-15T06:30:51.173 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:51.164+0000 7fa920be1700 1 --2- 172.21.15.81:0/3150108837 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fa90802ff60 0x7fa908032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:30:51.173 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:51.165+0000 7fa920be1700 1 --2- 172.21.15.81:0/3150108837 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fa90802ff60 0x7fa908032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fa9180096a0 tx=0x7fa918006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:30:51.174 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:50 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:30:49.825377+0000 mgr.smithi062.rwurtk (mgr.14164) 149 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:30:51.175 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:51.169+0000 7fa911ffb700 1 -- 172.21.15.81:0/3150108837 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa90c01d7f0 con 0x7fa91c0f6510 2023-11-15T06:30:51.376 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:51.374+0000 7fa923646700 1 -- 172.21.15.81:0/3150108837 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa91c0fbf60 con 0x7fa91c0f6510 2023-11-15T06:30:51.377 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:51.374+0000 7fa911ffb700 1 -- 172.21.15.81:0/3150108837 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa90c010230 con 0x7fa91c0f6510 2023-11-15T06:30:51.377 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:30:51.377 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:30:51.378 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:51.376+0000 7fa923646700 1 -- 172.21.15.81:0/3150108837 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fa90802ff60 msgr2=0x7fa908032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:51.378 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:51.376+0000 7fa923646700 1 --2- 172.21.15.81:0/3150108837 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fa90802ff60 0x7fa908032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fa9180096a0 tx=0x7fa918006b40).stop 2023-11-15T06:30:51.379 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:51.376+0000 7fa923646700 1 -- 172.21.15.81:0/3150108837 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa91c0f6510 msgr2=0x7fa91c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:51.379 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:51.376+0000 7fa923646700 1 --2- 172.21.15.81:0/3150108837 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa91c0f6510 0x7fa91c106730 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7fa90c013510 tx=0x7fa90c00b040).stop 2023-11-15T06:30:51.379 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:51.376+0000 7fa923646700 1 -- 172.21.15.81:0/3150108837 shutdown_connections 2023-11-15T06:30:51.379 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:51.376+0000 7fa923646700 1 --2- 172.21.15.81:0/3150108837 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fa90802ff60 0x7fa908032420 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:51.379 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:51.376+0000 7fa923646700 1 --2- 172.21.15.81:0/3150108837 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa91c0f6510 0x7fa91c106730 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:51.380 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:51.376+0000 7fa923646700 1 -- 172.21.15.81:0/3150108837 >> 172.21.15.81:0/3150108837 conn(0x7fa91c0f1c70 msgr2=0x7fa91c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:30:51.380 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:51.376+0000 7fa923646700 1 -- 172.21.15.81:0/3150108837 shutdown_connections 2023-11-15T06:30:51.380 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:51.376+0000 7fa923646700 1 -- 172.21.15.81:0/3150108837 wait complete. 2023-11-15T06:30:51.381 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:30:52.169 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:51 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:30:51.375290+0000 mon.smithi062 (mon.0) 219 : audit [DBG] from='client.? 172.21.15.81:0/3150108837' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:30:52.758 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:30:52.758 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:30:53.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:52 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:30:51.825650+0000 mgr.smithi062.rwurtk (mgr.14164) 150 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:30:53.908 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:53.904+0000 7f7e27bca700 1 -- 172.21.15.81:0/2948144108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e20060280 msgr2=0x7f7e200f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:53.908 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:53.904+0000 7f7e27bca700 1 --2- 172.21.15.81:0/2948144108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e20060280 0x7f7e200f90f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f7e10009fb0 tx=0x7f7e10009b30).stop 2023-11-15T06:30:53.909 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:53.904+0000 7f7e27bca700 1 -- 172.21.15.81:0/2948144108 shutdown_connections 2023-11-15T06:30:53.909 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:53.904+0000 7f7e27bca700 1 --2- 172.21.15.81:0/2948144108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e20060280 0x7f7e200f90f0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:53.909 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:53.904+0000 7f7e27bca700 1 -- 172.21.15.81:0/2948144108 >> 172.21.15.81:0/2948144108 conn(0x7f7e200f0b70 msgr2=0x7f7e200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:30:53.909 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:53.904+0000 7f7e27bca700 1 -- 172.21.15.81:0/2948144108 shutdown_connections 2023-11-15T06:30:53.909 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:53.904+0000 7f7e27bca700 1 -- 172.21.15.81:0/2948144108 wait complete. 2023-11-15T06:30:53.910 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:53.905+0000 7f7e27bca700 1 Processor -- start 2023-11-15T06:30:53.910 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:53.905+0000 7f7e27bca700 1 -- start start 2023-11-15T06:30:53.910 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:53.905+0000 7f7e27bca700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e20060280 0x7f7e200f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:30:53.910 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:53.905+0000 7f7e27bca700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e200f4ff0 con 0x7f7e20060280 2023-11-15T06:30:53.911 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:53.905+0000 7f7e25966700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e20060280 0x7f7e200f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:30:53.911 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:53.905+0000 7f7e25966700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e20060280 0x7f7e200f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:57028/0 (socket says 172.21.15.81:57028) 2023-11-15T06:30:53.911 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:53.905+0000 7f7e25966700 1 -- 172.21.15.81:0/2247718766 learned_addr learned my addr 172.21.15.81:0/2247718766 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:30:53.911 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:53.905+0000 7f7e25966700 1 -- 172.21.15.81:0/2247718766 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e10005040 con 0x7f7e20060280 2023-11-15T06:30:53.912 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:53.906+0000 7f7e25966700 1 --2- 172.21.15.81:0/2247718766 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e20060280 0x7f7e200f6b70 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f7e10013510 tx=0x7f7e1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:30:53.912 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:53.906+0000 7f7e167fc700 1 -- 172.21.15.81:0/2247718766 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e10009320 con 0x7f7e20060280 2023-11-15T06:30:53.912 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:53.906+0000 7f7e27bca700 1 -- 172.21.15.81:0/2247718766 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e200f5190 con 0x7f7e20060280 2023-11-15T06:30:53.912 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:53.906+0000 7f7e167fc700 1 -- 172.21.15.81:0/2247718766 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7e10013f00 con 0x7f7e20060280 2023-11-15T06:30:53.913 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:53.906+0000 7f7e167fc700 1 -- 172.21.15.81:0/2247718766 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e10014dc0 con 0x7f7e20060280 2023-11-15T06:30:53.913 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:53.906+0000 7f7e27bca700 1 -- 172.21.15.81:0/2247718766 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e200f5620 con 0x7f7e20060280 2023-11-15T06:30:53.915 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:53.907+0000 7f7e167fc700 1 -- 172.21.15.81:0/2247718766 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7e100095f0 con 0x7f7e20060280 2023-11-15T06:30:53.915 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:53.907+0000 7f7e27bca700 1 -- 172.21.15.81:0/2247718766 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e20059100 con 0x7f7e20060280 2023-11-15T06:30:53.915 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:53.907+0000 7f7e167fc700 1 --2- 172.21.15.81:0/2247718766 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f7e0c02ff60 0x7f7e0c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:30:53.916 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:53.907+0000 7f7e167fc700 1 -- 172.21.15.81:0/2247718766 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7e10041000 con 0x7f7e20060280 2023-11-15T06:30:53.916 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:53.907+0000 7f7e25165700 1 --2- 172.21.15.81:0/2247718766 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f7e0c02ff60 0x7f7e0c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:30:53.916 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:53.907+0000 7f7e25165700 1 --2- 172.21.15.81:0/2247718766 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f7e0c02ff60 0x7f7e0c032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f7e1c0096a0 tx=0x7f7e1c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:30:53.916 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:53.911+0000 7f7e167fc700 1 -- 172.21.15.81:0/2247718766 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7e10014420 con 0x7f7e20060280 2023-11-15T06:30:54.123 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:54.121+0000 7f7e27bca700 1 -- 172.21.15.81:0/2247718766 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7e200fbf50 con 0x7f7e20060280 2023-11-15T06:30:54.124 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:54.122+0000 7f7e167fc700 1 -- 172.21.15.81:0/2247718766 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7e10010230 con 0x7f7e20060280 2023-11-15T06:30:54.124 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:30:54.124 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:30:54.126 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:54.123+0000 7f7e27bca700 1 -- 172.21.15.81:0/2247718766 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f7e0c02ff60 msgr2=0x7f7e0c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:54.126 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:54.123+0000 7f7e27bca700 1 --2- 172.21.15.81:0/2247718766 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f7e0c02ff60 0x7f7e0c032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f7e1c0096a0 tx=0x7f7e1c006b40).stop 2023-11-15T06:30:54.126 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:54.123+0000 7f7e27bca700 1 -- 172.21.15.81:0/2247718766 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e20060280 msgr2=0x7f7e200f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:54.127 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:54.123+0000 7f7e27bca700 1 --2- 172.21.15.81:0/2247718766 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e20060280 0x7f7e200f6b70 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f7e10013510 tx=0x7f7e1000b040).stop 2023-11-15T06:30:54.127 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:54.123+0000 7f7e27bca700 1 -- 172.21.15.81:0/2247718766 shutdown_connections 2023-11-15T06:30:54.127 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:54.123+0000 7f7e27bca700 1 --2- 172.21.15.81:0/2247718766 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f7e0c02ff60 0x7f7e0c032420 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:54.127 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:54.123+0000 7f7e27bca700 1 --2- 172.21.15.81:0/2247718766 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e20060280 0x7f7e200f6b70 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:54.127 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:54.123+0000 7f7e27bca700 1 -- 172.21.15.81:0/2247718766 >> 172.21.15.81:0/2247718766 conn(0x7f7e200f0b70 msgr2=0x7f7e200f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:30:54.128 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:54.123+0000 7f7e27bca700 1 -- 172.21.15.81:0/2247718766 shutdown_connections 2023-11-15T06:30:54.128 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:54.123+0000 7f7e27bca700 1 -- 172.21.15.81:0/2247718766 wait complete. 2023-11-15T06:30:54.128 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:30:55.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:54 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:30:53.825919+0000 mgr.smithi062.rwurtk (mgr.14164) 151 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:30:55.169 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:54 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:30:53.918437+0000 mon.smithi062 (mon.0) 220 : audit [INF] from='mgr.14164 172.21.15.62:0/147479783' entity='mgr.smithi062.rwurtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi062.rwurtk/mirror_snapshot_schedule"}]: dispatch 2023-11-15T06:30:55.169 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:54 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:30:53.919778+0000 mon.smithi062 (mon.0) 221 : audit [INF] from='mgr.14164 172.21.15.62:0/147479783' entity='mgr.smithi062.rwurtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi062.rwurtk/trash_purge_schedule"}]: dispatch 2023-11-15T06:30:55.169 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:54 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:30:54.122710+0000 mon.smithi062 (mon.0) 222 : audit [DBG] from='client.? 172.21.15.81:0/2247718766' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:30:55.511 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:30:55.512 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:30:56.716 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:56.713+0000 7f1ad25a0700 1 -- 172.21.15.81:0/687223969 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1acc0f51a0 msgr2=0x7f1acc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:56.716 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:56.713+0000 7f1ad25a0700 1 --2- 172.21.15.81:0/687223969 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1acc0f51a0 0x7f1acc0f55c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f1ac8009fb0 tx=0x7f1ac8009b30).stop 2023-11-15T06:30:56.716 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:56.713+0000 7f1ad25a0700 1 -- 172.21.15.81:0/687223969 shutdown_connections 2023-11-15T06:30:56.717 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:56.713+0000 7f1ad25a0700 1 --2- 172.21.15.81:0/687223969 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1acc0f51a0 0x7f1acc0f55c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:56.717 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:56.713+0000 7f1ad25a0700 1 -- 172.21.15.81:0/687223969 >> 172.21.15.81:0/687223969 conn(0x7f1acc0f09a0 msgr2=0x7f1acc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:30:56.717 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:56.713+0000 7f1ad25a0700 1 -- 172.21.15.81:0/687223969 shutdown_connections 2023-11-15T06:30:56.717 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:56.713+0000 7f1ad25a0700 1 -- 172.21.15.81:0/687223969 wait complete. 2023-11-15T06:30:56.717 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:56.713+0000 7f1ad25a0700 1 Processor -- start 2023-11-15T06:30:56.718 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:56.714+0000 7f1ad25a0700 1 -- start start 2023-11-15T06:30:56.718 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:56.714+0000 7f1ad25a0700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1acc0f51a0 0x7f1acc1091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:30:56.718 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:56.714+0000 7f1ad25a0700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1acc060240 con 0x7f1acc0f51a0 2023-11-15T06:30:56.718 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:56.714+0000 7f1ad159e700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1acc0f51a0 0x7f1acc1091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:30:56.718 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:56.714+0000 7f1ad159e700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1acc0f51a0 0x7f1acc1091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:57046/0 (socket says 172.21.15.81:57046) 2023-11-15T06:30:56.719 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:56.714+0000 7f1ad159e700 1 -- 172.21.15.81:0/11132312 learned_addr learned my addr 172.21.15.81:0/11132312 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:30:56.720 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:56.714+0000 7f1ad159e700 1 -- 172.21.15.81:0/11132312 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac8005040 con 0x7f1acc0f51a0 2023-11-15T06:30:56.720 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:56.715+0000 7f1ad159e700 1 --2- 172.21.15.81:0/11132312 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1acc0f51a0 0x7f1acc1091b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f1ac8013510 tx=0x7f1ac800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:30:56.721 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:56.715+0000 7f1ac1ffb700 1 -- 172.21.15.81:0/11132312 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ac8009320 con 0x7f1acc0f51a0 2023-11-15T06:30:56.721 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:56.715+0000 7f1ad25a0700 1 -- 172.21.15.81:0/11132312 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1acc060440 con 0x7f1acc0f51a0 2023-11-15T06:30:56.721 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:56.715+0000 7f1ac1ffb700 1 -- 172.21.15.81:0/11132312 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1ac8013f00 con 0x7f1acc0f51a0 2023-11-15T06:30:56.721 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:56.715+0000 7f1ac1ffb700 1 -- 172.21.15.81:0/11132312 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ac8014dc0 con 0x7f1acc0f51a0 2023-11-15T06:30:56.721 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:56.715+0000 7f1ad25a0700 1 -- 172.21.15.81:0/11132312 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1acc109860 con 0x7f1acc0f51a0 2023-11-15T06:30:56.722 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:56.716+0000 7f1ac1ffb700 1 -- 172.21.15.81:0/11132312 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1ac8014420 con 0x7f1acc0f51a0 2023-11-15T06:30:56.722 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:56.716+0000 7f1ad25a0700 1 -- 172.21.15.81:0/11132312 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1acc045ad0 con 0x7f1acc0f51a0 2023-11-15T06:30:56.722 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:56.716+0000 7f1ac1ffb700 1 --2- 172.21.15.81:0/11132312 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f1ab802ff60 0x7f1ab8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:30:56.722 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:56.716+0000 7f1ac1ffb700 1 -- 172.21.15.81:0/11132312 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1ac8041030 con 0x7f1acc0f51a0 2023-11-15T06:30:56.722 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:56.716+0000 7f1ad0d9d700 1 --2- 172.21.15.81:0/11132312 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f1ab802ff60 0x7f1ab8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:30:56.723 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:56.716+0000 7f1ad0d9d700 1 --2- 172.21.15.81:0/11132312 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f1ab802ff60 0x7f1ab8032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f1abc0096a0 tx=0x7f1abc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:30:56.723 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:56.721+0000 7f1ac1ffb700 1 -- 172.21.15.81:0/11132312 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1ac80095f0 con 0x7f1acc0f51a0 2023-11-15T06:30:56.927 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:56.925+0000 7f1ad25a0700 1 -- 172.21.15.81:0/11132312 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1acc10a080 con 0x7f1acc0f51a0 2023-11-15T06:30:56.927 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:56.926+0000 7f1ac1ffb700 1 -- 172.21.15.81:0/11132312 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1ac801cb10 con 0x7f1acc0f51a0 2023-11-15T06:30:56.930 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:30:56.930 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:30:56.931 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:56.927+0000 7f1ad25a0700 1 -- 172.21.15.81:0/11132312 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f1ab802ff60 msgr2=0x7f1ab8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:56.931 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:56.927+0000 7f1ad25a0700 1 --2- 172.21.15.81:0/11132312 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f1ab802ff60 0x7f1ab8032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f1abc0096a0 tx=0x7f1abc006b40).stop 2023-11-15T06:30:56.931 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:56.927+0000 7f1ad25a0700 1 -- 172.21.15.81:0/11132312 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1acc0f51a0 msgr2=0x7f1acc1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:56.931 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:56.927+0000 7f1ad25a0700 1 --2- 172.21.15.81:0/11132312 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1acc0f51a0 0x7f1acc1091b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f1ac8013510 tx=0x7f1ac800b040).stop 2023-11-15T06:30:56.932 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:56.927+0000 7f1ad25a0700 1 -- 172.21.15.81:0/11132312 shutdown_connections 2023-11-15T06:30:56.932 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:56.927+0000 7f1ad25a0700 1 --2- 172.21.15.81:0/11132312 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f1ab802ff60 0x7f1ab8032420 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:56.932 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:56.927+0000 7f1ad25a0700 1 --2- 172.21.15.81:0/11132312 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1acc0f51a0 0x7f1acc1091b0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:56.932 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:56.927+0000 7f1ad25a0700 1 -- 172.21.15.81:0/11132312 >> 172.21.15.81:0/11132312 conn(0x7f1acc0f09a0 msgr2=0x7f1acc0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:30:56.932 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:56.927+0000 7f1ad25a0700 1 -- 172.21.15.81:0/11132312 shutdown_connections 2023-11-15T06:30:56.933 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:56.927+0000 7f1ad25a0700 1 -- 172.21.15.81:0/11132312 wait complete. 2023-11-15T06:30:56.933 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:30:57.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:56 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:30:55.826208+0000 mgr.smithi062.rwurtk (mgr.14164) 152 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:30:58.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:57 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:30:56.926467+0000 mon.smithi062 (mon.0) 223 : audit [DBG] from='client.? 172.21.15.81:0/11132312' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:30:58.361 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:30:58.362 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:30:59.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:58 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:30:57.826471+0000 mgr.smithi062.rwurtk (mgr.14164) 153 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:30:59.599 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:59.595+0000 7f054b8f6700 1 -- 172.21.15.81:0/1575176741 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05440f6510 msgr2=0x7f05440f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:59.599 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:59.595+0000 7f054b8f6700 1 --2- 172.21.15.81:0/1575176741 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05440f6510 0x7f05440f6930 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f0538009fb0 tx=0x7f0538009b30).stop 2023-11-15T06:30:59.599 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:59.595+0000 7f054b8f6700 1 -- 172.21.15.81:0/1575176741 shutdown_connections 2023-11-15T06:30:59.599 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:59.595+0000 7f054b8f6700 1 --2- 172.21.15.81:0/1575176741 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05440f6510 0x7f05440f6930 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:59.599 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:59.595+0000 7f054b8f6700 1 -- 172.21.15.81:0/1575176741 >> 172.21.15.81:0/1575176741 conn(0x7f05440f1c70 msgr2=0x7f05440f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:30:59.601 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:59.595+0000 7f054b8f6700 1 -- 172.21.15.81:0/1575176741 shutdown_connections 2023-11-15T06:30:59.601 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:59.595+0000 7f054b8f6700 1 -- 172.21.15.81:0/1575176741 wait complete. 2023-11-15T06:30:59.601 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:59.596+0000 7f054b8f6700 1 Processor -- start 2023-11-15T06:30:59.602 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:59.596+0000 7f054b8f6700 1 -- start start 2023-11-15T06:30:59.602 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:59.596+0000 7f054b8f6700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05440f6510 0x7f0544106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:30:59.602 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:59.596+0000 7f054b8f6700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0544106c20 con 0x7f05440f6510 2023-11-15T06:30:59.602 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:59.596+0000 7f0549692700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05440f6510 0x7f0544106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:30:59.602 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:59.596+0000 7f0549692700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05440f6510 0x7f0544106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:57064/0 (socket says 172.21.15.81:57064) 2023-11-15T06:30:59.603 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:59.596+0000 7f0549692700 1 -- 172.21.15.81:0/829394192 learned_addr learned my addr 172.21.15.81:0/829394192 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:30:59.603 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:59.596+0000 7f0549692700 1 -- 172.21.15.81:0/829394192 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0538005040 con 0x7f05440f6510 2023-11-15T06:30:59.603 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:59.597+0000 7f0549692700 1 --2- 172.21.15.81:0/829394192 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05440f6510 0x7f0544106730 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f0538013510 tx=0x7f053800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:30:59.603 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:59.597+0000 7f0535ffb700 1 -- 172.21.15.81:0/829394192 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0538009320 con 0x7f05440f6510 2023-11-15T06:30:59.603 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:59.597+0000 7f0535ffb700 1 -- 172.21.15.81:0/829394192 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0538013f00 con 0x7f05440f6510 2023-11-15T06:30:59.603 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:59.597+0000 7f054b8f6700 1 -- 172.21.15.81:0/829394192 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0544106e20 con 0x7f05440f6510 2023-11-15T06:30:59.604 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:59.597+0000 7f0535ffb700 1 -- 172.21.15.81:0/829394192 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0538014d90 con 0x7f05440f6510 2023-11-15T06:30:59.604 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:59.597+0000 7f054b8f6700 1 -- 172.21.15.81:0/829394192 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05441072b0 con 0x7f05440f6510 2023-11-15T06:30:59.605 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:59.598+0000 7f0535ffb700 1 -- 172.21.15.81:0/829394192 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0538014420 con 0x7f05440f6510 2023-11-15T06:30:59.606 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:59.598+0000 7f054b8f6700 1 -- 172.21.15.81:0/829394192 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0544045ad0 con 0x7f05440f6510 2023-11-15T06:30:59.606 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:59.598+0000 7f0535ffb700 1 --2- 172.21.15.81:0/829394192 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f053002ff60 0x7f0530032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:30:59.606 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:59.598+0000 7f0535ffb700 1 -- 172.21.15.81:0/829394192 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0538041000 con 0x7f05440f6510 2023-11-15T06:30:59.606 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:59.598+0000 7f0548e91700 1 --2- 172.21.15.81:0/829394192 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f053002ff60 0x7f0530032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:30:59.607 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:59.599+0000 7f0548e91700 1 --2- 172.21.15.81:0/829394192 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f053002ff60 0x7f0530032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f05400096a0 tx=0x7f0540006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:30:59.607 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:59.602+0000 7f0535ffb700 1 -- 172.21.15.81:0/829394192 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f05380095f0 con 0x7f05440f6510 2023-11-15T06:30:59.812 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:59.810+0000 7f054b8f6700 1 -- 172.21.15.81:0/829394192 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f05440fbf60 con 0x7f05440f6510 2023-11-15T06:30:59.812 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:59.810+0000 7f0535ffb700 1 -- 172.21.15.81:0/829394192 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f053801db70 con 0x7f05440f6510 2023-11-15T06:30:59.813 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:30:59.813 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:30:59.814 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:59.812+0000 7f054b8f6700 1 -- 172.21.15.81:0/829394192 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f053002ff60 msgr2=0x7f0530032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:59.814 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:59.812+0000 7f054b8f6700 1 --2- 172.21.15.81:0/829394192 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f053002ff60 0x7f0530032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f05400096a0 tx=0x7f0540006b40).stop 2023-11-15T06:30:59.815 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:59.812+0000 7f054b8f6700 1 -- 172.21.15.81:0/829394192 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05440f6510 msgr2=0x7f0544106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:30:59.815 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:59.812+0000 7f054b8f6700 1 --2- 172.21.15.81:0/829394192 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05440f6510 0x7f0544106730 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f0538013510 tx=0x7f053800b040).stop 2023-11-15T06:30:59.815 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:59.812+0000 7f054b8f6700 1 -- 172.21.15.81:0/829394192 shutdown_connections 2023-11-15T06:30:59.815 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:59.812+0000 7f054b8f6700 1 --2- 172.21.15.81:0/829394192 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f053002ff60 0x7f0530032420 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:59.815 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:59.812+0000 7f054b8f6700 1 --2- 172.21.15.81:0/829394192 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05440f6510 0x7f0544106730 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:30:59.816 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:59.812+0000 7f054b8f6700 1 -- 172.21.15.81:0/829394192 >> 172.21.15.81:0/829394192 conn(0x7f05440f1c70 msgr2=0x7f05440f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:30:59.816 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:59.812+0000 7f054b8f6700 1 -- 172.21.15.81:0/829394192 shutdown_connections 2023-11-15T06:30:59.816 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:30:59.812+0000 7f054b8f6700 1 -- 172.21.15.81:0/829394192 wait complete. 2023-11-15T06:30:59.816 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:31:00.086 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cephadm 2023-11-15T06:30:59.056442+0000 mgr.smithi062.rwurtk (mgr.14164) 154 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi062 ... 2023-11-15T06:31:00.086 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi062 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2023-11-15T06:31:00.086 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-15T06:31:00.086 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: stat: stderr time="2023-11-15T06:26:38Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T06:31:00.087 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: stat: stderr time="2023-11-15T06:27:15Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T06:31:00.087 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: stat: stderr time="2023-11-15T06:27:53Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T06:31:00.087 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-15T06:31:00.087 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi062 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2023-11-15T06:31:00.087 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-15T06:31:00.088 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: stat: stderr time="2023-11-15T06:29:06Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T06:31:00.088 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: stat: stderr time="2023-11-15T06:29:44Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T06:31:00.088 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: stat: stderr time="2023-11-15T06:30:21Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T06:31:00.088 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-15T06:31:00.088 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: Traceback (most recent call last): 2023-11-15T06:31:00.089 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: File "/var/lib/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-15T06:31:00.089 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: main() 2023-11-15T06:31:00.089 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: File "/var/lib/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-15T06:31:00.089 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: r = ctx.func(ctx) 2023-11-15T06:31:00.090 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: File "/var/lib/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-15T06:31:00.090 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: return func(ctx) 2023-11-15T06:31:00.090 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: File "/var/lib/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-11-15T06:31:00.090 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-11-15T06:31:00.090 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: File "/var/lib/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2023-11-15T06:31:00.091 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2023-11-15T06:31:00.091 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: File "/var/lib/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-11-15T06:31:00.091 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: raise RuntimeError('uid/gid not found') 2023-11-15T06:31:00.091 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: RuntimeError: uid/gid not found 2023-11-15T06:31:00.091 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: Traceback (most recent call last): 2023-11-15T06:31:00.091 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-11-15T06:31:00.092 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: yield (conn, connr) 2023-11-15T06:31:00.092 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-11-15T06:31:00.092 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: code, '\n'.join(err))) 2023-11-15T06:31:00.092 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi062 ... 2023-11-15T06:31:00.092 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi062 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2023-11-15T06:31:00.093 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-15T06:31:00.093 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: stat: stderr time="2023-11-15T06:26:38Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T06:31:00.093 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: stat: stderr time="2023-11-15T06:27:15Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T06:31:00.093 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: stat: stderr time="2023-11-15T06:27:53Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T06:31:00.093 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-15T06:31:00.094 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi062 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2023-11-15T06:31:00.094 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-15T06:31:00.094 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: stat: stderr time="2023-11-15T06:29:06Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T06:31:00.094 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: stat: stderr time="2023-11-15T06:29:44Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T06:31:00.094 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: stat: stderr time="2023-11-15T06:30:21Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T06:31:00.095 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-15T06:31:00.095 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: Traceback (most recent call last): 2023-11-15T06:31:00.095 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: File "/var/lib/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-15T06:31:00.095 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: main() 2023-11-15T06:31:00.095 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: File "/var/lib/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-15T06:31:00.096 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: r = ctx.func(ctx) 2023-11-15T06:31:00.096 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: File "/var/lib/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-15T06:31:00.096 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: return func(ctx) 2023-11-15T06:31:00.096 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: File "/var/lib/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-11-15T06:31:00.096 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-11-15T06:31:00.097 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: File "/var/lib/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2023-11-15T06:31:00.097 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2023-11-15T06:31:00.097 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: File "/var/lib/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-11-15T06:31:00.097 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: raise RuntimeError('uid/gid not found') 2023-11-15T06:31:00.097 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: RuntimeError: uid/gid not found 2023-11-15T06:31:00.097 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:30:59.057731+0000 mon.smithi062 (mon.0) 224 : audit [INF] from='mgr.14164 172.21.15.62:0/147479783' entity='mgr.smithi062.rwurtk' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi062", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-11-15T06:31:00.098 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:30:59.060456+0000 mon.smithi062 (mon.0) 225 : audit [INF] from='mgr.14164 172.21.15.62:0/147479783' entity='mgr.smithi062.rwurtk' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi062", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2023-11-15T06:31:00.098 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:30:59.061269+0000 mon.smithi062 (mon.0) 226 : audit [DBG] from='mgr.14164 172.21.15.62:0/147479783' entity='mgr.smithi062.rwurtk' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-15T06:31:00.098 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cephadm 2023-11-15T06:30:59.062043+0000 mgr.smithi062.rwurtk (mgr.14164) 155 : cephadm [INF] Deploying daemon crash.smithi062 on smithi062 2023-11-15T06:31:00.098 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:30:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:30:59.811272+0000 mon.smithi062 (mon.0) 227 : audit [DBG] from='client.? 172.21.15.81:0/829394192' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:31:01.200 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:31:01.200 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:31:01.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:01 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:30:59.826766+0000 mgr.smithi062.rwurtk (mgr.14164) 156 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:31:01.919 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:01 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:31:00.580357+0000 mon.smithi062 (mon.0) 228 : audit [INF] from='mgr.14164 172.21.15.62:0/147479783' entity='mgr.smithi062.rwurtk' 2023-11-15T06:31:01.919 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:01 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:31:00.722645+0000 mon.smithi062 (mon.0) 229 : audit [INF] from='mgr.14164 172.21.15.62:0/147479783' entity='mgr.smithi062.rwurtk' 2023-11-15T06:31:01.919 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:01 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:31:00.725053+0000 mon.smithi062 (mon.0) 230 : audit [INF] from='mgr.14164 172.21.15.62:0/147479783' entity='mgr.smithi062.rwurtk' 2023-11-15T06:31:01.919 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:01 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:31:00.725531+0000 mon.smithi062 (mon.0) 231 : audit [INF] from='mgr.14164 172.21.15.62:0/147479783' entity='mgr.smithi062.rwurtk' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-11-15T06:31:01.919 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:01 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:31:00.728247+0000 mon.smithi062 (mon.0) 232 : audit [INF] from='mgr.14164 172.21.15.62:0/147479783' entity='mgr.smithi062.rwurtk' 2023-11-15T06:31:02.369 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:02.366+0000 7f7cba349700 1 -- 172.21.15.81:0/1389196154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cb40f5410 msgr2=0x7f7cb40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:02.370 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:02.366+0000 7f7cba349700 1 --2- 172.21.15.81:0/1389196154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cb40f5410 0x7f7cb40f5830 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f7ca4009fb0 tx=0x7f7ca4009b30).stop 2023-11-15T06:31:02.370 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:02.366+0000 7f7cba349700 1 -- 172.21.15.81:0/1389196154 shutdown_connections 2023-11-15T06:31:02.370 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:02.366+0000 7f7cba349700 1 --2- 172.21.15.81:0/1389196154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cb40f5410 0x7f7cb40f5830 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:02.370 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:02.366+0000 7f7cba349700 1 -- 172.21.15.81:0/1389196154 >> 172.21.15.81:0/1389196154 conn(0x7f7cb40f0b70 msgr2=0x7f7cb40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:31:02.370 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:02.366+0000 7f7cba349700 1 -- 172.21.15.81:0/1389196154 shutdown_connections 2023-11-15T06:31:02.371 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:02.366+0000 7f7cba349700 1 -- 172.21.15.81:0/1389196154 wait complete. 2023-11-15T06:31:02.371 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:02.367+0000 7f7cba349700 1 Processor -- start 2023-11-15T06:31:02.371 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:02.367+0000 7f7cba349700 1 -- start start 2023-11-15T06:31:02.371 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:02.367+0000 7f7cba349700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cb40f5410 0x7f7cb4106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:31:02.371 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:02.367+0000 7f7cba349700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb4106c10 con 0x7f7cb40f5410 2023-11-15T06:31:02.372 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:02.367+0000 7f7cb3fff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cb40f5410 0x7f7cb4106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:31:02.372 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:02.367+0000 7f7cb3fff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cb40f5410 0x7f7cb4106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:36832/0 (socket says 172.21.15.81:36832) 2023-11-15T06:31:02.372 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:02.367+0000 7f7cb3fff700 1 -- 172.21.15.81:0/1674319289 learned_addr learned my addr 172.21.15.81:0/1674319289 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:31:02.372 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:02.368+0000 7f7cb3fff700 1 -- 172.21.15.81:0/1674319289 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ca4005040 con 0x7f7cb40f5410 2023-11-15T06:31:02.372 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:02.368+0000 7f7cb3fff700 1 --2- 172.21.15.81:0/1674319289 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cb40f5410 0x7f7cb4106720 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f7ca4013510 tx=0x7f7ca400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:31:02.374 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:02.368+0000 7f7cb0ff9700 1 -- 172.21.15.81:0/1674319289 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ca4009320 con 0x7f7cb40f5410 2023-11-15T06:31:02.374 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:02.368+0000 7f7cba349700 1 -- 172.21.15.81:0/1674319289 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cb4106e10 con 0x7f7cb40f5410 2023-11-15T06:31:02.374 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:02.368+0000 7f7cb0ff9700 1 -- 172.21.15.81:0/1674319289 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7ca4013f00 con 0x7f7cb40f5410 2023-11-15T06:31:02.374 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:02.368+0000 7f7cb0ff9700 1 -- 172.21.15.81:0/1674319289 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ca4014d90 con 0x7f7cb40f5410 2023-11-15T06:31:02.374 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:02.368+0000 7f7cba349700 1 -- 172.21.15.81:0/1674319289 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cb41072a0 con 0x7f7cb40f5410 2023-11-15T06:31:02.375 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:02.369+0000 7f7cb0ff9700 1 -- 172.21.15.81:0/1674319289 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7ca40095f0 con 0x7f7cb40f5410 2023-11-15T06:31:02.375 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:02.369+0000 7f7cba349700 1 -- 172.21.15.81:0/1674319289 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cb4045ad0 con 0x7f7cb40f5410 2023-11-15T06:31:02.375 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:02.369+0000 7f7cb0ff9700 1 --2- 172.21.15.81:0/1674319289 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f7ca002ff60 0x7f7ca0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:31:02.375 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:02.369+0000 7f7cb0ff9700 1 -- 172.21.15.81:0/1674319289 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7ca4041090 con 0x7f7cb40f5410 2023-11-15T06:31:02.375 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:02.369+0000 7f7cb37fe700 1 --2- 172.21.15.81:0/1674319289 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f7ca002ff60 0x7f7ca0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:31:02.376 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:02.369+0000 7f7cb37fe700 1 --2- 172.21.15.81:0/1674319289 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f7ca002ff60 0x7f7ca0032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f7c9c0096a0 tx=0x7f7c9c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:31:02.376 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:02.374+0000 7f7cb0ff9700 1 -- 172.21.15.81:0/1674319289 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7ca4014420 con 0x7f7cb40f5410 2023-11-15T06:31:02.582 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:02.580+0000 7f7cba349700 1 -- 172.21.15.81:0/1674319289 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7cb4108150 con 0x7f7cb40f5410 2023-11-15T06:31:02.584 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:02.583+0000 7f7cb0ff9700 1 -- 172.21.15.81:0/1674319289 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7ca401d7f0 con 0x7f7cb40f5410 2023-11-15T06:31:02.585 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:31:02.585 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:31:02.586 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:02.584+0000 7f7cba349700 1 -- 172.21.15.81:0/1674319289 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f7ca002ff60 msgr2=0x7f7ca0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:02.586 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:02.584+0000 7f7cba349700 1 --2- 172.21.15.81:0/1674319289 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f7ca002ff60 0x7f7ca0032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f7c9c0096a0 tx=0x7f7c9c006b40).stop 2023-11-15T06:31:02.586 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:02.584+0000 7f7cba349700 1 -- 172.21.15.81:0/1674319289 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cb40f5410 msgr2=0x7f7cb4106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:02.587 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:02.584+0000 7f7cba349700 1 --2- 172.21.15.81:0/1674319289 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cb40f5410 0x7f7cb4106720 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f7ca4013510 tx=0x7f7ca400b040).stop 2023-11-15T06:31:02.587 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:02.584+0000 7f7cba349700 1 -- 172.21.15.81:0/1674319289 shutdown_connections 2023-11-15T06:31:02.587 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:02.584+0000 7f7cba349700 1 --2- 172.21.15.81:0/1674319289 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f7ca002ff60 0x7f7ca0032420 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:02.587 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:02.584+0000 7f7cba349700 1 --2- 172.21.15.81:0/1674319289 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cb40f5410 0x7f7cb4106720 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:02.587 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:02.584+0000 7f7cba349700 1 -- 172.21.15.81:0/1674319289 >> 172.21.15.81:0/1674319289 conn(0x7f7cb40f0b70 msgr2=0x7f7cb405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:31:02.588 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:02.584+0000 7f7cba349700 1 -- 172.21.15.81:0/1674319289 shutdown_connections 2023-11-15T06:31:02.588 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:02.584+0000 7f7cba349700 1 -- 172.21.15.81:0/1674319289 wait complete. 2023-11-15T06:31:02.588 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:31:02.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:31:00.725988+0000 mgr.smithi062.rwurtk (mgr.14164) 157 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-11-15T06:31:02.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cephadm 2023-11-15T06:31:00.731671+0000 mgr.smithi062.rwurtk (mgr.14164) 158 : cephadm [INF] Deploying daemon grafana.smithi062 on smithi062 2023-11-15T06:31:03.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:03 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:31:01.827017+0000 mgr.smithi062.rwurtk (mgr.14164) 159 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:31:03.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:03 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:31:02.583269+0000 mon.smithi062 (mon.0) 233 : audit [DBG] from='client.? 172.21.15.81:0/1674319289' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:31:03.949 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:31:03.949 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:31:05.175 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:05.172+0000 7f6f7b59e700 1 -- 172.21.15.81:0/3071818212 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f7c0f62a0 msgr2=0x7f6f7c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:05.176 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:05.172+0000 7f6f7b59e700 1 --2- 172.21.15.81:0/3071818212 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f7c0f62a0 0x7f6f7c0f66c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f6f70009fb0 tx=0x7f6f70009b30).stop 2023-11-15T06:31:05.176 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:05.172+0000 7f6f7b59e700 1 -- 172.21.15.81:0/3071818212 shutdown_connections 2023-11-15T06:31:05.177 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:05.172+0000 7f6f7b59e700 1 --2- 172.21.15.81:0/3071818212 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f7c0f62a0 0x7f6f7c0f66c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:05.177 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:05.172+0000 7f6f7b59e700 1 -- 172.21.15.81:0/3071818212 >> 172.21.15.81:0/3071818212 conn(0x7f6f7c0f1aa0 msgr2=0x7f6f7c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:31:05.177 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:05.172+0000 7f6f7b59e700 1 -- 172.21.15.81:0/3071818212 shutdown_connections 2023-11-15T06:31:05.177 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:05.172+0000 7f6f7b59e700 1 -- 172.21.15.81:0/3071818212 wait complete. 2023-11-15T06:31:05.177 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:05.172+0000 7f6f7b59e700 1 Processor -- start 2023-11-15T06:31:05.178 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:05.172+0000 7f6f7b59e700 1 -- start start 2023-11-15T06:31:05.178 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:05.172+0000 7f6f7b59e700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f7c0f62a0 0x7f6f7c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:31:05.178 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:05.172+0000 7f6f7b59e700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7c069e20 con 0x7f6f7c0f62a0 2023-11-15T06:31:05.179 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:05.173+0000 7f6f7a59c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f7c0f62a0 0x7f6f7c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:31:05.179 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:05.173+0000 7f6f7a59c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f7c0f62a0 0x7f6f7c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:36844/0 (socket says 172.21.15.81:36844) 2023-11-15T06:31:05.179 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:05.173+0000 7f6f7a59c700 1 -- 172.21.15.81:0/3202268227 learned_addr learned my addr 172.21.15.81:0/3202268227 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:31:05.179 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:05.173+0000 7f6f7a59c700 1 -- 172.21.15.81:0/3202268227 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f70005040 con 0x7f6f7c0f62a0 2023-11-15T06:31:05.180 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:05.173+0000 7f6f7a59c700 1 --2- 172.21.15.81:0/3202268227 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f7c0f62a0 0x7f6f7c06b9a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f6f70013510 tx=0x7f6f7000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:31:05.180 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:05.173+0000 7f6f6affd700 1 -- 172.21.15.81:0/3202268227 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f70009320 con 0x7f6f7c0f62a0 2023-11-15T06:31:05.180 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:05.173+0000 7f6f7b59e700 1 -- 172.21.15.81:0/3202268227 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f7c069fc0 con 0x7f6f7c0f62a0 2023-11-15T06:31:05.181 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:05.173+0000 7f6f6affd700 1 -- 172.21.15.81:0/3202268227 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6f70013f00 con 0x7f6f7c0f62a0 2023-11-15T06:31:05.181 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:05.173+0000 7f6f6affd700 1 -- 172.21.15.81:0/3202268227 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f70014dc0 con 0x7f6f7c0f62a0 2023-11-15T06:31:05.181 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:05.174+0000 7f6f7b59e700 1 -- 172.21.15.81:0/3202268227 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f7c06a450 con 0x7f6f7c0f62a0 2023-11-15T06:31:05.183 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:05.174+0000 7f6f7b59e700 1 -- 172.21.15.81:0/3202268227 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f7c059100 con 0x7f6f7c0f62a0 2023-11-15T06:31:05.183 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:05.174+0000 7f6f6affd700 1 -- 172.21.15.81:0/3202268227 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6f700095f0 con 0x7f6f7c0f62a0 2023-11-15T06:31:05.184 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:05.174+0000 7f6f6affd700 1 --2- 172.21.15.81:0/3202268227 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f6f6402ff60 0x7f6f64032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:31:05.184 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:05.175+0000 7f6f6affd700 1 -- 172.21.15.81:0/3202268227 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6f70041030 con 0x7f6f7c0f62a0 2023-11-15T06:31:05.184 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:05.175+0000 7f6f79d9b700 1 --2- 172.21.15.81:0/3202268227 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f6f6402ff60 0x7f6f64032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:31:05.185 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:05.175+0000 7f6f79d9b700 1 --2- 172.21.15.81:0/3202268227 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f6f6402ff60 0x7f6f64032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f6f6c0096a0 tx=0x7f6f6c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:31:05.185 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:05.179+0000 7f6f6affd700 1 -- 172.21.15.81:0/3202268227 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6f70014420 con 0x7f6f7c0f62a0 2023-11-15T06:31:05.385 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:05.383+0000 7f6f7b59e700 1 -- 172.21.15.81:0/3202268227 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6f7c0fbce0 con 0x7f6f7c0f62a0 2023-11-15T06:31:05.386 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:05.384+0000 7f6f6affd700 1 -- 172.21.15.81:0/3202268227 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6f70010230 con 0x7f6f7c0f62a0 2023-11-15T06:31:05.386 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:31:05.387 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:31:05.388 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:05.386+0000 7f6f7b59e700 1 -- 172.21.15.81:0/3202268227 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f6f6402ff60 msgr2=0x7f6f64032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:05.388 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:05.386+0000 7f6f7b59e700 1 --2- 172.21.15.81:0/3202268227 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f6f6402ff60 0x7f6f64032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f6f6c0096a0 tx=0x7f6f6c006b40).stop 2023-11-15T06:31:05.388 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:05.386+0000 7f6f7b59e700 1 -- 172.21.15.81:0/3202268227 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f7c0f62a0 msgr2=0x7f6f7c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:05.388 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:05.386+0000 7f6f7b59e700 1 --2- 172.21.15.81:0/3202268227 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f7c0f62a0 0x7f6f7c06b9a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f6f70013510 tx=0x7f6f7000b040).stop 2023-11-15T06:31:05.388 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:05.386+0000 7f6f7b59e700 1 -- 172.21.15.81:0/3202268227 shutdown_connections 2023-11-15T06:31:05.389 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:05.386+0000 7f6f7b59e700 1 --2- 172.21.15.81:0/3202268227 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f6f6402ff60 0x7f6f64032420 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:05.389 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:05.386+0000 7f6f7b59e700 1 --2- 172.21.15.81:0/3202268227 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f7c0f62a0 0x7f6f7c06b9a0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:05.389 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:05.386+0000 7f6f7b59e700 1 -- 172.21.15.81:0/3202268227 >> 172.21.15.81:0/3202268227 conn(0x7f6f7c0f1aa0 msgr2=0x7f6f7c0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:31:05.389 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:05.386+0000 7f6f7b59e700 1 -- 172.21.15.81:0/3202268227 shutdown_connections 2023-11-15T06:31:05.389 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:05.386+0000 7f6f7b59e700 1 -- 172.21.15.81:0/3202268227 wait complete. 2023-11-15T06:31:05.390 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:31:05.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:05 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:31:03.827360+0000 mgr.smithi062.rwurtk (mgr.14164) 160 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:31:05.919 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:05 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:31:05.384931+0000 mon.smithi062 (mon.0) 234 : audit [DBG] from='client.? 172.21.15.81:0/3202268227' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:31:06.826 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:31:07.196 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:31:07.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:07 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:31:05.827670+0000 mgr.smithi062.rwurtk (mgr.14164) 161 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:31:08.392 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:08.390+0000 7fc5fd8bb700 1 -- 172.21.15.81:0/3361979812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5f80f5410 msgr2=0x7fc5f80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:08.392 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:08.390+0000 7fc5fd8bb700 1 --2- 172.21.15.81:0/3361979812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5f80f5410 0x7fc5f80f5830 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7fc5ec009fb0 tx=0x7fc5ec009b30).stop 2023-11-15T06:31:08.392 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:08.390+0000 7fc5fd8bb700 1 -- 172.21.15.81:0/3361979812 shutdown_connections 2023-11-15T06:31:08.393 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:08.390+0000 7fc5fd8bb700 1 --2- 172.21.15.81:0/3361979812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5f80f5410 0x7fc5f80f5830 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:08.393 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:08.390+0000 7fc5fd8bb700 1 -- 172.21.15.81:0/3361979812 >> 172.21.15.81:0/3361979812 conn(0x7fc5f80f0b70 msgr2=0x7fc5f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:31:08.393 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:08.390+0000 7fc5fd8bb700 1 -- 172.21.15.81:0/3361979812 shutdown_connections 2023-11-15T06:31:08.393 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:08.390+0000 7fc5fd8bb700 1 -- 172.21.15.81:0/3361979812 wait complete. 2023-11-15T06:31:08.395 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:08.390+0000 7fc5fd8bb700 1 Processor -- start 2023-11-15T06:31:08.395 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:08.390+0000 7fc5fd8bb700 1 -- start start 2023-11-15T06:31:08.396 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:08.391+0000 7fc5fd8bb700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5f80f5410 0x7fc5f8105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:31:08.396 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:08.391+0000 7fc5fd8bb700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f8105b20 con 0x7fc5f80f5410 2023-11-15T06:31:08.396 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:08.391+0000 7fc5f6ffd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5f80f5410 0x7fc5f8105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:31:08.396 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:08.391+0000 7fc5f6ffd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5f80f5410 0x7fc5f8105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:36858/0 (socket says 172.21.15.81:36858) 2023-11-15T06:31:08.397 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:08.391+0000 7fc5f6ffd700 1 -- 172.21.15.81:0/3801518472 learned_addr learned my addr 172.21.15.81:0/3801518472 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:31:08.397 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:08.391+0000 7fc5f6ffd700 1 -- 172.21.15.81:0/3801518472 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5ec005040 con 0x7fc5f80f5410 2023-11-15T06:31:08.397 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:08.391+0000 7fc5f6ffd700 1 --2- 172.21.15.81:0/3801518472 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5f80f5410 0x7fc5f8105630 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7fc5ec013510 tx=0x7fc5ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:31:08.397 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:08.391+0000 7fc5df7fe700 1 -- 172.21.15.81:0/3801518472 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5ec009320 con 0x7fc5f80f5410 2023-11-15T06:31:08.397 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:08.391+0000 7fc5df7fe700 1 -- 172.21.15.81:0/3801518472 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc5ec013f00 con 0x7fc5f80f5410 2023-11-15T06:31:08.397 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:08.391+0000 7fc5fd8bb700 1 -- 172.21.15.81:0/3801518472 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5f8105d20 con 0x7fc5f80f5410 2023-11-15T06:31:08.398 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:08.391+0000 7fc5df7fe700 1 -- 172.21.15.81:0/3801518472 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5ec014d90 con 0x7fc5f80f5410 2023-11-15T06:31:08.399 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:08.391+0000 7fc5fd8bb700 1 -- 172.21.15.81:0/3801518472 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5f81061b0 con 0x7fc5f80f5410 2023-11-15T06:31:08.399 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:08.392+0000 7fc5df7fe700 1 -- 172.21.15.81:0/3801518472 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc5ec0263a0 con 0x7fc5f80f5410 2023-11-15T06:31:08.399 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:08.392+0000 7fc5df7fe700 1 --2- 172.21.15.81:0/3801518472 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fc5e002ffb0 0x7fc5e0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:31:08.399 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:08.392+0000 7fc5df7fe700 1 -- 172.21.15.81:0/3801518472 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc5ec0412b0 con 0x7fc5f80f5410 2023-11-15T06:31:08.400 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:08.392+0000 7fc5fd8bb700 1 -- 172.21.15.81:0/3801518472 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5f8045ad0 con 0x7fc5f80f5410 2023-11-15T06:31:08.400 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:08.393+0000 7fc5f67fc700 1 --2- 172.21.15.81:0/3801518472 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fc5e002ffb0 0x7fc5e0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:31:08.400 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:08.393+0000 7fc5f67fc700 1 --2- 172.21.15.81:0/3801518472 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fc5e002ffb0 0x7fc5e0032470 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fc5e80096a0 tx=0x7fc5e8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:31:08.401 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:08.397+0000 7fc5df7fe700 1 -- 172.21.15.81:0/3801518472 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc5ec01d380 con 0x7fc5f80f5410 2023-11-15T06:31:08.604 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:08.602+0000 7fc5fd8bb700 1 -- 172.21.15.81:0/3801518472 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc5f8106ed0 con 0x7fc5f80f5410 2023-11-15T06:31:08.605 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:08.603+0000 7fc5df7fe700 1 -- 172.21.15.81:0/3801518472 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc5ec02b030 con 0x7fc5f80f5410 2023-11-15T06:31:08.605 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:31:08.606 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:31:08.606 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:08.604+0000 7fc5fd8bb700 1 -- 172.21.15.81:0/3801518472 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fc5e002ffb0 msgr2=0x7fc5e0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:08.607 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:08.605+0000 7fc5fd8bb700 1 --2- 172.21.15.81:0/3801518472 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fc5e002ffb0 0x7fc5e0032470 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fc5e80096a0 tx=0x7fc5e8006b40).stop 2023-11-15T06:31:08.607 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:08.605+0000 7fc5fd8bb700 1 -- 172.21.15.81:0/3801518472 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5f80f5410 msgr2=0x7fc5f8105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:08.607 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:08.605+0000 7fc5fd8bb700 1 --2- 172.21.15.81:0/3801518472 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5f80f5410 0x7fc5f8105630 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7fc5ec013510 tx=0x7fc5ec00b040).stop 2023-11-15T06:31:08.607 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:08.605+0000 7fc5fd8bb700 1 -- 172.21.15.81:0/3801518472 shutdown_connections 2023-11-15T06:31:08.608 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:08.605+0000 7fc5fd8bb700 1 --2- 172.21.15.81:0/3801518472 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fc5e002ffb0 0x7fc5e0032470 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:08.608 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:08.605+0000 7fc5fd8bb700 1 --2- 172.21.15.81:0/3801518472 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5f80f5410 0x7fc5f8105630 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:08.608 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:08.605+0000 7fc5fd8bb700 1 -- 172.21.15.81:0/3801518472 >> 172.21.15.81:0/3801518472 conn(0x7fc5f80f0b70 msgr2=0x7fc5f80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:31:08.608 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:08.605+0000 7fc5fd8bb700 1 -- 172.21.15.81:0/3801518472 shutdown_connections 2023-11-15T06:31:08.608 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:08.605+0000 7fc5fd8bb700 1 -- 172.21.15.81:0/3801518472 wait complete. 2023-11-15T06:31:08.609 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:31:08.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:08 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:31:07.827888+0000 mgr.smithi062.rwurtk (mgr.14164) 162 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:31:08.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:08 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:31:08.603764+0000 mon.smithi062 (mon.0) 235 : audit [DBG] from='client.? 172.21.15.81:0/3801518472' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:31:09.995 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:31:09.996 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:31:11.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:10 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:31:09.828108+0000 mgr.smithi062.rwurtk (mgr.14164) 163 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:31:11.202 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:11.200+0000 7f2fe5670700 1 -- 172.21.15.81:0/3280849383 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fe00f51a0 msgr2=0x7f2fe00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:11.203 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:11.200+0000 7f2fe5670700 1 --2- 172.21.15.81:0/3280849383 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fe00f51a0 0x7f2fe00f55c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f2fd0009fb0 tx=0x7f2fd0009b30).stop 2023-11-15T06:31:11.203 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:11.200+0000 7f2fe5670700 1 -- 172.21.15.81:0/3280849383 shutdown_connections 2023-11-15T06:31:11.203 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:11.200+0000 7f2fe5670700 1 --2- 172.21.15.81:0/3280849383 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fe00f51a0 0x7f2fe00f55c0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:11.204 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:11.200+0000 7f2fe5670700 1 -- 172.21.15.81:0/3280849383 >> 172.21.15.81:0/3280849383 conn(0x7f2fe00f09a0 msgr2=0x7f2fe00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:31:11.204 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:11.200+0000 7f2fe5670700 1 -- 172.21.15.81:0/3280849383 shutdown_connections 2023-11-15T06:31:11.204 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:11.200+0000 7f2fe5670700 1 -- 172.21.15.81:0/3280849383 wait complete. 2023-11-15T06:31:11.204 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:11.201+0000 7f2fe5670700 1 Processor -- start 2023-11-15T06:31:11.206 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:11.201+0000 7f2fe5670700 1 -- start start 2023-11-15T06:31:11.206 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:11.201+0000 7f2fe5670700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fe00f51a0 0x7f2fe01053d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:31:11.206 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:11.201+0000 7f2fe5670700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe01058c0 con 0x7f2fe00f51a0 2023-11-15T06:31:11.207 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:11.201+0000 7f2fdffff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fe00f51a0 0x7f2fe01053d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:31:11.207 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:11.201+0000 7f2fdffff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fe00f51a0 0x7f2fe01053d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:36866/0 (socket says 172.21.15.81:36866) 2023-11-15T06:31:11.207 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:11.201+0000 7f2fdffff700 1 -- 172.21.15.81:0/1248673778 learned_addr learned my addr 172.21.15.81:0/1248673778 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:31:11.207 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:11.202+0000 7f2fdffff700 1 -- 172.21.15.81:0/1248673778 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd0005040 con 0x7f2fe00f51a0 2023-11-15T06:31:11.207 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:11.202+0000 7f2fdffff700 1 --2- 172.21.15.81:0/1248673778 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fe00f51a0 0x7f2fe01053d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f2fd0013510 tx=0x7f2fd000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:31:11.208 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:11.202+0000 7f2fdcff9700 1 -- 172.21.15.81:0/1248673778 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fd0009320 con 0x7f2fe00f51a0 2023-11-15T06:31:11.208 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:11.202+0000 7f2fdcff9700 1 -- 172.21.15.81:0/1248673778 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2fd0013f00 con 0x7f2fe00f51a0 2023-11-15T06:31:11.208 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:11.202+0000 7f2fe5670700 1 -- 172.21.15.81:0/1248673778 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fe0105ac0 con 0x7f2fe00f51a0 2023-11-15T06:31:11.208 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:11.202+0000 7f2fdcff9700 1 -- 172.21.15.81:0/1248673778 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fd0014dc0 con 0x7f2fe00f51a0 2023-11-15T06:31:11.209 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:11.202+0000 7f2fe5670700 1 -- 172.21.15.81:0/1248673778 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fe0105f50 con 0x7f2fe00f51a0 2023-11-15T06:31:11.209 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:11.203+0000 7f2fdcff9700 1 -- 172.21.15.81:0/1248673778 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2fd001d3d0 con 0x7f2fe00f51a0 2023-11-15T06:31:11.209 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:11.203+0000 7f2fdcff9700 1 --2- 172.21.15.81:0/1248673778 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f2fc802ff60 0x7f2fc8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:31:11.209 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:11.203+0000 7f2fdcff9700 1 -- 172.21.15.81:0/1248673778 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2fd00095f0 con 0x7f2fe00f51a0 2023-11-15T06:31:11.209 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:11.203+0000 7f2fdf7fe700 1 --2- 172.21.15.81:0/1248673778 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f2fc802ff60 0x7f2fc8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:31:11.210 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:11.203+0000 7f2fe5670700 1 -- 172.21.15.81:0/1248673778 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fe0059100 con 0x7f2fe00f51a0 2023-11-15T06:31:11.210 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:11.203+0000 7f2fdf7fe700 1 --2- 172.21.15.81:0/1248673778 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f2fc802ff60 0x7f2fc8032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f2fd40096a0 tx=0x7f2fd4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:31:11.210 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:11.208+0000 7f2fdcff9700 1 -- 172.21.15.81:0/1248673778 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2fd0026490 con 0x7f2fe00f51a0 2023-11-15T06:31:11.411 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:11.409+0000 7f2fe5670700 1 -- 172.21.15.81:0/1248673778 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2fe00fabe0 con 0x7f2fe00f51a0 2023-11-15T06:31:11.412 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:11.410+0000 7f2fdcff9700 1 -- 172.21.15.81:0/1248673778 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2fd00126f0 con 0x7f2fe00f51a0 2023-11-15T06:31:11.412 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:31:11.412 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:31:11.413 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:11.411+0000 7f2fe5670700 1 -- 172.21.15.81:0/1248673778 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f2fc802ff60 msgr2=0x7f2fc8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:11.414 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:11.411+0000 7f2fe5670700 1 --2- 172.21.15.81:0/1248673778 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f2fc802ff60 0x7f2fc8032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f2fd40096a0 tx=0x7f2fd4006b40).stop 2023-11-15T06:31:11.414 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:11.411+0000 7f2fe5670700 1 -- 172.21.15.81:0/1248673778 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fe00f51a0 msgr2=0x7f2fe01053d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:11.414 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:11.411+0000 7f2fe5670700 1 --2- 172.21.15.81:0/1248673778 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fe00f51a0 0x7f2fe01053d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f2fd0013510 tx=0x7f2fd000b040).stop 2023-11-15T06:31:11.414 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:11.411+0000 7f2fe5670700 1 -- 172.21.15.81:0/1248673778 shutdown_connections 2023-11-15T06:31:11.414 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:11.411+0000 7f2fe5670700 1 --2- 172.21.15.81:0/1248673778 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f2fc802ff60 0x7f2fc8032420 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:11.415 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:11.411+0000 7f2fe5670700 1 --2- 172.21.15.81:0/1248673778 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fe00f51a0 0x7f2fe01053d0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:11.415 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:11.411+0000 7f2fe5670700 1 -- 172.21.15.81:0/1248673778 >> 172.21.15.81:0/1248673778 conn(0x7f2fe00f09a0 msgr2=0x7f2fe00f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:31:11.415 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:11.411+0000 7f2fe5670700 1 -- 172.21.15.81:0/1248673778 shutdown_connections 2023-11-15T06:31:11.415 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:11.411+0000 7f2fe5670700 1 -- 172.21.15.81:0/1248673778 wait complete. 2023-11-15T06:31:11.416 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:31:12.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:11 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:31:11.410450+0000 mon.smithi062 (mon.0) 236 : audit [DBG] from='client.? 172.21.15.81:0/1248673778' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:31:12.847 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:31:12.847 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:31:13.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:12 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:31:11.828338+0000 mgr.smithi062.rwurtk (mgr.14164) 164 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:31:14.090 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:14.087+0000 7fb54695c700 1 -- 172.21.15.81:0/3802296097 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb540060200 msgr2=0x7fb5400fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:14.090 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:14.087+0000 7fb54695c700 1 --2- 172.21.15.81:0/3802296097 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb540060200 0x7fb5400fb570 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7fb528009fb0 tx=0x7fb528009b30).stop 2023-11-15T06:31:14.090 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:14.087+0000 7fb54695c700 1 -- 172.21.15.81:0/3802296097 shutdown_connections 2023-11-15T06:31:14.090 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:14.087+0000 7fb54695c700 1 --2- 172.21.15.81:0/3802296097 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb540060200 0x7fb5400fb570 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:14.091 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:14.087+0000 7fb54695c700 1 -- 172.21.15.81:0/3802296097 >> 172.21.15.81:0/3802296097 conn(0x7fb5400f0b70 msgr2=0x7fb5400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:31:14.091 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:14.087+0000 7fb54695c700 1 -- 172.21.15.81:0/3802296097 shutdown_connections 2023-11-15T06:31:14.091 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:14.087+0000 7fb54695c700 1 -- 172.21.15.81:0/3802296097 wait complete. 2023-11-15T06:31:14.091 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:14.088+0000 7fb54695c700 1 Processor -- start 2023-11-15T06:31:14.093 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:14.088+0000 7fb54695c700 1 -- start start 2023-11-15T06:31:14.093 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:14.088+0000 7fb54695c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb540060200 0x7fb5401088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:31:14.093 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:14.088+0000 7fb54695c700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb540108dc0 con 0x7fb540060200 2023-11-15T06:31:14.093 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:14.088+0000 7fb53ffff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb540060200 0x7fb5401088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:31:14.094 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:14.088+0000 7fb53ffff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb540060200 0x7fb5401088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:43378/0 (socket says 172.21.15.81:43378) 2023-11-15T06:31:14.094 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:14.088+0000 7fb53ffff700 1 -- 172.21.15.81:0/3634385749 learned_addr learned my addr 172.21.15.81:0/3634385749 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:31:14.094 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:14.089+0000 7fb53ffff700 1 -- 172.21.15.81:0/3634385749 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb528005040 con 0x7fb540060200 2023-11-15T06:31:14.094 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:14.089+0000 7fb53ffff700 1 --2- 172.21.15.81:0/3634385749 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb540060200 0x7fb5401088d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7fb528013510 tx=0x7fb52800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:31:14.095 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:14.089+0000 7fb53cff9700 1 -- 172.21.15.81:0/3634385749 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb528009320 con 0x7fb540060200 2023-11-15T06:31:14.095 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:14.089+0000 7fb54695c700 1 -- 172.21.15.81:0/3634385749 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb540108fc0 con 0x7fb540060200 2023-11-15T06:31:14.095 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:14.089+0000 7fb53cff9700 1 -- 172.21.15.81:0/3634385749 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb528013f00 con 0x7fb540060200 2023-11-15T06:31:14.095 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:14.089+0000 7fb53cff9700 1 -- 172.21.15.81:0/3634385749 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb528014dc0 con 0x7fb540060200 2023-11-15T06:31:14.095 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:14.089+0000 7fb54695c700 1 -- 172.21.15.81:0/3634385749 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb540109450 con 0x7fb540060200 2023-11-15T06:31:14.095 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:14.090+0000 7fb53cff9700 1 -- 172.21.15.81:0/3634385749 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb52801d3d0 con 0x7fb540060200 2023-11-15T06:31:14.096 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:14.090+0000 7fb54695c700 1 -- 172.21.15.81:0/3634385749 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb540045ad0 con 0x7fb540060200 2023-11-15T06:31:14.096 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:14.090+0000 7fb53cff9700 1 --2- 172.21.15.81:0/3634385749 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fb52c02ff60 0x7fb52c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:31:14.096 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:14.090+0000 7fb53cff9700 1 -- 172.21.15.81:0/3634385749 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb528041340 con 0x7fb540060200 2023-11-15T06:31:14.096 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:14.090+0000 7fb53f7fe700 1 --2- 172.21.15.81:0/3634385749 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fb52c02ff60 0x7fb52c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:31:14.096 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:14.091+0000 7fb53f7fe700 1 --2- 172.21.15.81:0/3634385749 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fb52c02ff60 0x7fb52c032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fb5300096a0 tx=0x7fb530006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:31:14.097 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:14.095+0000 7fb53cff9700 1 -- 172.21.15.81:0/3634385749 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb5280126f0 con 0x7fb540060200 2023-11-15T06:31:14.301 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:14.299+0000 7fb54695c700 1 -- 172.21.15.81:0/3634385749 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb54010a300 con 0x7fb540060200 2023-11-15T06:31:14.302 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:14.300+0000 7fb53cff9700 1 -- 172.21.15.81:0/3634385749 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb528012330 con 0x7fb540060200 2023-11-15T06:31:14.303 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:31:14.303 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:31:14.304 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:14.301+0000 7fb54695c700 1 -- 172.21.15.81:0/3634385749 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fb52c02ff60 msgr2=0x7fb52c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:14.305 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:14.301+0000 7fb54695c700 1 --2- 172.21.15.81:0/3634385749 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fb52c02ff60 0x7fb52c032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fb5300096a0 tx=0x7fb530006b40).stop 2023-11-15T06:31:14.305 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:14.301+0000 7fb54695c700 1 -- 172.21.15.81:0/3634385749 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb540060200 msgr2=0x7fb5401088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:14.305 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:14.301+0000 7fb54695c700 1 --2- 172.21.15.81:0/3634385749 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb540060200 0x7fb5401088d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7fb528013510 tx=0x7fb52800b040).stop 2023-11-15T06:31:14.305 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:14.301+0000 7fb54695c700 1 -- 172.21.15.81:0/3634385749 shutdown_connections 2023-11-15T06:31:14.306 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:14.301+0000 7fb54695c700 1 --2- 172.21.15.81:0/3634385749 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fb52c02ff60 0x7fb52c032420 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:14.306 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:14.301+0000 7fb54695c700 1 --2- 172.21.15.81:0/3634385749 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb540060200 0x7fb5401088d0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:14.306 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:14.301+0000 7fb54695c700 1 -- 172.21.15.81:0/3634385749 >> 172.21.15.81:0/3634385749 conn(0x7fb5400f0b70 msgr2=0x7fb5400f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:31:14.307 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:14.302+0000 7fb54695c700 1 -- 172.21.15.81:0/3634385749 shutdown_connections 2023-11-15T06:31:14.307 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:14.302+0000 7fb54695c700 1 -- 172.21.15.81:0/3634385749 wait complete. 2023-11-15T06:31:14.307 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:31:15.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:14 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:31:13.828597+0000 mgr.smithi062.rwurtk (mgr.14164) 165 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:31:15.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:14 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:31:14.300798+0000 mon.smithi062 (mon.0) 237 : audit [DBG] from='client.? 172.21.15.81:0/3634385749' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:31:15.682 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:31:15.682 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:31:16.851 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:16.847+0000 7f374b703700 1 -- 172.21.15.81:0/1047165717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37440f5410 msgr2=0x7f37440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:16.851 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:16.847+0000 7f374b703700 1 --2- 172.21.15.81:0/1047165717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37440f5410 0x7f37440f5830 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f3734009fb0 tx=0x7f3734009b30).stop 2023-11-15T06:31:16.851 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:16.847+0000 7f374b703700 1 -- 172.21.15.81:0/1047165717 shutdown_connections 2023-11-15T06:31:16.852 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:16.847+0000 7f374b703700 1 --2- 172.21.15.81:0/1047165717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37440f5410 0x7f37440f5830 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:16.852 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:16.847+0000 7f374b703700 1 -- 172.21.15.81:0/1047165717 >> 172.21.15.81:0/1047165717 conn(0x7f37440f0b70 msgr2=0x7f37440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:31:16.852 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:16.847+0000 7f374b703700 1 -- 172.21.15.81:0/1047165717 shutdown_connections 2023-11-15T06:31:16.852 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:16.847+0000 7f374b703700 1 -- 172.21.15.81:0/1047165717 wait complete. 2023-11-15T06:31:16.852 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:16.848+0000 7f374b703700 1 Processor -- start 2023-11-15T06:31:16.853 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:16.848+0000 7f374b703700 1 -- start start 2023-11-15T06:31:16.853 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:16.848+0000 7f374b703700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37440f5410 0x7f3744108920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:31:16.853 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:16.848+0000 7f374b703700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3744108e10 con 0x7f37440f5410 2023-11-15T06:31:16.853 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:16.848+0000 7f374949f700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37440f5410 0x7f3744108920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:31:16.853 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:16.848+0000 7f374949f700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37440f5410 0x7f3744108920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:43400/0 (socket says 172.21.15.81:43400) 2023-11-15T06:31:16.853 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:16.848+0000 7f374949f700 1 -- 172.21.15.81:0/1614352389 learned_addr learned my addr 172.21.15.81:0/1614352389 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:31:16.854 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:16.848+0000 7f374949f700 1 -- 172.21.15.81:0/1614352389 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3734005040 con 0x7f37440f5410 2023-11-15T06:31:16.854 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:16.849+0000 7f374949f700 1 --2- 172.21.15.81:0/1614352389 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37440f5410 0x7f3744108920 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f3734013510 tx=0x7f373400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:31:16.854 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:16.849+0000 7f3739ffb700 1 -- 172.21.15.81:0/1614352389 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3734009320 con 0x7f37440f5410 2023-11-15T06:31:16.854 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:16.849+0000 7f3739ffb700 1 -- 172.21.15.81:0/1614352389 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3734013f00 con 0x7f37440f5410 2023-11-15T06:31:16.854 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:16.849+0000 7f374b703700 1 -- 172.21.15.81:0/1614352389 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3744109010 con 0x7f37440f5410 2023-11-15T06:31:16.855 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:16.849+0000 7f3739ffb700 1 -- 172.21.15.81:0/1614352389 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3734014d90 con 0x7f37440f5410 2023-11-15T06:31:16.855 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:16.849+0000 7f374b703700 1 -- 172.21.15.81:0/1614352389 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37441094a0 con 0x7f37440f5410 2023-11-15T06:31:16.857 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:16.850+0000 7f3739ffb700 1 -- 172.21.15.81:0/1614352389 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f37340263a0 con 0x7f37440f5410 2023-11-15T06:31:16.857 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:16.850+0000 7f3739ffb700 1 --2- 172.21.15.81:0/1614352389 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f373002ff60 0x7f3730032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:31:16.857 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:16.850+0000 7f3739ffb700 1 -- 172.21.15.81:0/1614352389 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3734041330 con 0x7f37440f5410 2023-11-15T06:31:16.857 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:16.850+0000 7f374b703700 1 -- 172.21.15.81:0/1614352389 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3744045ad0 con 0x7f37440f5410 2023-11-15T06:31:16.857 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:16.850+0000 7f3748c9e700 1 --2- 172.21.15.81:0/1614352389 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f373002ff60 0x7f3730032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:31:16.858 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:16.850+0000 7f3748c9e700 1 --2- 172.21.15.81:0/1614352389 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f373002ff60 0x7f3730032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f37400096a0 tx=0x7f3740006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:31:16.858 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:16.854+0000 7f3739ffb700 1 -- 172.21.15.81:0/1614352389 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3734012510 con 0x7f37440f5410 2023-11-15T06:31:17.062 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:17.059+0000 7f374b703700 1 -- 172.21.15.81:0/1614352389 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f374410a260 con 0x7f37440f5410 2023-11-15T06:31:17.062 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:17.060+0000 7f3739ffb700 1 -- 172.21.15.81:0/1614352389 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f373402b030 con 0x7f37440f5410 2023-11-15T06:31:17.063 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:31:17.063 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:31:17.064 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:17.062+0000 7f374b703700 1 -- 172.21.15.81:0/1614352389 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f373002ff60 msgr2=0x7f3730032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:17.064 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:17.062+0000 7f374b703700 1 --2- 172.21.15.81:0/1614352389 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f373002ff60 0x7f3730032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f37400096a0 tx=0x7f3740006b40).stop 2023-11-15T06:31:17.064 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:17.062+0000 7f374b703700 1 -- 172.21.15.81:0/1614352389 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37440f5410 msgr2=0x7f3744108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:17.064 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:17.062+0000 7f374b703700 1 --2- 172.21.15.81:0/1614352389 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37440f5410 0x7f3744108920 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f3734013510 tx=0x7f373400b040).stop 2023-11-15T06:31:17.065 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:17.062+0000 7f374b703700 1 -- 172.21.15.81:0/1614352389 shutdown_connections 2023-11-15T06:31:17.065 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:17.062+0000 7f374b703700 1 --2- 172.21.15.81:0/1614352389 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f373002ff60 0x7f3730032420 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:17.065 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:17.062+0000 7f374b703700 1 --2- 172.21.15.81:0/1614352389 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37440f5410 0x7f3744108920 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:17.065 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:17.062+0000 7f374b703700 1 -- 172.21.15.81:0/1614352389 >> 172.21.15.81:0/1614352389 conn(0x7f37440f0b70 msgr2=0x7f374405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:31:17.065 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:17.062+0000 7f374b703700 1 -- 172.21.15.81:0/1614352389 shutdown_connections 2023-11-15T06:31:17.066 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:17.062+0000 7f374b703700 1 -- 172.21.15.81:0/1614352389 wait complete. 2023-11-15T06:31:17.067 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:31:17.142 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:16 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:31:15.828872+0000 mgr.smithi062.rwurtk (mgr.14164) 166 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:31:18.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:17 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:31:17.061115+0000 mon.smithi062 (mon.0) 238 : audit [DBG] from='client.? 172.21.15.81:0/1614352389' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:31:18.169 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:17 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:31:17.206017+0000 mon.smithi062 (mon.0) 239 : audit [INF] from='mgr.14164 172.21.15.62:0/147479783' entity='mgr.smithi062.rwurtk' 2023-11-15T06:31:18.169 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:17 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cephadm 2023-11-15T06:31:17.207528+0000 mgr.smithi062.rwurtk (mgr.14164) 167 : cephadm [INF] Deploying daemon node-exporter.smithi062 on smithi062 2023-11-15T06:31:18.450 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:31:18.450 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:31:19.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:31:17.829126+0000 mgr.smithi062.rwurtk (mgr.14164) 168 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:31:19.619 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:19.616+0000 7f6bef856700 1 -- 172.21.15.81:0/1130299582 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6be80f5410 msgr2=0x7f6be80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:19.619 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:19.616+0000 7f6bef856700 1 --2- 172.21.15.81:0/1130299582 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6be80f5410 0x7f6be80f5830 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f6bd8009fb0 tx=0x7f6bd8009b30).stop 2023-11-15T06:31:19.619 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:19.617+0000 7f6bef856700 1 -- 172.21.15.81:0/1130299582 shutdown_connections 2023-11-15T06:31:19.620 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:19.617+0000 7f6bef856700 1 --2- 172.21.15.81:0/1130299582 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6be80f5410 0x7f6be80f5830 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:19.620 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:19.617+0000 7f6bef856700 1 -- 172.21.15.81:0/1130299582 >> 172.21.15.81:0/1130299582 conn(0x7f6be80f0b70 msgr2=0x7f6be80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:31:19.620 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:19.617+0000 7f6bef856700 1 -- 172.21.15.81:0/1130299582 shutdown_connections 2023-11-15T06:31:19.620 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:19.617+0000 7f6bef856700 1 -- 172.21.15.81:0/1130299582 wait complete. 2023-11-15T06:31:19.622 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:19.617+0000 7f6bef856700 1 Processor -- start 2023-11-15T06:31:19.623 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:19.617+0000 7f6bef856700 1 -- start start 2023-11-15T06:31:19.623 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:19.617+0000 7f6bef856700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6be80f5410 0x7f6be8105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:31:19.623 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:19.617+0000 7f6bef856700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be8105af0 con 0x7f6be80f5410 2023-11-15T06:31:19.623 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:19.618+0000 7f6bed5f2700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6be80f5410 0x7f6be8105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:31:19.624 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:19.618+0000 7f6bed5f2700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6be80f5410 0x7f6be8105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:43422/0 (socket says 172.21.15.81:43422) 2023-11-15T06:31:19.624 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:19.618+0000 7f6bed5f2700 1 -- 172.21.15.81:0/693230775 learned_addr learned my addr 172.21.15.81:0/693230775 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:31:19.624 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:19.618+0000 7f6bed5f2700 1 -- 172.21.15.81:0/693230775 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd8005040 con 0x7f6be80f5410 2023-11-15T06:31:19.624 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:19.618+0000 7f6bed5f2700 1 --2- 172.21.15.81:0/693230775 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6be80f5410 0x7f6be8105600 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f6bd8014000 tx=0x7f6bd800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:31:19.624 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:19.618+0000 7f6bddffb700 1 -- 172.21.15.81:0/693230775 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bd80095f0 con 0x7f6be80f5410 2023-11-15T06:31:19.625 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:19.618+0000 7f6bef856700 1 -- 172.21.15.81:0/693230775 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6be8105cf0 con 0x7f6be80f5410 2023-11-15T06:31:19.625 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:19.618+0000 7f6bddffb700 1 -- 172.21.15.81:0/693230775 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6bd8007e70 con 0x7f6be80f5410 2023-11-15T06:31:19.625 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:19.618+0000 7f6bddffb700 1 -- 172.21.15.81:0/693230775 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bd8014ea0 con 0x7f6be80f5410 2023-11-15T06:31:19.626 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:19.619+0000 7f6bef856700 1 -- 172.21.15.81:0/693230775 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6be8106180 con 0x7f6be80f5410 2023-11-15T06:31:19.626 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:19.619+0000 7f6bddffb700 1 -- 172.21.15.81:0/693230775 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6bd8025070 con 0x7f6be80f5410 2023-11-15T06:31:19.626 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:19.619+0000 7f6bef856700 1 -- 172.21.15.81:0/693230775 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6be8045ad0 con 0x7f6be80f5410 2023-11-15T06:31:19.626 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:19.620+0000 7f6bddffb700 1 --2- 172.21.15.81:0/693230775 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f6bd402ff60 0x7f6bd4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:31:19.626 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:19.620+0000 7f6bddffb700 1 -- 172.21.15.81:0/693230775 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6bd8040970 con 0x7f6be80f5410 2023-11-15T06:31:19.627 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:19.620+0000 7f6becdf1700 1 --2- 172.21.15.81:0/693230775 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f6bd402ff60 0x7f6bd4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:31:19.627 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:19.620+0000 7f6becdf1700 1 --2- 172.21.15.81:0/693230775 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f6bd402ff60 0x7f6bd4032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f6be40096a0 tx=0x7f6be4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:31:19.627 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:19.625+0000 7f6bddffb700 1 -- 172.21.15.81:0/693230775 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6bd80079c0 con 0x7f6be80f5410 2023-11-15T06:31:19.833 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:19.831+0000 7f6bef856700 1 -- 172.21.15.81:0/693230775 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6be80fae60 con 0x7f6be80f5410 2023-11-15T06:31:19.833 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:19.831+0000 7f6bddffb700 1 -- 172.21.15.81:0/693230775 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6bd800f300 con 0x7f6be80f5410 2023-11-15T06:31:19.834 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:31:19.834 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:31:19.835 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:19.833+0000 7f6bef856700 1 -- 172.21.15.81:0/693230775 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f6bd402ff60 msgr2=0x7f6bd4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:19.835 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:19.833+0000 7f6bef856700 1 --2- 172.21.15.81:0/693230775 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f6bd402ff60 0x7f6bd4032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f6be40096a0 tx=0x7f6be4006b40).stop 2023-11-15T06:31:19.835 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:19.833+0000 7f6bef856700 1 -- 172.21.15.81:0/693230775 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6be80f5410 msgr2=0x7f6be8105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:19.835 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:19.833+0000 7f6bef856700 1 --2- 172.21.15.81:0/693230775 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6be80f5410 0x7f6be8105600 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f6bd8014000 tx=0x7f6bd800b040).stop 2023-11-15T06:31:19.836 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:19.833+0000 7f6bef856700 1 -- 172.21.15.81:0/693230775 shutdown_connections 2023-11-15T06:31:19.836 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:19.833+0000 7f6bef856700 1 --2- 172.21.15.81:0/693230775 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f6bd402ff60 0x7f6bd4032420 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:19.836 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:19.833+0000 7f6bef856700 1 --2- 172.21.15.81:0/693230775 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6be80f5410 0x7f6be8105600 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:19.836 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:19.833+0000 7f6bef856700 1 -- 172.21.15.81:0/693230775 >> 172.21.15.81:0/693230775 conn(0x7f6be80f0b70 msgr2=0x7f6be80f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:31:19.836 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:19.833+0000 7f6bef856700 1 -- 172.21.15.81:0/693230775 shutdown_connections 2023-11-15T06:31:19.837 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:19.833+0000 7f6bef856700 1 -- 172.21.15.81:0/693230775 wait complete. 2023-11-15T06:31:19.837 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:31:20.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:19 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:31:19.832180+0000 mon.smithi062 (mon.0) 240 : audit [DBG] from='client.? 172.21.15.81:0/693230775' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:31:21.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:20 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:31:19.829456+0000 mgr.smithi062.rwurtk (mgr.14164) 169 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:31:21.218 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:31:21.218 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:31:22.388 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:22.384+0000 7ff7795d4700 1 -- 172.21.15.81:0/360109513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7740f6510 msgr2=0x7ff7740f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:22.388 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:22.385+0000 7ff7795d4700 1 --2- 172.21.15.81:0/360109513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7740f6510 0x7ff7740f6930 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7ff764009fb0 tx=0x7ff764009b30).stop 2023-11-15T06:31:22.388 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:22.385+0000 7ff7795d4700 1 -- 172.21.15.81:0/360109513 shutdown_connections 2023-11-15T06:31:22.389 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:22.385+0000 7ff7795d4700 1 --2- 172.21.15.81:0/360109513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7740f6510 0x7ff7740f6930 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:22.389 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:22.385+0000 7ff7795d4700 1 -- 172.21.15.81:0/360109513 >> 172.21.15.81:0/360109513 conn(0x7ff7740f1c70 msgr2=0x7ff7740f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:31:22.389 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:22.385+0000 7ff7795d4700 1 -- 172.21.15.81:0/360109513 shutdown_connections 2023-11-15T06:31:22.389 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:22.385+0000 7ff7795d4700 1 -- 172.21.15.81:0/360109513 wait complete. 2023-11-15T06:31:22.389 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:22.385+0000 7ff7795d4700 1 Processor -- start 2023-11-15T06:31:22.390 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:22.385+0000 7ff7795d4700 1 -- start start 2023-11-15T06:31:22.390 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:22.385+0000 7ff7795d4700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7740f6510 0x7ff77406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:31:22.390 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:22.385+0000 7ff7795d4700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff774069e20 con 0x7ff7740f6510 2023-11-15T06:31:22.390 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:22.386+0000 7ff772ffd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7740f6510 0x7ff77406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:31:22.390 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:22.386+0000 7ff772ffd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7740f6510 0x7ff77406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:33348/0 (socket says 172.21.15.81:33348) 2023-11-15T06:31:22.391 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:22.386+0000 7ff772ffd700 1 -- 172.21.15.81:0/781306831 learned_addr learned my addr 172.21.15.81:0/781306831 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:31:22.391 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:22.386+0000 7ff772ffd700 1 -- 172.21.15.81:0/781306831 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff764005040 con 0x7ff7740f6510 2023-11-15T06:31:22.391 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:22.386+0000 7ff772ffd700 1 --2- 172.21.15.81:0/781306831 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7740f6510 0x7ff77406b9a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7ff764013510 tx=0x7ff76400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:31:22.392 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:22.386+0000 7ff75b7fe700 1 -- 172.21.15.81:0/781306831 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff764009320 con 0x7ff7740f6510 2023-11-15T06:31:22.392 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:22.386+0000 7ff75b7fe700 1 -- 172.21.15.81:0/781306831 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff764013f00 con 0x7ff7740f6510 2023-11-15T06:31:22.393 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:22.386+0000 7ff7795d4700 1 -- 172.21.15.81:0/781306831 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff774069fc0 con 0x7ff7740f6510 2023-11-15T06:31:22.393 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:22.386+0000 7ff75b7fe700 1 -- 172.21.15.81:0/781306831 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff764014d90 con 0x7ff7740f6510 2023-11-15T06:31:22.393 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:22.386+0000 7ff7795d4700 1 -- 172.21.15.81:0/781306831 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff77406a450 con 0x7ff7740f6510 2023-11-15T06:31:22.393 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:22.387+0000 7ff75b7fe700 1 -- 172.21.15.81:0/781306831 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff764014420 con 0x7ff7740f6510 2023-11-15T06:31:22.393 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:22.387+0000 7ff7795d4700 1 -- 172.21.15.81:0/781306831 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff774059100 con 0x7ff7740f6510 2023-11-15T06:31:22.394 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:22.387+0000 7ff75b7fe700 1 --2- 172.21.15.81:0/781306831 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff75c02ffb0 0x7ff75c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:31:22.394 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:22.387+0000 7ff75b7fe700 1 -- 172.21.15.81:0/781306831 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff764041070 con 0x7ff7740f6510 2023-11-15T06:31:22.394 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:22.387+0000 7ff7727fc700 1 --2- 172.21.15.81:0/781306831 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff75c02ffb0 0x7ff75c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:31:22.394 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:22.388+0000 7ff7727fc700 1 --2- 172.21.15.81:0/781306831 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff75c02ffb0 0x7ff75c032470 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7ff7680096a0 tx=0x7ff768006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:31:22.395 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:22.392+0000 7ff75b7fe700 1 -- 172.21.15.81:0/781306831 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff76406d2b0 con 0x7ff7740f6510 2023-11-15T06:31:22.603 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:22.600+0000 7ff7795d4700 1 -- 172.21.15.81:0/781306831 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff7740fbf60 con 0x7ff7740f6510 2023-11-15T06:31:22.604 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:22.600+0000 7ff75b7fe700 1 -- 172.21.15.81:0/781306831 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff7640095f0 con 0x7ff7740f6510 2023-11-15T06:31:22.605 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:31:22.606 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:31:22.608 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:22.602+0000 7ff7795d4700 1 -- 172.21.15.81:0/781306831 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff75c02ffb0 msgr2=0x7ff75c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:22.608 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:22.602+0000 7ff7795d4700 1 --2- 172.21.15.81:0/781306831 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff75c02ffb0 0x7ff75c032470 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7ff7680096a0 tx=0x7ff768006b40).stop 2023-11-15T06:31:22.608 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:22.602+0000 7ff7795d4700 1 -- 172.21.15.81:0/781306831 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7740f6510 msgr2=0x7ff77406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:22.609 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:22.602+0000 7ff7795d4700 1 --2- 172.21.15.81:0/781306831 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7740f6510 0x7ff77406b9a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7ff764013510 tx=0x7ff76400b040).stop 2023-11-15T06:31:22.609 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:22.602+0000 7ff7795d4700 1 -- 172.21.15.81:0/781306831 shutdown_connections 2023-11-15T06:31:22.609 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:22.602+0000 7ff7795d4700 1 --2- 172.21.15.81:0/781306831 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff75c02ffb0 0x7ff75c032470 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:22.609 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:22.602+0000 7ff7795d4700 1 --2- 172.21.15.81:0/781306831 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7740f6510 0x7ff77406b9a0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:22.609 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:22.602+0000 7ff7795d4700 1 -- 172.21.15.81:0/781306831 >> 172.21.15.81:0/781306831 conn(0x7ff7740f1c70 msgr2=0x7ff7740f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:31:22.609 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:22.602+0000 7ff7795d4700 1 -- 172.21.15.81:0/781306831 shutdown_connections 2023-11-15T06:31:22.610 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:22.602+0000 7ff7795d4700 1 -- 172.21.15.81:0/781306831 wait complete. 2023-11-15T06:31:22.610 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:31:23.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:22 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:31:21.829668+0000 mgr.smithi062.rwurtk (mgr.14164) 170 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:31:23.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:22 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:31:22.601448+0000 mon.smithi062 (mon.0) 241 : audit [DBG] from='client.? 172.21.15.81:0/781306831' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:31:23.974 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:31:23.975 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:31:25.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:24 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:31:23.829931+0000 mgr.smithi062.rwurtk (mgr.14164) 171 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:31:25.185 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:25.182+0000 7f2c35dbd700 1 -- 172.21.15.81:0/2009513505 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c30060220 msgr2=0x7f2c300f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:25.185 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:25.182+0000 7f2c35dbd700 1 --2- 172.21.15.81:0/2009513505 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c30060220 0x7f2c300f9150 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f2c24009fb0 tx=0x7f2c24009b30).stop 2023-11-15T06:31:25.186 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:25.182+0000 7f2c35dbd700 1 -- 172.21.15.81:0/2009513505 shutdown_connections 2023-11-15T06:31:25.186 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:25.182+0000 7f2c35dbd700 1 --2- 172.21.15.81:0/2009513505 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c30060220 0x7f2c300f9150 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:25.186 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:25.182+0000 7f2c35dbd700 1 -- 172.21.15.81:0/2009513505 >> 172.21.15.81:0/2009513505 conn(0x7f2c300f0b70 msgr2=0x7f2c300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:31:25.186 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:25.182+0000 7f2c35dbd700 1 -- 172.21.15.81:0/2009513505 shutdown_connections 2023-11-15T06:31:25.186 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:25.182+0000 7f2c35dbd700 1 -- 172.21.15.81:0/2009513505 wait complete. 2023-11-15T06:31:25.188 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:25.183+0000 7f2c35dbd700 1 Processor -- start 2023-11-15T06:31:25.189 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:25.183+0000 7f2c35dbd700 1 -- start start 2023-11-15T06:31:25.189 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:25.183+0000 7f2c35dbd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c30060220 0x7f2c301066b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:31:25.189 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:25.183+0000 7f2c35dbd700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c30106ba0 con 0x7f2c30060220 2023-11-15T06:31:25.189 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:25.183+0000 7f2c2f7fe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c30060220 0x7f2c301066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:31:25.190 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:25.183+0000 7f2c2f7fe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c30060220 0x7f2c301066b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:33358/0 (socket says 172.21.15.81:33358) 2023-11-15T06:31:25.191 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:25.183+0000 7f2c2f7fe700 1 -- 172.21.15.81:0/1910368965 learned_addr learned my addr 172.21.15.81:0/1910368965 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:31:25.191 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:25.184+0000 7f2c2f7fe700 1 -- 172.21.15.81:0/1910368965 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c24005040 con 0x7f2c30060220 2023-11-15T06:31:25.191 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:25.184+0000 7f2c2f7fe700 1 --2- 172.21.15.81:0/1910368965 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c30060220 0x7f2c301066b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f2c24013510 tx=0x7f2c2400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:31:25.191 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:25.184+0000 7f2c17fff700 1 -- 172.21.15.81:0/1910368965 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c24009320 con 0x7f2c30060220 2023-11-15T06:31:25.191 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:25.184+0000 7f2c17fff700 1 -- 172.21.15.81:0/1910368965 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2c24013f00 con 0x7f2c30060220 2023-11-15T06:31:25.192 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:25.184+0000 7f2c35dbd700 1 -- 172.21.15.81:0/1910368965 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c30106da0 con 0x7f2c30060220 2023-11-15T06:31:25.192 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:25.184+0000 7f2c17fff700 1 -- 172.21.15.81:0/1910368965 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c24014dc0 con 0x7f2c30060220 2023-11-15T06:31:25.192 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:25.184+0000 7f2c35dbd700 1 -- 172.21.15.81:0/1910368965 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c30107230 con 0x7f2c30060220 2023-11-15T06:31:25.192 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:25.185+0000 7f2c17fff700 1 -- 172.21.15.81:0/1910368965 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2c240095f0 con 0x7f2c30060220 2023-11-15T06:31:25.193 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:25.185+0000 7f2c35dbd700 1 -- 172.21.15.81:0/1910368965 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c30045ad0 con 0x7f2c30060220 2023-11-15T06:31:25.193 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:25.185+0000 7f2c17fff700 1 --2- 172.21.15.81:0/1910368965 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f2c1802ffb0 0x7f2c18032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:31:25.193 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:25.185+0000 7f2c17fff700 1 -- 172.21.15.81:0/1910368965 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2c24041100 con 0x7f2c30060220 2023-11-15T06:31:25.193 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:25.185+0000 7f2c2effd700 1 --2- 172.21.15.81:0/1910368965 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f2c1802ffb0 0x7f2c18032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:31:25.193 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:25.186+0000 7f2c2effd700 1 --2- 172.21.15.81:0/1910368965 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f2c1802ffb0 0x7f2c18032470 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f2c200096a0 tx=0x7f2c20006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:31:25.194 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:25.190+0000 7f2c17fff700 1 -- 172.21.15.81:0/1910368965 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2c24014420 con 0x7f2c30060220 2023-11-15T06:31:25.398 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:25.394+0000 7f2c35dbd700 1 -- 172.21.15.81:0/1910368965 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2c30108070 con 0x7f2c30060220 2023-11-15T06:31:25.399 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:25.395+0000 7f2c17fff700 1 -- 172.21.15.81:0/1910368965 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2c24010230 con 0x7f2c30060220 2023-11-15T06:31:25.399 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:31:25.400 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:31:25.401 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:25.396+0000 7f2c35dbd700 1 -- 172.21.15.81:0/1910368965 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f2c1802ffb0 msgr2=0x7f2c18032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:25.402 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:25.396+0000 7f2c35dbd700 1 --2- 172.21.15.81:0/1910368965 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f2c1802ffb0 0x7f2c18032470 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f2c200096a0 tx=0x7f2c20006b40).stop 2023-11-15T06:31:25.402 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:25.396+0000 7f2c35dbd700 1 -- 172.21.15.81:0/1910368965 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c30060220 msgr2=0x7f2c301066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:25.403 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:25.396+0000 7f2c35dbd700 1 --2- 172.21.15.81:0/1910368965 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c30060220 0x7f2c301066b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f2c24013510 tx=0x7f2c2400b040).stop 2023-11-15T06:31:25.403 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:25.396+0000 7f2c35dbd700 1 -- 172.21.15.81:0/1910368965 shutdown_connections 2023-11-15T06:31:25.403 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:25.396+0000 7f2c35dbd700 1 --2- 172.21.15.81:0/1910368965 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f2c1802ffb0 0x7f2c18032470 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:25.404 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:25.396+0000 7f2c35dbd700 1 --2- 172.21.15.81:0/1910368965 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c30060220 0x7f2c301066b0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:25.404 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:25.396+0000 7f2c35dbd700 1 -- 172.21.15.81:0/1910368965 >> 172.21.15.81:0/1910368965 conn(0x7f2c300f0b70 msgr2=0x7f2c300f1650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:31:25.404 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:25.397+0000 7f2c35dbd700 1 -- 172.21.15.81:0/1910368965 shutdown_connections 2023-11-15T06:31:25.405 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:25.397+0000 7f2c35dbd700 1 -- 172.21.15.81:0/1910368965 wait complete. 2023-11-15T06:31:25.405 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:31:26.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:25 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:31:25.396838+0000 mon.smithi062 (mon.0) 242 : audit [DBG] from='client.? 172.21.15.81:0/1910368965' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:31:26.831 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:31:26.832 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:31:27.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:26 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:31:25.830264+0000 mgr.smithi062.rwurtk (mgr.14164) 172 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:31:28.161 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:28.156+0000 7f3283fff700 1 -- 172.21.15.81:0/1171596650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32840f51a0 msgr2=0x7f32840f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:28.162 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:28.156+0000 7f3283fff700 1 --2- 172.21.15.81:0/1171596650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32840f51a0 0x7f32840f55c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f3274009fb0 tx=0x7f3274009b30).stop 2023-11-15T06:31:28.162 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:28.156+0000 7f3283fff700 1 -- 172.21.15.81:0/1171596650 shutdown_connections 2023-11-15T06:31:28.162 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:28.156+0000 7f3283fff700 1 --2- 172.21.15.81:0/1171596650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32840f51a0 0x7f32840f55c0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:28.162 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:28.156+0000 7f3283fff700 1 -- 172.21.15.81:0/1171596650 >> 172.21.15.81:0/1171596650 conn(0x7f32840f09a0 msgr2=0x7f32840f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:31:28.163 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:28.156+0000 7f3283fff700 1 -- 172.21.15.81:0/1171596650 shutdown_connections 2023-11-15T06:31:28.163 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:28.156+0000 7f3283fff700 1 -- 172.21.15.81:0/1171596650 wait complete. 2023-11-15T06:31:28.163 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:28.157+0000 7f3283fff700 1 Processor -- start 2023-11-15T06:31:28.163 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:28.157+0000 7f3283fff700 1 -- start start 2023-11-15T06:31:28.163 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:28.157+0000 7f3283fff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32840f51a0 0x7f32840fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:31:28.164 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:28.157+0000 7f3283fff700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32840fb5a0 con 0x7f32840f51a0 2023-11-15T06:31:28.164 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:28.157+0000 7f3282ffd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32840f51a0 0x7f32840fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:31:28.164 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:28.157+0000 7f3282ffd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32840f51a0 0x7f32840fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:33380/0 (socket says 172.21.15.81:33380) 2023-11-15T06:31:28.164 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:28.157+0000 7f3282ffd700 1 -- 172.21.15.81:0/3114423070 learned_addr learned my addr 172.21.15.81:0/3114423070 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:31:28.165 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:28.158+0000 7f3282ffd700 1 -- 172.21.15.81:0/3114423070 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3274005040 con 0x7f32840f51a0 2023-11-15T06:31:28.165 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:28.158+0000 7f3282ffd700 1 --2- 172.21.15.81:0/3114423070 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32840f51a0 0x7f32840fd120 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f3274013510 tx=0x7f327400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:31:28.165 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:28.158+0000 7f326b7fe700 1 -- 172.21.15.81:0/3114423070 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3274009320 con 0x7f32840f51a0 2023-11-15T06:31:28.165 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:28.158+0000 7f3283fff700 1 -- 172.21.15.81:0/3114423070 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32840fb740 con 0x7f32840f51a0 2023-11-15T06:31:28.165 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:28.158+0000 7f326b7fe700 1 -- 172.21.15.81:0/3114423070 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3274013f00 con 0x7f32840f51a0 2023-11-15T06:31:28.166 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:28.158+0000 7f326b7fe700 1 -- 172.21.15.81:0/3114423070 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3274014dc0 con 0x7f32840f51a0 2023-11-15T06:31:28.166 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:28.158+0000 7f3283fff700 1 -- 172.21.15.81:0/3114423070 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32840fbbd0 con 0x7f32840f51a0 2023-11-15T06:31:28.167 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:28.159+0000 7f326b7fe700 1 -- 172.21.15.81:0/3114423070 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f327401d3d0 con 0x7f32840f51a0 2023-11-15T06:31:28.167 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:28.159+0000 7f326b7fe700 1 --2- 172.21.15.81:0/3114423070 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f326c02ff60 0x7f326c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:31:28.167 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:28.159+0000 7f326b7fe700 1 -- 172.21.15.81:0/3114423070 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3274040fe0 con 0x7f32840f51a0 2023-11-15T06:31:28.168 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:28.159+0000 7f3283fff700 1 -- 172.21.15.81:0/3114423070 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3284059100 con 0x7f32840f51a0 2023-11-15T06:31:28.168 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:28.159+0000 7f32827fc700 1 --2- 172.21.15.81:0/3114423070 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f326c02ff60 0x7f326c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:31:28.168 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:28.160+0000 7f32827fc700 1 --2- 172.21.15.81:0/3114423070 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f326c02ff60 0x7f326c032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f32780096a0 tx=0x7f3278006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:31:28.168 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:28.164+0000 7f326b7fe700 1 -- 172.21.15.81:0/3114423070 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f327401db80 con 0x7f32840f51a0 2023-11-15T06:31:28.370 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:28.367+0000 7f3283fff700 1 -- 172.21.15.81:0/3114423070 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f32840fabe0 con 0x7f32840f51a0 2023-11-15T06:31:28.371 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:28.368+0000 7f326b7fe700 1 -- 172.21.15.81:0/3114423070 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f327402b020 con 0x7f32840f51a0 2023-11-15T06:31:28.371 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:31:28.372 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:31:28.373 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:28.370+0000 7f3283fff700 1 -- 172.21.15.81:0/3114423070 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f326c02ff60 msgr2=0x7f326c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:28.373 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:28.370+0000 7f3283fff700 1 --2- 172.21.15.81:0/3114423070 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f326c02ff60 0x7f326c032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f32780096a0 tx=0x7f3278006b40).stop 2023-11-15T06:31:28.373 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:28.370+0000 7f3283fff700 1 -- 172.21.15.81:0/3114423070 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32840f51a0 msgr2=0x7f32840fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:28.373 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:28.370+0000 7f3283fff700 1 --2- 172.21.15.81:0/3114423070 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32840f51a0 0x7f32840fd120 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f3274013510 tx=0x7f327400b040).stop 2023-11-15T06:31:28.374 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:28.370+0000 7f3283fff700 1 -- 172.21.15.81:0/3114423070 shutdown_connections 2023-11-15T06:31:28.374 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:28.370+0000 7f3283fff700 1 --2- 172.21.15.81:0/3114423070 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f326c02ff60 0x7f326c032420 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:28.374 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:28.370+0000 7f3283fff700 1 --2- 172.21.15.81:0/3114423070 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32840f51a0 0x7f32840fd120 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:28.374 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:28.370+0000 7f3283fff700 1 -- 172.21.15.81:0/3114423070 >> 172.21.15.81:0/3114423070 conn(0x7f32840f09a0 msgr2=0x7f328405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:31:28.374 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:28.370+0000 7f3283fff700 1 -- 172.21.15.81:0/3114423070 shutdown_connections 2023-11-15T06:31:28.375 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:28.370+0000 7f3283fff700 1 -- 172.21.15.81:0/3114423070 wait complete. 2023-11-15T06:31:28.375 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:31:28.919 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:28 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:31:27.830651+0000 mgr.smithi062.rwurtk (mgr.14164) 173 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:31:28.919 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:28 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:31:28.369969+0000 mon.smithi062 (mon.0) 243 : audit [DBG] from='client.? 172.21.15.81:0/3114423070' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:31:29.798 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:31:29.799 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:31:31.081 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:31.074+0000 7f2664707700 1 -- 172.21.15.81:0/199654727 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f265c060280 msgr2=0x7f265c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:31.082 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:31.074+0000 7f2664707700 1 --2- 172.21.15.81:0/199654727 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f265c060280 0x7f265c0f90f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f2658009fb0 tx=0x7f2658009b30).stop 2023-11-15T06:31:31.082 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:31.074+0000 7f2664707700 1 -- 172.21.15.81:0/199654727 shutdown_connections 2023-11-15T06:31:31.083 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:31.074+0000 7f2664707700 1 --2- 172.21.15.81:0/199654727 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f265c060280 0x7f265c0f90f0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:31.083 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:31.074+0000 7f2664707700 1 -- 172.21.15.81:0/199654727 >> 172.21.15.81:0/199654727 conn(0x7f265c0f0b70 msgr2=0x7f265c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:31:31.084 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:31.074+0000 7f2664707700 1 -- 172.21.15.81:0/199654727 shutdown_connections 2023-11-15T06:31:31.084 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:31.074+0000 7f2664707700 1 -- 172.21.15.81:0/199654727 wait complete. 2023-11-15T06:31:31.085 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:31.075+0000 7f2664707700 1 Processor -- start 2023-11-15T06:31:31.085 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:31.075+0000 7f2664707700 1 -- start start 2023-11-15T06:31:31.085 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:31.075+0000 7f2664707700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f265c060280 0x7f265c0f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:31:31.086 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:31.075+0000 7f2664707700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f265c0f4ff0 con 0x7f265c060280 2023-11-15T06:31:31.087 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:31.075+0000 7f26624a3700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f265c060280 0x7f265c0f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:31:31.087 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:31.075+0000 7f26624a3700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f265c060280 0x7f265c0f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:33410/0 (socket says 172.21.15.81:33410) 2023-11-15T06:31:31.087 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:31.075+0000 7f26624a3700 1 -- 172.21.15.81:0/3889921300 learned_addr learned my addr 172.21.15.81:0/3889921300 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:31:31.088 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:31.075+0000 7f26624a3700 1 -- 172.21.15.81:0/3889921300 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2658005040 con 0x7f265c060280 2023-11-15T06:31:31.088 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:31.075+0000 7f26624a3700 1 --2- 172.21.15.81:0/3889921300 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f265c060280 0x7f265c0f6b70 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f2658013510 tx=0x7f265800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:31:31.088 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:31.076+0000 7f2652ffd700 1 -- 172.21.15.81:0/3889921300 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2658009320 con 0x7f265c060280 2023-11-15T06:31:31.089 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:31.076+0000 7f2664707700 1 -- 172.21.15.81:0/3889921300 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f265c0f5190 con 0x7f265c060280 2023-11-15T06:31:31.089 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:31.076+0000 7f2652ffd700 1 -- 172.21.15.81:0/3889921300 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2658013f00 con 0x7f265c060280 2023-11-15T06:31:31.089 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:31.076+0000 7f2652ffd700 1 -- 172.21.15.81:0/3889921300 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2658014dc0 con 0x7f265c060280 2023-11-15T06:31:31.089 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:31.076+0000 7f2664707700 1 -- 172.21.15.81:0/3889921300 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f265c0f5620 con 0x7f265c060280 2023-11-15T06:31:31.090 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:31.076+0000 7f2652ffd700 1 -- 172.21.15.81:0/3889921300 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f26580095f0 con 0x7f265c060280 2023-11-15T06:31:31.090 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:31.076+0000 7f2664707700 1 -- 172.21.15.81:0/3889921300 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f265c059100 con 0x7f265c060280 2023-11-15T06:31:31.090 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:31.076+0000 7f2652ffd700 1 --2- 172.21.15.81:0/3889921300 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f264802ff60 0x7f2648032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:31:31.090 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:31.077+0000 7f2652ffd700 1 -- 172.21.15.81:0/3889921300 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f26580410c0 con 0x7f265c060280 2023-11-15T06:31:31.091 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:31.077+0000 7f2661ca2700 1 --2- 172.21.15.81:0/3889921300 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f264802ff60 0x7f2648032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:31:31.091 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:31.077+0000 7f2661ca2700 1 --2- 172.21.15.81:0/3889921300 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f264802ff60 0x7f2648032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f264c0096a0 tx=0x7f264c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:31:31.091 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:31.081+0000 7f2652ffd700 1 -- 172.21.15.81:0/3889921300 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2658014420 con 0x7f265c060280 2023-11-15T06:31:31.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:30 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:31:29.831008+0000 mgr.smithi062.rwurtk (mgr.14164) 174 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:31:31.288 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:31.285+0000 7f2664707700 1 -- 172.21.15.81:0/3889921300 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f265c0fbf50 con 0x7f265c060280 2023-11-15T06:31:31.289 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:31.286+0000 7f2652ffd700 1 -- 172.21.15.81:0/3889921300 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2658010230 con 0x7f265c060280 2023-11-15T06:31:31.289 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:31:31.290 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:31:31.291 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:31.287+0000 7f2664707700 1 -- 172.21.15.81:0/3889921300 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f264802ff60 msgr2=0x7f2648032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:31.291 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:31.287+0000 7f2664707700 1 --2- 172.21.15.81:0/3889921300 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f264802ff60 0x7f2648032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f264c0096a0 tx=0x7f264c006b40).stop 2023-11-15T06:31:31.291 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:31.287+0000 7f2664707700 1 -- 172.21.15.81:0/3889921300 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f265c060280 msgr2=0x7f265c0f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:31.291 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:31.287+0000 7f2664707700 1 --2- 172.21.15.81:0/3889921300 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f265c060280 0x7f265c0f6b70 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f2658013510 tx=0x7f265800b040).stop 2023-11-15T06:31:31.292 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:31.287+0000 7f2664707700 1 -- 172.21.15.81:0/3889921300 shutdown_connections 2023-11-15T06:31:31.292 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:31.287+0000 7f2664707700 1 --2- 172.21.15.81:0/3889921300 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f264802ff60 0x7f2648032420 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:31.292 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:31.287+0000 7f2664707700 1 --2- 172.21.15.81:0/3889921300 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f265c060280 0x7f265c0f6b70 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:31.292 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:31.287+0000 7f2664707700 1 -- 172.21.15.81:0/3889921300 >> 172.21.15.81:0/3889921300 conn(0x7f265c0f0b70 msgr2=0x7f265c0f1600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:31:31.292 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:31.287+0000 7f2664707700 1 -- 172.21.15.81:0/3889921300 shutdown_connections 2023-11-15T06:31:31.293 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:31.287+0000 7f2664707700 1 -- 172.21.15.81:0/3889921300 wait complete. 2023-11-15T06:31:31.293 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:31:32.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:31 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:31:31.287689+0000 mon.smithi062 (mon.0) 244 : audit [DBG] from='client.? 172.21.15.81:0/3889921300' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:31:32.679 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:31:32.679 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:31:33.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:32 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:31:31.831373+0000 mgr.smithi062.rwurtk (mgr.14164) 175 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:31:33.885 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:33.882+0000 7fdecab90700 1 -- 172.21.15.81:0/3618385736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdec4060240 msgr2=0x7fdec40f9130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:33.885 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:33.882+0000 7fdecab90700 1 --2- 172.21.15.81:0/3618385736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdec4060240 0x7fdec40f9130 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7fdeb8009fb0 tx=0x7fdeb8009b30).stop 2023-11-15T06:31:33.886 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:33.882+0000 7fdecab90700 1 -- 172.21.15.81:0/3618385736 shutdown_connections 2023-11-15T06:31:33.886 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:33.882+0000 7fdecab90700 1 --2- 172.21.15.81:0/3618385736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdec4060240 0x7fdec40f9130 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:33.886 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:33.882+0000 7fdecab90700 1 -- 172.21.15.81:0/3618385736 >> 172.21.15.81:0/3618385736 conn(0x7fdec40f0b70 msgr2=0x7fdec40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:31:33.886 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:33.882+0000 7fdecab90700 1 -- 172.21.15.81:0/3618385736 shutdown_connections 2023-11-15T06:31:33.887 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:33.882+0000 7fdecab90700 1 -- 172.21.15.81:0/3618385736 wait complete. 2023-11-15T06:31:33.887 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:33.882+0000 7fdecab90700 1 Processor -- start 2023-11-15T06:31:33.887 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:33.882+0000 7fdecab90700 1 -- start start 2023-11-15T06:31:33.889 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:33.883+0000 7fdecab90700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdec4060240 0x7fdec41066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:31:33.889 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:33.883+0000 7fdecab90700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec4106bc0 con 0x7fdec4060240 2023-11-15T06:31:33.889 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:33.883+0000 7fdec892c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdec4060240 0x7fdec41066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:31:33.889 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:33.883+0000 7fdec892c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdec4060240 0x7fdec41066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:44340/0 (socket says 172.21.15.81:44340) 2023-11-15T06:31:33.890 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:33.883+0000 7fdec892c700 1 -- 172.21.15.81:0/3311661610 learned_addr learned my addr 172.21.15.81:0/3311661610 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:31:33.890 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:33.883+0000 7fdec892c700 1 -- 172.21.15.81:0/3311661610 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeb8005040 con 0x7fdec4060240 2023-11-15T06:31:33.890 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:33.883+0000 7fdec892c700 1 --2- 172.21.15.81:0/3311661610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdec4060240 0x7fdec41066d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7fdeb8013510 tx=0x7fdeb800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:31:33.890 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:33.884+0000 7fdec17fa700 1 -- 172.21.15.81:0/3311661610 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdeb8009320 con 0x7fdec4060240 2023-11-15T06:31:33.890 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:33.884+0000 7fdecab90700 1 -- 172.21.15.81:0/3311661610 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdec4106dc0 con 0x7fdec4060240 2023-11-15T06:31:33.891 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:33.884+0000 7fdec17fa700 1 -- 172.21.15.81:0/3311661610 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdeb8013f00 con 0x7fdec4060240 2023-11-15T06:31:33.891 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:33.884+0000 7fdec17fa700 1 -- 172.21.15.81:0/3311661610 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdeb8014dc0 con 0x7fdec4060240 2023-11-15T06:31:33.891 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:33.884+0000 7fdecab90700 1 -- 172.21.15.81:0/3311661610 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdec4107250 con 0x7fdec4060240 2023-11-15T06:31:33.891 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:33.884+0000 7fdec17fa700 1 -- 172.21.15.81:0/3311661610 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdeb801d3d0 con 0x7fdec4060240 2023-11-15T06:31:33.891 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:33.884+0000 7fdec17fa700 1 --2- 172.21.15.81:0/3311661610 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fdeb402ff60 0x7fdeb4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:31:33.892 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:33.884+0000 7fdecab90700 1 -- 172.21.15.81:0/3311661610 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdec4045ad0 con 0x7fdec4060240 2023-11-15T06:31:33.892 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:33.885+0000 7fdec17fa700 1 -- 172.21.15.81:0/3311661610 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdeb8041090 con 0x7fdec4060240 2023-11-15T06:31:33.892 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:33.885+0000 7fdec3fff700 1 --2- 172.21.15.81:0/3311661610 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fdeb402ff60 0x7fdeb4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:31:33.892 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:33.885+0000 7fdec3fff700 1 --2- 172.21.15.81:0/3311661610 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fdeb402ff60 0x7fdeb4032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fdeb00096a0 tx=0x7fdeb0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:31:33.893 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:33.889+0000 7fdec17fa700 1 -- 172.21.15.81:0/3311661610 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdeb80126f0 con 0x7fdec4060240 2023-11-15T06:31:34.096 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:34.093+0000 7fdecab90700 1 -- 172.21.15.81:0/3311661610 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdec40fbf60 con 0x7fdec4060240 2023-11-15T06:31:34.097 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:34.094+0000 7fdec17fa700 1 -- 172.21.15.81:0/3311661610 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdeb8021120 con 0x7fdec4060240 2023-11-15T06:31:34.097 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:31:34.098 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:31:34.099 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:34.096+0000 7fdecab90700 1 -- 172.21.15.81:0/3311661610 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fdeb402ff60 msgr2=0x7fdeb4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:34.099 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:34.096+0000 7fdecab90700 1 --2- 172.21.15.81:0/3311661610 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fdeb402ff60 0x7fdeb4032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fdeb00096a0 tx=0x7fdeb0006b40).stop 2023-11-15T06:31:34.099 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:34.096+0000 7fdecab90700 1 -- 172.21.15.81:0/3311661610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdec4060240 msgr2=0x7fdec41066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:34.099 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:34.096+0000 7fdecab90700 1 --2- 172.21.15.81:0/3311661610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdec4060240 0x7fdec41066d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7fdeb8013510 tx=0x7fdeb800b040).stop 2023-11-15T06:31:34.100 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:34.096+0000 7fdecab90700 1 -- 172.21.15.81:0/3311661610 shutdown_connections 2023-11-15T06:31:34.100 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:34.096+0000 7fdecab90700 1 --2- 172.21.15.81:0/3311661610 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fdeb402ff60 0x7fdeb4032420 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:34.100 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:34.096+0000 7fdecab90700 1 --2- 172.21.15.81:0/3311661610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdec4060240 0x7fdec41066d0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:34.100 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:34.096+0000 7fdecab90700 1 -- 172.21.15.81:0/3311661610 >> 172.21.15.81:0/3311661610 conn(0x7fdec40f0b70 msgr2=0x7fdec40f1690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:31:34.100 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:34.096+0000 7fdecab90700 1 -- 172.21.15.81:0/3311661610 shutdown_connections 2023-11-15T06:31:34.100 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:34.096+0000 7fdecab90700 1 -- 172.21.15.81:0/3311661610 wait complete. 2023-11-15T06:31:34.101 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:31:35.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:34 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:31:33.831631+0000 mgr.smithi062.rwurtk (mgr.14164) 176 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:31:35.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:34 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:31:34.095990+0000 mon.smithi062 (mon.0) 245 : audit [DBG] from='client.? 172.21.15.81:0/3311661610' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:31:35.569 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:31:35.570 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:31:36.811 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:36.808+0000 7fa472625700 1 -- 172.21.15.81:0/814687985 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa46c0f51a0 msgr2=0x7fa46c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:36.812 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:36.808+0000 7fa472625700 1 --2- 172.21.15.81:0/814687985 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa46c0f51a0 0x7fa46c0f55c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7fa468009fb0 tx=0x7fa468009b30).stop 2023-11-15T06:31:36.812 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:36.808+0000 7fa472625700 1 -- 172.21.15.81:0/814687985 shutdown_connections 2023-11-15T06:31:36.812 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:36.808+0000 7fa472625700 1 --2- 172.21.15.81:0/814687985 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa46c0f51a0 0x7fa46c0f55c0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:36.812 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:36.808+0000 7fa472625700 1 -- 172.21.15.81:0/814687985 >> 172.21.15.81:0/814687985 conn(0x7fa46c0f09a0 msgr2=0x7fa46c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:31:36.813 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:36.808+0000 7fa472625700 1 -- 172.21.15.81:0/814687985 shutdown_connections 2023-11-15T06:31:36.813 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:36.808+0000 7fa472625700 1 -- 172.21.15.81:0/814687985 wait complete. 2023-11-15T06:31:36.813 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:36.808+0000 7fa472625700 1 Processor -- start 2023-11-15T06:31:36.813 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:36.808+0000 7fa472625700 1 -- start start 2023-11-15T06:31:36.813 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:36.809+0000 7fa472625700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa46c0f51a0 0x7fa46c1091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:31:36.813 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:36.809+0000 7fa472625700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa46c060240 con 0x7fa46c0f51a0 2023-11-15T06:31:36.815 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:36.809+0000 7fa471623700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa46c0f51a0 0x7fa46c1091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:31:36.815 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:36.809+0000 7fa471623700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa46c0f51a0 0x7fa46c1091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:44360/0 (socket says 172.21.15.81:44360) 2023-11-15T06:31:36.816 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:36.809+0000 7fa471623700 1 -- 172.21.15.81:0/3129958645 learned_addr learned my addr 172.21.15.81:0/3129958645 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:31:36.816 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:36.809+0000 7fa471623700 1 -- 172.21.15.81:0/3129958645 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa468005040 con 0x7fa46c0f51a0 2023-11-15T06:31:36.816 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:36.809+0000 7fa471623700 1 --2- 172.21.15.81:0/3129958645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa46c0f51a0 0x7fa46c1091b0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7fa468013510 tx=0x7fa46800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:31:36.816 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:36.809+0000 7fa461ffb700 1 -- 172.21.15.81:0/3129958645 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa468009320 con 0x7fa46c0f51a0 2023-11-15T06:31:36.816 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:36.809+0000 7fa461ffb700 1 -- 172.21.15.81:0/3129958645 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa468013f00 con 0x7fa46c0f51a0 2023-11-15T06:31:36.817 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:36.809+0000 7fa461ffb700 1 -- 172.21.15.81:0/3129958645 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa468014dc0 con 0x7fa46c0f51a0 2023-11-15T06:31:36.817 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:36.809+0000 7fa472625700 1 -- 172.21.15.81:0/3129958645 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa46c060440 con 0x7fa46c0f51a0 2023-11-15T06:31:36.817 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:36.810+0000 7fa472625700 1 -- 172.21.15.81:0/3129958645 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa46c109860 con 0x7fa46c0f51a0 2023-11-15T06:31:36.817 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:36.810+0000 7fa461ffb700 1 -- 172.21.15.81:0/3129958645 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa4680095f0 con 0x7fa46c0f51a0 2023-11-15T06:31:36.817 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:36.810+0000 7fa472625700 1 -- 172.21.15.81:0/3129958645 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa46c045ad0 con 0x7fa46c0f51a0 2023-11-15T06:31:36.818 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:36.811+0000 7fa461ffb700 1 --2- 172.21.15.81:0/3129958645 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fa45802ff60 0x7fa458032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:31:36.818 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:36.811+0000 7fa461ffb700 1 -- 172.21.15.81:0/3129958645 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa468041040 con 0x7fa46c0f51a0 2023-11-15T06:31:36.818 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:36.811+0000 7fa470e22700 1 --2- 172.21.15.81:0/3129958645 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fa45802ff60 0x7fa458032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:31:36.818 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:36.811+0000 7fa470e22700 1 --2- 172.21.15.81:0/3129958645 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fa45802ff60 0x7fa458032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fa45c0096a0 tx=0x7fa45c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:31:36.819 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:36.815+0000 7fa461ffb700 1 -- 172.21.15.81:0/3129958645 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa468014420 con 0x7fa46c0f51a0 2023-11-15T06:31:37.021 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:37.018+0000 7fa472625700 1 -- 172.21.15.81:0/3129958645 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa46c10a080 con 0x7fa46c0f51a0 2023-11-15T06:31:37.022 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:37.019+0000 7fa461ffb700 1 -- 172.21.15.81:0/3129958645 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa468010230 con 0x7fa46c0f51a0 2023-11-15T06:31:37.022 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:31:37.023 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:31:37.023 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:37.020+0000 7fa472625700 1 -- 172.21.15.81:0/3129958645 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fa45802ff60 msgr2=0x7fa458032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:37.024 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:37.020+0000 7fa472625700 1 --2- 172.21.15.81:0/3129958645 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fa45802ff60 0x7fa458032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fa45c0096a0 tx=0x7fa45c006b40).stop 2023-11-15T06:31:37.024 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:37.021+0000 7fa472625700 1 -- 172.21.15.81:0/3129958645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa46c0f51a0 msgr2=0x7fa46c1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:37.024 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:37.021+0000 7fa472625700 1 --2- 172.21.15.81:0/3129958645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa46c0f51a0 0x7fa46c1091b0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7fa468013510 tx=0x7fa46800b040).stop 2023-11-15T06:31:37.024 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:37.021+0000 7fa472625700 1 -- 172.21.15.81:0/3129958645 shutdown_connections 2023-11-15T06:31:37.024 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:37.021+0000 7fa472625700 1 --2- 172.21.15.81:0/3129958645 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fa45802ff60 0x7fa458032420 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:37.025 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:37.021+0000 7fa472625700 1 --2- 172.21.15.81:0/3129958645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa46c0f51a0 0x7fa46c1091b0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:37.025 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:37.021+0000 7fa472625700 1 -- 172.21.15.81:0/3129958645 >> 172.21.15.81:0/3129958645 conn(0x7fa46c0f09a0 msgr2=0x7fa46c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:31:37.025 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:37.021+0000 7fa472625700 1 -- 172.21.15.81:0/3129958645 shutdown_connections 2023-11-15T06:31:37.025 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:37.021+0000 7fa472625700 1 -- 172.21.15.81:0/3129958645 wait complete. 2023-11-15T06:31:37.026 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:31:37.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:36 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:31:35.831984+0000 mgr.smithi062.rwurtk (mgr.14164) 177 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:31:38.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:37 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:31:37.020932+0000 mon.smithi062 (mon.0) 246 : audit [DBG] from='client.? 172.21.15.81:0/3129958645' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:31:38.413 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:31:38.414 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:31:39.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:38 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:31:37.832347+0000 mgr.smithi062.rwurtk (mgr.14164) 178 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:31:39.669 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:39.665+0000 7fc9ba53f700 1 -- 172.21.15.81:0/203128066 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9b40f6510 msgr2=0x7fc9b40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:39.669 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:39.665+0000 7fc9ba53f700 1 --2- 172.21.15.81:0/203128066 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9b40f6510 0x7fc9b40f6930 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7fc9a4009fb0 tx=0x7fc9a4009b30).stop 2023-11-15T06:31:39.669 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:39.665+0000 7fc9ba53f700 1 -- 172.21.15.81:0/203128066 shutdown_connections 2023-11-15T06:31:39.670 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:39.665+0000 7fc9ba53f700 1 --2- 172.21.15.81:0/203128066 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9b40f6510 0x7fc9b40f6930 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:39.670 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:39.665+0000 7fc9ba53f700 1 -- 172.21.15.81:0/203128066 >> 172.21.15.81:0/203128066 conn(0x7fc9b40f1c70 msgr2=0x7fc9b40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:31:39.671 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:39.665+0000 7fc9ba53f700 1 -- 172.21.15.81:0/203128066 shutdown_connections 2023-11-15T06:31:39.671 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:39.665+0000 7fc9ba53f700 1 -- 172.21.15.81:0/203128066 wait complete. 2023-11-15T06:31:39.671 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:39.666+0000 7fc9ba53f700 1 Processor -- start 2023-11-15T06:31:39.671 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:39.666+0000 7fc9ba53f700 1 -- start start 2023-11-15T06:31:39.673 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:39.666+0000 7fc9ba53f700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9b40f6510 0x7fc9b4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:31:39.674 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:39.666+0000 7fc9ba53f700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b4106c20 con 0x7fc9b40f6510 2023-11-15T06:31:39.674 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:39.666+0000 7fc9b3fff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9b40f6510 0x7fc9b4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:31:39.674 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:39.666+0000 7fc9b3fff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9b40f6510 0x7fc9b4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:44380/0 (socket says 172.21.15.81:44380) 2023-11-15T06:31:39.674 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:39.666+0000 7fc9b3fff700 1 -- 172.21.15.81:0/2578549662 learned_addr learned my addr 172.21.15.81:0/2578549662 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:31:39.674 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:39.667+0000 7fc9b3fff700 1 -- 172.21.15.81:0/2578549662 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a4005040 con 0x7fc9b40f6510 2023-11-15T06:31:39.675 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:39.667+0000 7fc9b3fff700 1 --2- 172.21.15.81:0/2578549662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9b40f6510 0x7fc9b4106730 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fc9a4013510 tx=0x7fc9a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:31:39.675 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:39.667+0000 7fc9b0ff9700 1 -- 172.21.15.81:0/2578549662 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc9a4009320 con 0x7fc9b40f6510 2023-11-15T06:31:39.675 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:39.667+0000 7fc9ba53f700 1 -- 172.21.15.81:0/2578549662 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9b4106e20 con 0x7fc9b40f6510 2023-11-15T06:31:39.675 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:39.667+0000 7fc9b0ff9700 1 -- 172.21.15.81:0/2578549662 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc9a4013f00 con 0x7fc9b40f6510 2023-11-15T06:31:39.675 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:39.667+0000 7fc9b0ff9700 1 -- 172.21.15.81:0/2578549662 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc9a4014d90 con 0x7fc9b40f6510 2023-11-15T06:31:39.676 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:39.667+0000 7fc9ba53f700 1 -- 172.21.15.81:0/2578549662 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9b41072b0 con 0x7fc9b40f6510 2023-11-15T06:31:39.676 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:39.668+0000 7fc9b0ff9700 1 -- 172.21.15.81:0/2578549662 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc9a4014420 con 0x7fc9b40f6510 2023-11-15T06:31:39.676 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:39.668+0000 7fc9ba53f700 1 -- 172.21.15.81:0/2578549662 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9b4045ad0 con 0x7fc9b40f6510 2023-11-15T06:31:39.676 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:39.668+0000 7fc9b0ff9700 1 --2- 172.21.15.81:0/2578549662 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fc99c02ff60 0x7fc99c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:31:39.676 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:39.668+0000 7fc9b0ff9700 1 -- 172.21.15.81:0/2578549662 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc9a4041000 con 0x7fc9b40f6510 2023-11-15T06:31:39.677 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:39.668+0000 7fc9b37fe700 1 --2- 172.21.15.81:0/2578549662 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fc99c02ff60 0x7fc99c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:31:39.677 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:39.669+0000 7fc9b37fe700 1 --2- 172.21.15.81:0/2578549662 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fc99c02ff60 0x7fc99c032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fc9a80096a0 tx=0x7fc9a8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:31:39.677 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:39.673+0000 7fc9b0ff9700 1 -- 172.21.15.81:0/2578549662 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc9a40095f0 con 0x7fc9b40f6510 2023-11-15T06:31:39.880 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:39.877+0000 7fc9ba53f700 1 -- 172.21.15.81:0/2578549662 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc9b40fbf60 con 0x7fc9b40f6510 2023-11-15T06:31:39.880 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:39.877+0000 7fc9b0ff9700 1 -- 172.21.15.81:0/2578549662 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc9a401db70 con 0x7fc9b40f6510 2023-11-15T06:31:39.881 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:31:39.881 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:31:39.882 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:39.879+0000 7fc9ba53f700 1 -- 172.21.15.81:0/2578549662 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fc99c02ff60 msgr2=0x7fc99c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:39.882 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:39.879+0000 7fc9ba53f700 1 --2- 172.21.15.81:0/2578549662 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fc99c02ff60 0x7fc99c032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fc9a80096a0 tx=0x7fc9a8006b40).stop 2023-11-15T06:31:39.882 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:39.879+0000 7fc9ba53f700 1 -- 172.21.15.81:0/2578549662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9b40f6510 msgr2=0x7fc9b4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:39.883 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:39.879+0000 7fc9ba53f700 1 --2- 172.21.15.81:0/2578549662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9b40f6510 0x7fc9b4106730 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fc9a4013510 tx=0x7fc9a400b040).stop 2023-11-15T06:31:39.883 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:39.879+0000 7fc9ba53f700 1 -- 172.21.15.81:0/2578549662 shutdown_connections 2023-11-15T06:31:39.883 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:39.879+0000 7fc9ba53f700 1 --2- 172.21.15.81:0/2578549662 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fc99c02ff60 0x7fc99c032420 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:39.883 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:39.879+0000 7fc9ba53f700 1 --2- 172.21.15.81:0/2578549662 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9b40f6510 0x7fc9b4106730 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:39.883 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:39.879+0000 7fc9ba53f700 1 -- 172.21.15.81:0/2578549662 >> 172.21.15.81:0/2578549662 conn(0x7fc9b40f1c70 msgr2=0x7fc9b40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:31:39.883 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:39.879+0000 7fc9ba53f700 1 -- 172.21.15.81:0/2578549662 shutdown_connections 2023-11-15T06:31:39.884 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:39.879+0000 7fc9ba53f700 1 -- 172.21.15.81:0/2578549662 wait complete. 2023-11-15T06:31:39.884 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:31:40.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:39 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:31:39.879282+0000 mon.smithi062 (mon.0) 247 : audit [DBG] from='client.? 172.21.15.81:0/2578549662' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:31:41.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:40 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:31:39.832697+0000 mgr.smithi062.rwurtk (mgr.14164) 179 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:31:41.338 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:31:41.339 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:31:42.593 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:42.590+0000 7f683f990700 1 -- 172.21.15.81:0/2061797186 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68380f6510 msgr2=0x7f68380f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:42.593 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:42.590+0000 7f683f990700 1 --2- 172.21.15.81:0/2061797186 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68380f6510 0x7f68380f6930 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f6834009fb0 tx=0x7f6834009b30).stop 2023-11-15T06:31:42.594 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:42.590+0000 7f683f990700 1 -- 172.21.15.81:0/2061797186 shutdown_connections 2023-11-15T06:31:42.594 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:42.590+0000 7f683f990700 1 --2- 172.21.15.81:0/2061797186 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68380f6510 0x7f68380f6930 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:42.594 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:42.590+0000 7f683f990700 1 -- 172.21.15.81:0/2061797186 >> 172.21.15.81:0/2061797186 conn(0x7f68380f1c70 msgr2=0x7f68380f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:31:42.594 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:42.590+0000 7f683f990700 1 -- 172.21.15.81:0/2061797186 shutdown_connections 2023-11-15T06:31:42.595 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:42.590+0000 7f683f990700 1 -- 172.21.15.81:0/2061797186 wait complete. 2023-11-15T06:31:42.596 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:42.591+0000 7f683f990700 1 Processor -- start 2023-11-15T06:31:42.597 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:42.591+0000 7f683f990700 1 -- start start 2023-11-15T06:31:42.597 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:42.591+0000 7f683f990700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68380f6510 0x7f6838106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:31:42.597 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:42.591+0000 7f683f990700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6838106c20 con 0x7f68380f6510 2023-11-15T06:31:42.597 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:42.591+0000 7f683d72c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68380f6510 0x7f6838106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:31:42.597 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:42.591+0000 7f683d72c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68380f6510 0x7f6838106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:44600/0 (socket says 172.21.15.81:44600) 2023-11-15T06:31:42.598 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:42.591+0000 7f683d72c700 1 -- 172.21.15.81:0/3113568567 learned_addr learned my addr 172.21.15.81:0/3113568567 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:31:42.598 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:42.591+0000 7f683d72c700 1 -- 172.21.15.81:0/3113568567 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6834005040 con 0x7f68380f6510 2023-11-15T06:31:42.598 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:42.591+0000 7f683d72c700 1 --2- 172.21.15.81:0/3113568567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68380f6510 0x7f6838106730 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f6834013510 tx=0x7f683400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:31:42.598 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:42.592+0000 7f682dffb700 1 -- 172.21.15.81:0/3113568567 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6834009320 con 0x7f68380f6510 2023-11-15T06:31:42.599 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:42.592+0000 7f683f990700 1 -- 172.21.15.81:0/3113568567 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6838106e20 con 0x7f68380f6510 2023-11-15T06:31:42.599 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:42.592+0000 7f682dffb700 1 -- 172.21.15.81:0/3113568567 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6834013f00 con 0x7f68380f6510 2023-11-15T06:31:42.599 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:42.592+0000 7f682dffb700 1 -- 172.21.15.81:0/3113568567 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6834014d90 con 0x7f68380f6510 2023-11-15T06:31:42.599 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:42.592+0000 7f683f990700 1 -- 172.21.15.81:0/3113568567 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68381072b0 con 0x7f68380f6510 2023-11-15T06:31:42.599 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:42.593+0000 7f683f990700 1 -- 172.21.15.81:0/3113568567 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6838045ad0 con 0x7f68380f6510 2023-11-15T06:31:42.600 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:42.593+0000 7f682dffb700 1 -- 172.21.15.81:0/3113568567 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f68340095f0 con 0x7f68380f6510 2023-11-15T06:31:42.600 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:42.593+0000 7f682dffb700 1 --2- 172.21.15.81:0/3113568567 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f682402ff60 0x7f6824032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:31:42.600 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:42.593+0000 7f682dffb700 1 -- 172.21.15.81:0/3113568567 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6834040fc0 con 0x7f68380f6510 2023-11-15T06:31:42.601 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:42.593+0000 7f683cf2b700 1 --2- 172.21.15.81:0/3113568567 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f682402ff60 0x7f6824032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:31:42.601 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:42.593+0000 7f683cf2b700 1 --2- 172.21.15.81:0/3113568567 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f682402ff60 0x7f6824032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f68280096a0 tx=0x7f6828006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:31:42.601 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:42.598+0000 7f682dffb700 1 -- 172.21.15.81:0/3113568567 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6834014420 con 0x7f68380f6510 2023-11-15T06:31:42.804 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:42.801+0000 7f683f990700 1 -- 172.21.15.81:0/3113568567 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f68380fbf60 con 0x7f68380f6510 2023-11-15T06:31:42.805 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:42.802+0000 7f682dffb700 1 -- 172.21.15.81:0/3113568567 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6834010230 con 0x7f68380f6510 2023-11-15T06:31:42.805 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:31:42.806 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:31:42.807 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:42.803+0000 7f683f990700 1 -- 172.21.15.81:0/3113568567 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f682402ff60 msgr2=0x7f6824032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:42.807 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:42.803+0000 7f683f990700 1 --2- 172.21.15.81:0/3113568567 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f682402ff60 0x7f6824032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f68280096a0 tx=0x7f6828006b40).stop 2023-11-15T06:31:42.807 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:42.803+0000 7f683f990700 1 -- 172.21.15.81:0/3113568567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68380f6510 msgr2=0x7f6838106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:42.807 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:42.803+0000 7f683f990700 1 --2- 172.21.15.81:0/3113568567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68380f6510 0x7f6838106730 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f6834013510 tx=0x7f683400b040).stop 2023-11-15T06:31:42.807 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:42.803+0000 7f683f990700 1 -- 172.21.15.81:0/3113568567 shutdown_connections 2023-11-15T06:31:42.808 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:42.803+0000 7f683f990700 1 --2- 172.21.15.81:0/3113568567 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f682402ff60 0x7f6824032420 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:42.808 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:42.803+0000 7f683f990700 1 --2- 172.21.15.81:0/3113568567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68380f6510 0x7f6838106730 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:42.808 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:42.803+0000 7f683f990700 1 -- 172.21.15.81:0/3113568567 >> 172.21.15.81:0/3113568567 conn(0x7f68380f1c70 msgr2=0x7f68380f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:31:42.808 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:42.804+0000 7f683f990700 1 -- 172.21.15.81:0/3113568567 shutdown_connections 2023-11-15T06:31:42.808 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:42.804+0000 7f683f990700 1 -- 172.21.15.81:0/3113568567 wait complete. 2023-11-15T06:31:42.809 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:31:43.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:42 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:31:41.832928+0000 mgr.smithi062.rwurtk (mgr.14164) 180 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:31:43.169 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:42 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:31:42.803843+0000 mon.smithi062 (mon.0) 248 : audit [DBG] from='client.? 172.21.15.81:0/3113568567' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:31:44.198 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:31:44.199 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:31:45.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:44 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:31:43.833158+0000 mgr.smithi062.rwurtk (mgr.14164) 181 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:31:45.322 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:45.319+0000 7f424c6a0700 1 -- 172.21.15.81:0/229627892 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42440f6510 msgr2=0x7f42440f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:45.323 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:45.319+0000 7f424c6a0700 1 --2- 172.21.15.81:0/229627892 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42440f6510 0x7f42440f6930 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f4234009fb0 tx=0x7f4234009b30).stop 2023-11-15T06:31:45.323 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:45.319+0000 7f424c6a0700 1 -- 172.21.15.81:0/229627892 shutdown_connections 2023-11-15T06:31:45.323 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:45.319+0000 7f424c6a0700 1 --2- 172.21.15.81:0/229627892 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42440f6510 0x7f42440f6930 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:45.323 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:45.319+0000 7f424c6a0700 1 -- 172.21.15.81:0/229627892 >> 172.21.15.81:0/229627892 conn(0x7f42440f1c70 msgr2=0x7f42440f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:31:45.324 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:45.319+0000 7f424c6a0700 1 -- 172.21.15.81:0/229627892 shutdown_connections 2023-11-15T06:31:45.324 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:45.319+0000 7f424c6a0700 1 -- 172.21.15.81:0/229627892 wait complete. 2023-11-15T06:31:45.324 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:45.319+0000 7f424c6a0700 1 Processor -- start 2023-11-15T06:31:45.324 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:45.320+0000 7f424c6a0700 1 -- start start 2023-11-15T06:31:45.324 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:45.320+0000 7f424c6a0700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42440f6510 0x7f4244106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:31:45.325 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:45.320+0000 7f424c6a0700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4244106c20 con 0x7f42440f6510 2023-11-15T06:31:45.325 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:45.320+0000 7f424a43c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42440f6510 0x7f4244106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:31:45.325 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:45.320+0000 7f424a43c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42440f6510 0x7f4244106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:44628/0 (socket says 172.21.15.81:44628) 2023-11-15T06:31:45.325 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:45.320+0000 7f424a43c700 1 -- 172.21.15.81:0/3739094982 learned_addr learned my addr 172.21.15.81:0/3739094982 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:31:45.326 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:45.320+0000 7f424a43c700 1 -- 172.21.15.81:0/3739094982 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4234005040 con 0x7f42440f6510 2023-11-15T06:31:45.326 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:45.320+0000 7f424a43c700 1 --2- 172.21.15.81:0/3739094982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42440f6510 0x7f4244106730 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f4234013510 tx=0x7f423400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:31:45.326 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:45.321+0000 7f423affd700 1 -- 172.21.15.81:0/3739094982 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4234009320 con 0x7f42440f6510 2023-11-15T06:31:45.326 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:45.321+0000 7f424c6a0700 1 -- 172.21.15.81:0/3739094982 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4244106e20 con 0x7f42440f6510 2023-11-15T06:31:45.326 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:45.321+0000 7f423affd700 1 -- 172.21.15.81:0/3739094982 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4234013f00 con 0x7f42440f6510 2023-11-15T06:31:45.327 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:45.321+0000 7f423affd700 1 -- 172.21.15.81:0/3739094982 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4234014d90 con 0x7f42440f6510 2023-11-15T06:31:45.327 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:45.321+0000 7f424c6a0700 1 -- 172.21.15.81:0/3739094982 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42441072b0 con 0x7f42440f6510 2023-11-15T06:31:45.328 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:45.322+0000 7f423affd700 1 -- 172.21.15.81:0/3739094982 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f42340263a0 con 0x7f42440f6510 2023-11-15T06:31:45.328 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:45.322+0000 7f424c6a0700 1 -- 172.21.15.81:0/3739094982 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4244045ad0 con 0x7f42440f6510 2023-11-15T06:31:45.328 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:45.322+0000 7f423affd700 1 --2- 172.21.15.81:0/3739094982 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f423002ff60 0x7f4230032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:31:45.328 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:45.322+0000 7f423affd700 1 -- 172.21.15.81:0/3739094982 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4234040fd0 con 0x7f42440f6510 2023-11-15T06:31:45.329 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:45.322+0000 7f4249c3b700 1 --2- 172.21.15.81:0/3739094982 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f423002ff60 0x7f4230032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:31:45.329 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:45.322+0000 7f4249c3b700 1 --2- 172.21.15.81:0/3739094982 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f423002ff60 0x7f4230032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f42400096a0 tx=0x7f4240006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:31:45.329 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:45.327+0000 7f423affd700 1 -- 172.21.15.81:0/3739094982 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f423406d2b0 con 0x7f42440f6510 2023-11-15T06:31:45.533 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:45.531+0000 7f424c6a0700 1 -- 172.21.15.81:0/3739094982 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f42440fbf60 con 0x7f42440f6510 2023-11-15T06:31:45.533 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:45.531+0000 7f423affd700 1 -- 172.21.15.81:0/3739094982 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f42340095f0 con 0x7f42440f6510 2023-11-15T06:31:45.534 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:31:45.534 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:31:45.535 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:45.533+0000 7f424c6a0700 1 -- 172.21.15.81:0/3739094982 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f423002ff60 msgr2=0x7f4230032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:45.535 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:45.533+0000 7f424c6a0700 1 --2- 172.21.15.81:0/3739094982 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f423002ff60 0x7f4230032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f42400096a0 tx=0x7f4240006b40).stop 2023-11-15T06:31:45.535 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:45.533+0000 7f424c6a0700 1 -- 172.21.15.81:0/3739094982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42440f6510 msgr2=0x7f4244106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:45.536 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:45.533+0000 7f424c6a0700 1 --2- 172.21.15.81:0/3739094982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42440f6510 0x7f4244106730 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f4234013510 tx=0x7f423400b040).stop 2023-11-15T06:31:45.536 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:45.533+0000 7f424c6a0700 1 -- 172.21.15.81:0/3739094982 shutdown_connections 2023-11-15T06:31:45.536 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:45.533+0000 7f424c6a0700 1 --2- 172.21.15.81:0/3739094982 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f423002ff60 0x7f4230032420 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:45.536 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:45.533+0000 7f424c6a0700 1 --2- 172.21.15.81:0/3739094982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42440f6510 0x7f4244106730 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:45.536 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:45.533+0000 7f424c6a0700 1 -- 172.21.15.81:0/3739094982 >> 172.21.15.81:0/3739094982 conn(0x7f42440f1c70 msgr2=0x7f42440f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:31:45.537 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:45.533+0000 7f424c6a0700 1 -- 172.21.15.81:0/3739094982 shutdown_connections 2023-11-15T06:31:45.537 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:45.533+0000 7f424c6a0700 1 -- 172.21.15.81:0/3739094982 wait complete. 2023-11-15T06:31:45.538 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:31:46.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:45 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:31:45.532150+0000 mon.smithi062 (mon.0) 249 : audit [DBG] from='client.? 172.21.15.81:0/3739094982' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:31:46.946 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:31:46.947 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:31:47.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:46 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:31:45.833514+0000 mgr.smithi062.rwurtk (mgr.14164) 182 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:31:48.159 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:48.155+0000 7f5fac8b6700 1 -- 172.21.15.81:0/31477642 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fa80f51a0 msgr2=0x7f5fa80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:48.159 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:48.155+0000 7f5fac8b6700 1 --2- 172.21.15.81:0/31477642 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fa80f51a0 0x7f5fa80f55c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f5fa0009fb0 tx=0x7f5fa0009b30).stop 2023-11-15T06:31:48.160 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:48.155+0000 7f5fac8b6700 1 -- 172.21.15.81:0/31477642 shutdown_connections 2023-11-15T06:31:48.160 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:48.155+0000 7f5fac8b6700 1 --2- 172.21.15.81:0/31477642 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fa80f51a0 0x7f5fa80f55c0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:48.160 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:48.155+0000 7f5fac8b6700 1 -- 172.21.15.81:0/31477642 >> 172.21.15.81:0/31477642 conn(0x7f5fa80f09a0 msgr2=0x7f5fa80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:31:48.160 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:48.155+0000 7f5fac8b6700 1 -- 172.21.15.81:0/31477642 shutdown_connections 2023-11-15T06:31:48.160 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:48.155+0000 7f5fac8b6700 1 -- 172.21.15.81:0/31477642 wait complete. 2023-11-15T06:31:48.161 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:48.156+0000 7f5fac8b6700 1 Processor -- start 2023-11-15T06:31:48.161 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:48.156+0000 7f5fac8b6700 1 -- start start 2023-11-15T06:31:48.161 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:48.156+0000 7f5fac8b6700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fa80f51a0 0x7f5fa81053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:31:48.161 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:48.156+0000 7f5fac8b6700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa81058a0 con 0x7f5fa80f51a0 2023-11-15T06:31:48.161 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:48.156+0000 7f5fa6d9d700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fa80f51a0 0x7f5fa81053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:31:48.162 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:48.156+0000 7f5fa6d9d700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fa80f51a0 0x7f5fa81053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:44658/0 (socket says 172.21.15.81:44658) 2023-11-15T06:31:48.162 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:48.156+0000 7f5fa6d9d700 1 -- 172.21.15.81:0/674608950 learned_addr learned my addr 172.21.15.81:0/674608950 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:31:48.162 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:48.156+0000 7f5fa6d9d700 1 -- 172.21.15.81:0/674608950 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa0005040 con 0x7f5fa80f51a0 2023-11-15T06:31:48.162 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:48.156+0000 7f5fa6d9d700 1 --2- 172.21.15.81:0/674608950 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fa80f51a0 0x7f5fa81053b0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f5fa0013510 tx=0x7f5fa000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:31:48.162 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:48.157+0000 7f5f9f7fe700 1 -- 172.21.15.81:0/674608950 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fa0009320 con 0x7f5fa80f51a0 2023-11-15T06:31:48.163 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:48.157+0000 7f5fac8b6700 1 -- 172.21.15.81:0/674608950 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fa8105aa0 con 0x7f5fa80f51a0 2023-11-15T06:31:48.163 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:48.157+0000 7f5f9f7fe700 1 -- 172.21.15.81:0/674608950 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5fa0013f00 con 0x7f5fa80f51a0 2023-11-15T06:31:48.163 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:48.157+0000 7f5f9f7fe700 1 -- 172.21.15.81:0/674608950 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fa0014dc0 con 0x7f5fa80f51a0 2023-11-15T06:31:48.163 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:48.157+0000 7f5fac8b6700 1 -- 172.21.15.81:0/674608950 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fa8105f30 con 0x7f5fa80f51a0 2023-11-15T06:31:48.164 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:48.157+0000 7f5f9f7fe700 1 -- 172.21.15.81:0/674608950 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5fa001d3d0 con 0x7f5fa80f51a0 2023-11-15T06:31:48.165 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:48.158+0000 7f5fac8b6700 1 -- 172.21.15.81:0/674608950 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fa8045ad0 con 0x7f5fa80f51a0 2023-11-15T06:31:48.165 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:48.158+0000 7f5f9f7fe700 1 --2- 172.21.15.81:0/674608950 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f5f9402ff60 0x7f5f94032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:31:48.165 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:48.158+0000 7f5f9f7fe700 1 -- 172.21.15.81:0/674608950 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5fa00412f0 con 0x7f5fa80f51a0 2023-11-15T06:31:48.165 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:48.158+0000 7f5fa659c700 1 --2- 172.21.15.81:0/674608950 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f5f9402ff60 0x7f5f94032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:31:48.165 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:48.158+0000 7f5fa659c700 1 --2- 172.21.15.81:0/674608950 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f5f9402ff60 0x7f5f94032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f5f900096a0 tx=0x7f5f90006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:31:48.166 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:48.163+0000 7f5f9f7fe700 1 -- 172.21.15.81:0/674608950 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5fa001cad0 con 0x7f5fa80f51a0 2023-11-15T06:31:48.369 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:48.367+0000 7f5fac8b6700 1 -- 172.21.15.81:0/674608950 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5fa8106de0 con 0x7f5fa80f51a0 2023-11-15T06:31:48.369 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:48.367+0000 7f5f9f7fe700 1 -- 172.21.15.81:0/674608950 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5fa001cad0 con 0x7f5fa80f51a0 2023-11-15T06:31:48.370 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:31:48.370 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:31:48.371 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:48.369+0000 7f5fac8b6700 1 -- 172.21.15.81:0/674608950 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f5f9402ff60 msgr2=0x7f5f94032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:48.371 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:48.369+0000 7f5fac8b6700 1 --2- 172.21.15.81:0/674608950 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f5f9402ff60 0x7f5f94032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f5f900096a0 tx=0x7f5f90006b40).stop 2023-11-15T06:31:48.371 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:48.369+0000 7f5fac8b6700 1 -- 172.21.15.81:0/674608950 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fa80f51a0 msgr2=0x7f5fa81053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:48.372 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:48.369+0000 7f5fac8b6700 1 --2- 172.21.15.81:0/674608950 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fa80f51a0 0x7f5fa81053b0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f5fa0013510 tx=0x7f5fa000b040).stop 2023-11-15T06:31:48.372 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:48.369+0000 7f5fac8b6700 1 -- 172.21.15.81:0/674608950 shutdown_connections 2023-11-15T06:31:48.372 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:48.369+0000 7f5fac8b6700 1 --2- 172.21.15.81:0/674608950 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f5f9402ff60 0x7f5f94032420 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:48.372 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:48.369+0000 7f5fac8b6700 1 --2- 172.21.15.81:0/674608950 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5fa80f51a0 0x7f5fa81053b0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:48.372 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:48.369+0000 7f5fac8b6700 1 -- 172.21.15.81:0/674608950 >> 172.21.15.81:0/674608950 conn(0x7f5fa80f09a0 msgr2=0x7f5fa80f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:31:48.373 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:48.369+0000 7f5fac8b6700 1 -- 172.21.15.81:0/674608950 shutdown_connections 2023-11-15T06:31:48.373 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:48.369+0000 7f5fac8b6700 1 -- 172.21.15.81:0/674608950 wait complete. 2023-11-15T06:31:48.373 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:31:48.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:48 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:31:47.833817+0000 mgr.smithi062.rwurtk (mgr.14164) 183 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:31:48.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:48 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:31:48.368207+0000 mon.smithi062 (mon.0) 250 : audit [DBG] from='client.? 172.21.15.81:0/674608950' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:31:49.816 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:31:49.817 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:31:51.125 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:51.122+0000 7f35574c2700 1 -- 172.21.15.81:0/2579793154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35500f6510 msgr2=0x7f35500f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:51.125 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:51.122+0000 7f35574c2700 1 --2- 172.21.15.81:0/2579793154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35500f6510 0x7f35500f6930 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f3540009fb0 tx=0x7f3540009b30).stop 2023-11-15T06:31:51.126 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:51.122+0000 7f35574c2700 1 -- 172.21.15.81:0/2579793154 shutdown_connections 2023-11-15T06:31:51.126 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:51.122+0000 7f35574c2700 1 --2- 172.21.15.81:0/2579793154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35500f6510 0x7f35500f6930 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:51.126 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:51.122+0000 7f35574c2700 1 -- 172.21.15.81:0/2579793154 >> 172.21.15.81:0/2579793154 conn(0x7f35500f1c70 msgr2=0x7f35500f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:31:51.126 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:51.122+0000 7f35574c2700 1 -- 172.21.15.81:0/2579793154 shutdown_connections 2023-11-15T06:31:51.126 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:51.122+0000 7f35574c2700 1 -- 172.21.15.81:0/2579793154 wait complete. 2023-11-15T06:31:51.127 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:51.123+0000 7f35574c2700 1 Processor -- start 2023-11-15T06:31:51.127 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:51.123+0000 7f35574c2700 1 -- start start 2023-11-15T06:31:51.127 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:51.123+0000 7f35574c2700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35500f6510 0x7f3550106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:31:51.127 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:51.123+0000 7f35574c2700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3550106c20 con 0x7f35500f6510 2023-11-15T06:31:51.127 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:51.123+0000 7f355525e700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35500f6510 0x7f3550106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:31:51.128 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:51.123+0000 7f355525e700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35500f6510 0x7f3550106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:44672/0 (socket says 172.21.15.81:44672) 2023-11-15T06:31:51.128 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:51.123+0000 7f355525e700 1 -- 172.21.15.81:0/2223662877 learned_addr learned my addr 172.21.15.81:0/2223662877 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:31:51.129 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:51.124+0000 7f355525e700 1 -- 172.21.15.81:0/2223662877 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3540005040 con 0x7f35500f6510 2023-11-15T06:31:51.129 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:51.124+0000 7f355525e700 1 --2- 172.21.15.81:0/2223662877 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35500f6510 0x7f3550106730 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f3540014000 tx=0x7f354000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:31:51.130 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:51.124+0000 7f3545ffb700 1 -- 172.21.15.81:0/2223662877 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35400095f0 con 0x7f35500f6510 2023-11-15T06:31:51.130 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:51.124+0000 7f35574c2700 1 -- 172.21.15.81:0/2223662877 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3550106e20 con 0x7f35500f6510 2023-11-15T06:31:51.130 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:51.124+0000 7f3545ffb700 1 -- 172.21.15.81:0/2223662877 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3540007e70 con 0x7f35500f6510 2023-11-15T06:31:51.130 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:51.124+0000 7f3545ffb700 1 -- 172.21.15.81:0/2223662877 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3540014ea0 con 0x7f35500f6510 2023-11-15T06:31:51.131 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:51.124+0000 7f35574c2700 1 -- 172.21.15.81:0/2223662877 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35501072b0 con 0x7f35500f6510 2023-11-15T06:31:51.131 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:51.125+0000 7f3545ffb700 1 -- 172.21.15.81:0/2223662877 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3540025070 con 0x7f35500f6510 2023-11-15T06:31:51.131 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:51.125+0000 7f3545ffb700 1 --2- 172.21.15.81:0/2223662877 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f353c02ff60 0x7f353c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:31:51.131 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:51.125+0000 7f3545ffb700 1 -- 172.21.15.81:0/2223662877 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3540040a00 con 0x7f35500f6510 2023-11-15T06:31:51.132 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:51.125+0000 7f35574c2700 1 -- 172.21.15.81:0/2223662877 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3550045ad0 con 0x7f35500f6510 2023-11-15T06:31:51.132 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:51.125+0000 7f3554a5d700 1 --2- 172.21.15.81:0/2223662877 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f353c02ff60 0x7f353c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:31:51.132 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:51.126+0000 7f3554a5d700 1 --2- 172.21.15.81:0/2223662877 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f353c02ff60 0x7f353c032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f354c0096a0 tx=0x7f354c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:31:51.133 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:51.130+0000 7f3545ffb700 1 -- 172.21.15.81:0/2223662877 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3540026350 con 0x7f35500f6510 2023-11-15T06:31:51.169 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:50 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:31:49.834114+0000 mgr.smithi062.rwurtk (mgr.14164) 184 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:31:51.338 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:51.336+0000 7f35574c2700 1 -- 172.21.15.81:0/2223662877 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f35500fbf60 con 0x7f35500f6510 2023-11-15T06:31:51.339 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:51.337+0000 7f3545ffb700 1 -- 172.21.15.81:0/2223662877 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3540012330 con 0x7f35500f6510 2023-11-15T06:31:51.339 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:31:51.339 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:31:51.340 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:51.338+0000 7f35574c2700 1 -- 172.21.15.81:0/2223662877 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f353c02ff60 msgr2=0x7f353c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:51.341 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:51.338+0000 7f35574c2700 1 --2- 172.21.15.81:0/2223662877 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f353c02ff60 0x7f353c032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f354c0096a0 tx=0x7f354c006b40).stop 2023-11-15T06:31:51.341 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:51.338+0000 7f35574c2700 1 -- 172.21.15.81:0/2223662877 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35500f6510 msgr2=0x7f3550106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:51.341 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:51.338+0000 7f35574c2700 1 --2- 172.21.15.81:0/2223662877 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35500f6510 0x7f3550106730 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f3540014000 tx=0x7f354000b040).stop 2023-11-15T06:31:51.341 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:51.338+0000 7f35574c2700 1 -- 172.21.15.81:0/2223662877 shutdown_connections 2023-11-15T06:31:51.341 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:51.338+0000 7f35574c2700 1 --2- 172.21.15.81:0/2223662877 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f353c02ff60 0x7f353c032420 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:51.342 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:51.338+0000 7f35574c2700 1 --2- 172.21.15.81:0/2223662877 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35500f6510 0x7f3550106730 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:51.342 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:51.338+0000 7f35574c2700 1 -- 172.21.15.81:0/2223662877 >> 172.21.15.81:0/2223662877 conn(0x7f35500f1c70 msgr2=0x7f35500f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:31:51.342 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:51.338+0000 7f35574c2700 1 -- 172.21.15.81:0/2223662877 shutdown_connections 2023-11-15T06:31:51.342 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:51.338+0000 7f35574c2700 1 -- 172.21.15.81:0/2223662877 wait complete. 2023-11-15T06:31:51.343 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:31:52.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:51 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:31:51.337645+0000 mon.smithi062 (mon.0) 251 : audit [DBG] from='client.? 172.21.15.81:0/2223662877' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:31:52.685 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:31:52.686 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:31:53.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:52 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:31:51.834497+0000 mgr.smithi062.rwurtk (mgr.14164) 185 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:31:53.919 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:53.914+0000 7f7428949700 1 -- 172.21.15.81:0/1082625646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74240f5410 msgr2=0x7f74240f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:53.919 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:53.914+0000 7f7428949700 1 --2- 172.21.15.81:0/1082625646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74240f5410 0x7f74240f5830 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f740c009fb0 tx=0x7f740c009b30).stop 2023-11-15T06:31:53.920 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:53.914+0000 7f7428949700 1 -- 172.21.15.81:0/1082625646 shutdown_connections 2023-11-15T06:31:53.920 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:53.914+0000 7f7428949700 1 --2- 172.21.15.81:0/1082625646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74240f5410 0x7f74240f5830 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:53.920 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:53.914+0000 7f7428949700 1 -- 172.21.15.81:0/1082625646 >> 172.21.15.81:0/1082625646 conn(0x7f74240f0b70 msgr2=0x7f74240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:31:53.920 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:53.914+0000 7f7428949700 1 -- 172.21.15.81:0/1082625646 shutdown_connections 2023-11-15T06:31:53.920 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:53.914+0000 7f7428949700 1 -- 172.21.15.81:0/1082625646 wait complete. 2023-11-15T06:31:53.920 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:53.915+0000 7f7428949700 1 Processor -- start 2023-11-15T06:31:53.921 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:53.915+0000 7f7428949700 1 -- start start 2023-11-15T06:31:53.921 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:53.915+0000 7f7428949700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74240f5410 0x7f7424109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:31:53.921 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:53.915+0000 7f7428949700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7424060220 con 0x7f74240f5410 2023-11-15T06:31:53.921 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:53.915+0000 7f742259c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74240f5410 0x7f7424109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:31:53.921 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:53.915+0000 7f742259c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74240f5410 0x7f7424109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:46506/0 (socket says 172.21.15.81:46506) 2023-11-15T06:31:53.922 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:53.915+0000 7f742259c700 1 -- 172.21.15.81:0/3232930547 learned_addr learned my addr 172.21.15.81:0/3232930547 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:31:53.922 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:53.915+0000 7f742259c700 1 -- 172.21.15.81:0/3232930547 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f740c005040 con 0x7f74240f5410 2023-11-15T06:31:53.922 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:53.915+0000 7f742259c700 1 --2- 172.21.15.81:0/3232930547 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74240f5410 0x7f7424109450 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f740c013510 tx=0x7f740c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:31:53.922 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:53.916+0000 7f741affd700 1 -- 172.21.15.81:0/3232930547 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f740c009320 con 0x7f74240f5410 2023-11-15T06:31:53.922 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:53.916+0000 7f7428949700 1 -- 172.21.15.81:0/3232930547 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7424060420 con 0x7f74240f5410 2023-11-15T06:31:53.923 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:53.916+0000 7f741affd700 1 -- 172.21.15.81:0/3232930547 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f740c013f00 con 0x7f74240f5410 2023-11-15T06:31:53.923 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:53.916+0000 7f741affd700 1 -- 172.21.15.81:0/3232930547 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f740c014d90 con 0x7f74240f5410 2023-11-15T06:31:53.923 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:53.916+0000 7f7428949700 1 -- 172.21.15.81:0/3232930547 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7424109b00 con 0x7f74240f5410 2023-11-15T06:31:53.923 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:53.916+0000 7f741affd700 1 -- 172.21.15.81:0/3232930547 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f740c0095f0 con 0x7f74240f5410 2023-11-15T06:31:53.923 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:53.916+0000 7f7428949700 1 -- 172.21.15.81:0/3232930547 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7424059100 con 0x7f74240f5410 2023-11-15T06:31:53.924 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:53.917+0000 7f741affd700 1 --2- 172.21.15.81:0/3232930547 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f741002ff60 0x7f7410032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:31:53.924 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:53.917+0000 7f741affd700 1 -- 172.21.15.81:0/3232930547 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f740c041000 con 0x7f74240f5410 2023-11-15T06:31:53.924 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:53.917+0000 7f7421d9b700 1 --2- 172.21.15.81:0/3232930547 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f741002ff60 0x7f7410032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:31:53.925 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:53.917+0000 7f7421d9b700 1 --2- 172.21.15.81:0/3232930547 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f741002ff60 0x7f7410032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f74140096a0 tx=0x7f7414006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:31:53.926 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:53.921+0000 7f741affd700 1 -- 172.21.15.81:0/3232930547 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f740c014420 con 0x7f74240f5410 2023-11-15T06:31:54.131 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:54.129+0000 7f7428949700 1 -- 172.21.15.81:0/3232930547 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7424045ad0 con 0x7f74240f5410 2023-11-15T06:31:54.132 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:54.130+0000 7f741affd700 1 -- 172.21.15.81:0/3232930547 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f740c010230 con 0x7f74240f5410 2023-11-15T06:31:54.132 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:31:54.133 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:31:54.134 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:54.131+0000 7f7428949700 1 -- 172.21.15.81:0/3232930547 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f741002ff60 msgr2=0x7f7410032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:54.134 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:54.131+0000 7f7428949700 1 --2- 172.21.15.81:0/3232930547 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f741002ff60 0x7f7410032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f74140096a0 tx=0x7f7414006b40).stop 2023-11-15T06:31:54.134 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:54.132+0000 7f7428949700 1 -- 172.21.15.81:0/3232930547 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74240f5410 msgr2=0x7f7424109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:54.134 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:54.132+0000 7f7428949700 1 --2- 172.21.15.81:0/3232930547 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74240f5410 0x7f7424109450 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f740c013510 tx=0x7f740c00b040).stop 2023-11-15T06:31:54.134 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:54.132+0000 7f7428949700 1 -- 172.21.15.81:0/3232930547 shutdown_connections 2023-11-15T06:31:54.135 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:54.132+0000 7f7428949700 1 --2- 172.21.15.81:0/3232930547 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f741002ff60 0x7f7410032420 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:54.135 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:54.132+0000 7f7428949700 1 --2- 172.21.15.81:0/3232930547 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74240f5410 0x7f7424109450 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:54.135 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:54.132+0000 7f7428949700 1 -- 172.21.15.81:0/3232930547 >> 172.21.15.81:0/3232930547 conn(0x7f74240f0b70 msgr2=0x7f74240f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:31:54.135 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:54.132+0000 7f7428949700 1 -- 172.21.15.81:0/3232930547 shutdown_connections 2023-11-15T06:31:54.135 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:54.132+0000 7f7428949700 1 -- 172.21.15.81:0/3232930547 wait complete. 2023-11-15T06:31:54.136 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:31:55.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:54 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:31:53.834848+0000 mgr.smithi062.rwurtk (mgr.14164) 186 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:31:55.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:54 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:31:53.922289+0000 mon.smithi062 (mon.0) 252 : audit [INF] from='mgr.14164 172.21.15.62:0/147479783' entity='mgr.smithi062.rwurtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi062.rwurtk/mirror_snapshot_schedule"}]: dispatch 2023-11-15T06:31:55.169 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:54 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:31:53.922530+0000 mon.smithi062 (mon.0) 253 : audit [INF] from='mgr.14164 172.21.15.62:0/147479783' entity='mgr.smithi062.rwurtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi062.rwurtk/trash_purge_schedule"}]: dispatch 2023-11-15T06:31:55.169 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:54 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:31:54.130782+0000 mon.smithi062 (mon.0) 254 : audit [DBG] from='client.? 172.21.15.81:0/3232930547' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:31:55.560 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:31:55.560 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:31:56.838 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:56.836+0000 7fb8e14ff700 1 -- 172.21.15.81:0/1493105806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8dc060160 msgr2=0x7fb8dc0fb3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:56.839 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:56.836+0000 7fb8e14ff700 1 --2- 172.21.15.81:0/1493105806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8dc060160 0x7fb8dc0fb3a0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7fb8c4009fb0 tx=0x7fb8c4009b30).stop 2023-11-15T06:31:56.839 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:56.836+0000 7fb8e14ff700 1 -- 172.21.15.81:0/1493105806 shutdown_connections 2023-11-15T06:31:56.839 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:56.836+0000 7fb8e14ff700 1 --2- 172.21.15.81:0/1493105806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8dc060160 0x7fb8dc0fb3a0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:56.839 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:56.836+0000 7fb8e14ff700 1 -- 172.21.15.81:0/1493105806 >> 172.21.15.81:0/1493105806 conn(0x7fb8dc0f09a0 msgr2=0x7fb8dc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:31:56.840 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:56.836+0000 7fb8e14ff700 1 -- 172.21.15.81:0/1493105806 shutdown_connections 2023-11-15T06:31:56.840 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:56.836+0000 7fb8e14ff700 1 -- 172.21.15.81:0/1493105806 wait complete. 2023-11-15T06:31:56.840 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:56.836+0000 7fb8e14ff700 1 Processor -- start 2023-11-15T06:31:56.840 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:56.836+0000 7fb8e14ff700 1 -- start start 2023-11-15T06:31:56.840 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:56.837+0000 7fb8e14ff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8dc060160 0x7fb8dc1086f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:31:56.841 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:56.837+0000 7fb8e14ff700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8dc108be0 con 0x7fb8dc060160 2023-11-15T06:31:56.842 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:56.837+0000 7fb8dbfff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8dc060160 0x7fb8dc1086f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:31:56.843 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:56.837+0000 7fb8dbfff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8dc060160 0x7fb8dc1086f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:46534/0 (socket says 172.21.15.81:46534) 2023-11-15T06:31:56.843 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:56.837+0000 7fb8dbfff700 1 -- 172.21.15.81:0/775576188 learned_addr learned my addr 172.21.15.81:0/775576188 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:31:56.843 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:56.837+0000 7fb8dbfff700 1 -- 172.21.15.81:0/775576188 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c4005040 con 0x7fb8dc060160 2023-11-15T06:31:56.843 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:56.837+0000 7fb8dbfff700 1 --2- 172.21.15.81:0/775576188 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8dc060160 0x7fb8dc1086f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7fb8c4013510 tx=0x7fb8c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:31:56.844 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:56.838+0000 7fb8d8ff9700 1 -- 172.21.15.81:0/775576188 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8c4009320 con 0x7fb8dc060160 2023-11-15T06:31:56.844 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:56.838+0000 7fb8d8ff9700 1 -- 172.21.15.81:0/775576188 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb8c4013f00 con 0x7fb8dc060160 2023-11-15T06:31:56.844 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:56.838+0000 7fb8e14ff700 1 -- 172.21.15.81:0/775576188 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8dc108de0 con 0x7fb8dc060160 2023-11-15T06:31:56.844 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:56.838+0000 7fb8d8ff9700 1 -- 172.21.15.81:0/775576188 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8c4014d90 con 0x7fb8dc060160 2023-11-15T06:31:56.845 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:56.838+0000 7fb8e14ff700 1 -- 172.21.15.81:0/775576188 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8dc109270 con 0x7fb8dc060160 2023-11-15T06:31:56.845 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:56.838+0000 7fb8d8ff9700 1 -- 172.21.15.81:0/775576188 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb8c40095f0 con 0x7fb8dc060160 2023-11-15T06:31:56.845 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:56.838+0000 7fb8e14ff700 1 -- 172.21.15.81:0/775576188 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8dc045ad0 con 0x7fb8dc060160 2023-11-15T06:31:56.845 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:56.839+0000 7fb8d8ff9700 1 --2- 172.21.15.81:0/775576188 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fb8c802ff60 0x7fb8c8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:31:56.845 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:56.839+0000 7fb8d8ff9700 1 -- 172.21.15.81:0/775576188 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb8c4041000 con 0x7fb8dc060160 2023-11-15T06:31:56.846 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:56.839+0000 7fb8db7fe700 1 --2- 172.21.15.81:0/775576188 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fb8c802ff60 0x7fb8c8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:31:56.846 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:56.839+0000 7fb8db7fe700 1 --2- 172.21.15.81:0/775576188 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fb8c802ff60 0x7fb8c8032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fb8cc0096a0 tx=0x7fb8cc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:31:56.846 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:56.843+0000 7fb8d8ff9700 1 -- 172.21.15.81:0/775576188 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb8c401d7f0 con 0x7fb8dc060160 2023-11-15T06:31:57.048 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:57.046+0000 7fb8e14ff700 1 -- 172.21.15.81:0/775576188 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb8dc10a120 con 0x7fb8dc060160 2023-11-15T06:31:57.049 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:57.047+0000 7fb8d8ff9700 1 -- 172.21.15.81:0/775576188 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb8c4010230 con 0x7fb8dc060160 2023-11-15T06:31:57.049 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:31:57.049 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:31:57.050 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:57.048+0000 7fb8e14ff700 1 -- 172.21.15.81:0/775576188 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fb8c802ff60 msgr2=0x7fb8c8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:57.050 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:57.048+0000 7fb8e14ff700 1 --2- 172.21.15.81:0/775576188 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fb8c802ff60 0x7fb8c8032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fb8cc0096a0 tx=0x7fb8cc006b40).stop 2023-11-15T06:31:57.051 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:57.048+0000 7fb8e14ff700 1 -- 172.21.15.81:0/775576188 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8dc060160 msgr2=0x7fb8dc1086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:57.051 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:57.048+0000 7fb8e14ff700 1 --2- 172.21.15.81:0/775576188 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8dc060160 0x7fb8dc1086f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7fb8c4013510 tx=0x7fb8c400b040).stop 2023-11-15T06:31:57.051 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:57.048+0000 7fb8e14ff700 1 -- 172.21.15.81:0/775576188 shutdown_connections 2023-11-15T06:31:57.051 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:57.048+0000 7fb8e14ff700 1 --2- 172.21.15.81:0/775576188 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fb8c802ff60 0x7fb8c8032420 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:57.052 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:57.048+0000 7fb8e14ff700 1 --2- 172.21.15.81:0/775576188 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8dc060160 0x7fb8dc1086f0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:57.052 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:57.048+0000 7fb8e14ff700 1 -- 172.21.15.81:0/775576188 >> 172.21.15.81:0/775576188 conn(0x7fb8dc0f09a0 msgr2=0x7fb8dc0f14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:31:57.052 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:57.048+0000 7fb8e14ff700 1 -- 172.21.15.81:0/775576188 shutdown_connections 2023-11-15T06:31:57.052 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:57.048+0000 7fb8e14ff700 1 -- 172.21.15.81:0/775576188 wait complete. 2023-11-15T06:31:57.053 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:31:57.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:56 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:31:55.835147+0000 mgr.smithi062.rwurtk (mgr.14164) 187 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:31:58.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:57 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:31:57.047686+0000 mon.smithi062 (mon.0) 255 : audit [DBG] from='client.? 172.21.15.81:0/775576188' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:31:58.444 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:31:58.445 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:31:59.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:31:58 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:31:57.835434+0000 mgr.smithi062.rwurtk (mgr.14164) 188 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:31:59.692 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:59.689+0000 7f8691a47700 1 -- 172.21.15.81:0/3958201689 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f868c0f5410 msgr2=0x7f868c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:59.692 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:59.689+0000 7f8691a47700 1 --2- 172.21.15.81:0/3958201689 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f868c0f5410 0x7f868c0f5830 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f8680009fb0 tx=0x7f8680009b30).stop 2023-11-15T06:31:59.693 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:59.689+0000 7f8691a47700 1 -- 172.21.15.81:0/3958201689 shutdown_connections 2023-11-15T06:31:59.693 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:59.689+0000 7f8691a47700 1 --2- 172.21.15.81:0/3958201689 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f868c0f5410 0x7f868c0f5830 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:59.693 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:59.689+0000 7f8691a47700 1 -- 172.21.15.81:0/3958201689 >> 172.21.15.81:0/3958201689 conn(0x7f868c0f0b70 msgr2=0x7f868c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:31:59.693 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:59.689+0000 7f8691a47700 1 -- 172.21.15.81:0/3958201689 shutdown_connections 2023-11-15T06:31:59.693 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:59.689+0000 7f8691a47700 1 -- 172.21.15.81:0/3958201689 wait complete. 2023-11-15T06:31:59.694 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:59.690+0000 7f8691a47700 1 Processor -- start 2023-11-15T06:31:59.694 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:59.690+0000 7f8691a47700 1 -- start start 2023-11-15T06:31:59.694 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:59.690+0000 7f8691a47700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f868c0601e0 0x7f868c109580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:31:59.694 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:59.690+0000 7f8691a47700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f868c109a70 con 0x7f868c0601e0 2023-11-15T06:31:59.696 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:59.690+0000 7f868affd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f868c0601e0 0x7f868c109580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:31:59.697 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:59.690+0000 7f868affd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f868c0601e0 0x7f868c109580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:46546/0 (socket says 172.21.15.81:46546) 2023-11-15T06:31:59.697 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:59.690+0000 7f868affd700 1 -- 172.21.15.81:0/420240819 learned_addr learned my addr 172.21.15.81:0/420240819 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:31:59.697 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:59.691+0000 7f868affd700 1 -- 172.21.15.81:0/420240819 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8680005040 con 0x7f868c0601e0 2023-11-15T06:31:59.697 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:59.691+0000 7f868affd700 1 --2- 172.21.15.81:0/420240819 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f868c0601e0 0x7f868c109580 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f8680009f80 tx=0x7f8680009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:31:59.697 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:59.691+0000 7f86737fe700 1 -- 172.21.15.81:0/420240819 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86800136c0 con 0x7f868c0601e0 2023-11-15T06:31:59.698 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:59.691+0000 7f8691a47700 1 -- 172.21.15.81:0/420240819 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f868c109bb0 con 0x7f868c0601e0 2023-11-15T06:31:59.698 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:59.691+0000 7f86737fe700 1 -- 172.21.15.81:0/420240819 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8680013e70 con 0x7f868c0601e0 2023-11-15T06:31:59.698 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:59.691+0000 7f86737fe700 1 -- 172.21.15.81:0/420240819 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8680014420 con 0x7f868c0601e0 2023-11-15T06:31:59.698 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:59.691+0000 7f8691a47700 1 -- 172.21.15.81:0/420240819 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f868c109fe0 con 0x7f868c0601e0 2023-11-15T06:31:59.698 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:59.692+0000 7f86737fe700 1 -- 172.21.15.81:0/420240819 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8680025060 con 0x7f868c0601e0 2023-11-15T06:31:59.699 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:59.692+0000 7f8691a47700 1 -- 172.21.15.81:0/420240819 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f868c059100 con 0x7f868c0601e0 2023-11-15T06:31:59.699 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:59.692+0000 7f86737fe700 1 --2- 172.21.15.81:0/420240819 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f867402ff60 0x7f8674032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:31:59.699 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:59.692+0000 7f86737fe700 1 -- 172.21.15.81:0/420240819 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8680040e50 con 0x7f868c0601e0 2023-11-15T06:31:59.699 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:59.692+0000 7f868a7fc700 1 --2- 172.21.15.81:0/420240819 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f867402ff60 0x7f8674032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:31:59.699 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:59.693+0000 7f868a7fc700 1 --2- 172.21.15.81:0/420240819 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f867402ff60 0x7f8674032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f867c0096a0 tx=0x7f867c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:31:59.700 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:59.697+0000 7f86737fe700 1 -- 172.21.15.81:0/420240819 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f868001d3a0 con 0x7f868c0601e0 2023-11-15T06:31:59.904 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:59.902+0000 7f8691a47700 1 -- 172.21.15.81:0/420240819 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f868c045ad0 con 0x7f868c0601e0 2023-11-15T06:31:59.905 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:59.903+0000 7f86737fe700 1 -- 172.21.15.81:0/420240819 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f868002d2d0 con 0x7f868c0601e0 2023-11-15T06:31:59.905 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:31:59.905 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:31:59.907 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:59.904+0000 7f8691a47700 1 -- 172.21.15.81:0/420240819 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f867402ff60 msgr2=0x7f8674032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:59.907 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:59.904+0000 7f8691a47700 1 --2- 172.21.15.81:0/420240819 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f867402ff60 0x7f8674032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f867c0096a0 tx=0x7f867c006b40).stop 2023-11-15T06:31:59.907 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:59.904+0000 7f8691a47700 1 -- 172.21.15.81:0/420240819 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f868c0601e0 msgr2=0x7f868c109580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:31:59.907 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:59.904+0000 7f8691a47700 1 --2- 172.21.15.81:0/420240819 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f868c0601e0 0x7f868c109580 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f8680009f80 tx=0x7f8680009b30).stop 2023-11-15T06:31:59.907 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:59.904+0000 7f8691a47700 1 -- 172.21.15.81:0/420240819 shutdown_connections 2023-11-15T06:31:59.908 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:59.904+0000 7f8691a47700 1 --2- 172.21.15.81:0/420240819 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f867402ff60 0x7f8674032420 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:59.908 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:59.904+0000 7f8691a47700 1 --2- 172.21.15.81:0/420240819 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f868c0601e0 0x7f868c109580 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:31:59.908 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:59.904+0000 7f8691a47700 1 -- 172.21.15.81:0/420240819 >> 172.21.15.81:0/420240819 conn(0x7f868c0f0b70 msgr2=0x7f868c0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:31:59.908 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:59.904+0000 7f8691a47700 1 -- 172.21.15.81:0/420240819 shutdown_connections 2023-11-15T06:31:59.908 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:31:59.905+0000 7f8691a47700 1 -- 172.21.15.81:0/420240819 wait complete. 2023-11-15T06:31:59.909 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:32:01.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:00 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:31:59.835745+0000 mgr.smithi062.rwurtk (mgr.14164) 189 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:32:01.169 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:00 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:31:59.903680+0000 mon.smithi062 (mon.0) 256 : audit [DBG] from='client.? 172.21.15.81:0/420240819' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:32:01.339 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:32:01.339 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:32:02.624 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:02.621+0000 7f42ad89b700 1 -- 172.21.15.81:0/129515246 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42a80f5410 msgr2=0x7f42a80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:02.624 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:02.621+0000 7f42ad89b700 1 --2- 172.21.15.81:0/129515246 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42a80f5410 0x7f42a80f5830 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f429c009fb0 tx=0x7f429c009b30).stop 2023-11-15T06:32:02.624 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:02.621+0000 7f42ad89b700 1 -- 172.21.15.81:0/129515246 shutdown_connections 2023-11-15T06:32:02.624 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:02.621+0000 7f42ad89b700 1 --2- 172.21.15.81:0/129515246 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42a80f5410 0x7f42a80f5830 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:02.625 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:02.621+0000 7f42ad89b700 1 -- 172.21.15.81:0/129515246 >> 172.21.15.81:0/129515246 conn(0x7f42a80f0b70 msgr2=0x7f42a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:32:02.625 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:02.621+0000 7f42ad89b700 1 -- 172.21.15.81:0/129515246 shutdown_connections 2023-11-15T06:32:02.625 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:02.621+0000 7f42ad89b700 1 -- 172.21.15.81:0/129515246 wait complete. 2023-11-15T06:32:02.628 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:02.622+0000 7f42ad89b700 1 Processor -- start 2023-11-15T06:32:02.629 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:02.622+0000 7f42ad89b700 1 -- start start 2023-11-15T06:32:02.629 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:02.623+0000 7f42ad89b700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42a80f5410 0x7f42a8105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:32:02.629 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:02.623+0000 7f42ad89b700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a8105af0 con 0x7f42a80f5410 2023-11-15T06:32:02.630 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:02.623+0000 7f42a6ffd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42a80f5410 0x7f42a8105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:32:02.630 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:02.623+0000 7f42a6ffd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42a80f5410 0x7f42a8105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:45880/0 (socket says 172.21.15.81:45880) 2023-11-15T06:32:02.630 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:02.623+0000 7f42a6ffd700 1 -- 172.21.15.81:0/3493839440 learned_addr learned my addr 172.21.15.81:0/3493839440 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:32:02.630 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:02.623+0000 7f42a6ffd700 1 -- 172.21.15.81:0/3493839440 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f429c005040 con 0x7f42a80f5410 2023-11-15T06:32:02.631 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:02.623+0000 7f42a6ffd700 1 --2- 172.21.15.81:0/3493839440 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42a80f5410 0x7f42a8105600 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f429c013510 tx=0x7f429c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:32:02.631 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:02.624+0000 7f428f7fe700 1 -- 172.21.15.81:0/3493839440 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f429c009320 con 0x7f42a80f5410 2023-11-15T06:32:02.631 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:02.624+0000 7f42ad89b700 1 -- 172.21.15.81:0/3493839440 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42a8105cf0 con 0x7f42a80f5410 2023-11-15T06:32:02.631 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:02.624+0000 7f428f7fe700 1 -- 172.21.15.81:0/3493839440 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f429c013f00 con 0x7f42a80f5410 2023-11-15T06:32:02.631 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:02.624+0000 7f428f7fe700 1 -- 172.21.15.81:0/3493839440 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f429c014d90 con 0x7f42a80f5410 2023-11-15T06:32:02.632 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:02.624+0000 7f42ad89b700 1 -- 172.21.15.81:0/3493839440 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42a8106180 con 0x7f42a80f5410 2023-11-15T06:32:02.632 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:02.624+0000 7f428f7fe700 1 -- 172.21.15.81:0/3493839440 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f429c0095f0 con 0x7f42a80f5410 2023-11-15T06:32:02.632 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:02.624+0000 7f42ad89b700 1 -- 172.21.15.81:0/3493839440 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42a8045ad0 con 0x7f42a80f5410 2023-11-15T06:32:02.632 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:02.625+0000 7f428f7fe700 1 --2- 172.21.15.81:0/3493839440 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f429002ff60 0x7f4290032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:32:02.632 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:02.625+0000 7f428f7fe700 1 -- 172.21.15.81:0/3493839440 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f429c0410a0 con 0x7f42a80f5410 2023-11-15T06:32:02.634 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:02.625+0000 7f42a67fc700 1 --2- 172.21.15.81:0/3493839440 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f429002ff60 0x7f4290032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:32:02.634 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:02.625+0000 7f42a67fc700 1 --2- 172.21.15.81:0/3493839440 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f429002ff60 0x7f4290032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f42980096a0 tx=0x7f4298006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:32:02.635 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:02.629+0000 7f428f7fe700 1 -- 172.21.15.81:0/3493839440 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f429c01d7f0 con 0x7f42a80f5410 2023-11-15T06:32:02.834 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:02.832+0000 7f42ad89b700 1 -- 172.21.15.81:0/3493839440 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f42a80fae60 con 0x7f42a80f5410 2023-11-15T06:32:02.835 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:02.833+0000 7f428f7fe700 1 -- 172.21.15.81:0/3493839440 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f429c010230 con 0x7f42a80f5410 2023-11-15T06:32:02.835 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:32:02.835 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:32:02.836 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:02.834+0000 7f42ad89b700 1 -- 172.21.15.81:0/3493839440 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f429002ff60 msgr2=0x7f4290032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:02.837 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:02.834+0000 7f42ad89b700 1 --2- 172.21.15.81:0/3493839440 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f429002ff60 0x7f4290032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f42980096a0 tx=0x7f4298006b40).stop 2023-11-15T06:32:02.837 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:02.834+0000 7f42ad89b700 1 -- 172.21.15.81:0/3493839440 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42a80f5410 msgr2=0x7f42a8105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:02.837 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:02.834+0000 7f42ad89b700 1 --2- 172.21.15.81:0/3493839440 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42a80f5410 0x7f42a8105600 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f429c013510 tx=0x7f429c00b040).stop 2023-11-15T06:32:02.837 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:02.834+0000 7f42ad89b700 1 -- 172.21.15.81:0/3493839440 shutdown_connections 2023-11-15T06:32:02.838 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:02.834+0000 7f42ad89b700 1 --2- 172.21.15.81:0/3493839440 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f429002ff60 0x7f4290032420 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:02.838 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:02.834+0000 7f42ad89b700 1 --2- 172.21.15.81:0/3493839440 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42a80f5410 0x7f42a8105600 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:02.838 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:02.834+0000 7f42ad89b700 1 -- 172.21.15.81:0/3493839440 >> 172.21.15.81:0/3493839440 conn(0x7f42a80f0b70 msgr2=0x7f42a80f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:32:02.838 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:02.834+0000 7f42ad89b700 1 -- 172.21.15.81:0/3493839440 shutdown_connections 2023-11-15T06:32:02.838 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:02.834+0000 7f42ad89b700 1 -- 172.21.15.81:0/3493839440 wait complete. 2023-11-15T06:32:02.839 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:32:03.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:32:01.836065+0000 mgr.smithi062.rwurtk (mgr.14164) 190 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:32:03.169 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:32:02.833598+0000 mon.smithi062 (mon.0) 257 : audit [DBG] from='client.? 172.21.15.81:0/3493839440' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:32:04.235 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:32:04.236 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:32:05.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:04 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:32:03.836356+0000 mgr.smithi062.rwurtk (mgr.14164) 191 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:32:05.451 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:05.445+0000 7f5cb66f4700 1 -- 172.21.15.81:0/116596812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5cb00f5410 msgr2=0x7f5cb00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:05.451 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:05.445+0000 7f5cb66f4700 1 --2- 172.21.15.81:0/116596812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5cb00f5410 0x7f5cb00f5830 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f5ca0009fb0 tx=0x7f5ca0009b30).stop 2023-11-15T06:32:05.451 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:05.445+0000 7f5cb66f4700 1 -- 172.21.15.81:0/116596812 shutdown_connections 2023-11-15T06:32:05.451 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:05.445+0000 7f5cb66f4700 1 --2- 172.21.15.81:0/116596812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5cb00f5410 0x7f5cb00f5830 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:05.452 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:05.445+0000 7f5cb66f4700 1 -- 172.21.15.81:0/116596812 >> 172.21.15.81:0/116596812 conn(0x7f5cb00f0b70 msgr2=0x7f5cb00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:32:05.452 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:05.445+0000 7f5cb66f4700 1 -- 172.21.15.81:0/116596812 shutdown_connections 2023-11-15T06:32:05.452 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:05.445+0000 7f5cb66f4700 1 -- 172.21.15.81:0/116596812 wait complete. 2023-11-15T06:32:05.452 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:05.446+0000 7f5cb66f4700 1 Processor -- start 2023-11-15T06:32:05.452 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:05.446+0000 7f5cb66f4700 1 -- start start 2023-11-15T06:32:05.453 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:05.446+0000 7f5cb66f4700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5cb00f5410 0x7f5cb0109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:32:05.453 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:05.446+0000 7f5cb66f4700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb0060220 con 0x7f5cb00f5410 2023-11-15T06:32:05.453 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:05.446+0000 7f5caffff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5cb00f5410 0x7f5cb0109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:32:05.453 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:05.446+0000 7f5caffff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5cb00f5410 0x7f5cb0109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:45900/0 (socket says 172.21.15.81:45900) 2023-11-15T06:32:05.453 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:05.446+0000 7f5caffff700 1 -- 172.21.15.81:0/480397759 learned_addr learned my addr 172.21.15.81:0/480397759 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:32:05.454 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:05.447+0000 7f5caffff700 1 -- 172.21.15.81:0/480397759 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca0005040 con 0x7f5cb00f5410 2023-11-15T06:32:05.454 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:05.447+0000 7f5caffff700 1 --2- 172.21.15.81:0/480397759 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5cb00f5410 0x7f5cb0109450 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f5ca0013510 tx=0x7f5ca000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:32:05.454 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:05.447+0000 7f5cacff9700 1 -- 172.21.15.81:0/480397759 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ca0009320 con 0x7f5cb00f5410 2023-11-15T06:32:05.454 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:05.447+0000 7f5cb66f4700 1 -- 172.21.15.81:0/480397759 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cb0060420 con 0x7f5cb00f5410 2023-11-15T06:32:05.454 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:05.447+0000 7f5cacff9700 1 -- 172.21.15.81:0/480397759 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5ca0013f00 con 0x7f5cb00f5410 2023-11-15T06:32:05.455 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:05.447+0000 7f5cacff9700 1 -- 172.21.15.81:0/480397759 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ca0014d90 con 0x7f5cb00f5410 2023-11-15T06:32:05.455 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:05.447+0000 7f5cb66f4700 1 -- 172.21.15.81:0/480397759 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cb0109b00 con 0x7f5cb00f5410 2023-11-15T06:32:05.455 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:05.448+0000 7f5cacff9700 1 -- 172.21.15.81:0/480397759 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5ca00095f0 con 0x7f5cb00f5410 2023-11-15T06:32:05.455 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:05.448+0000 7f5cb66f4700 1 -- 172.21.15.81:0/480397759 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cb0059100 con 0x7f5cb00f5410 2023-11-15T06:32:05.456 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:05.448+0000 7f5cacff9700 1 --2- 172.21.15.81:0/480397759 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f5c9c02ff60 0x7f5c9c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:32:05.456 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:05.448+0000 7f5cacff9700 1 -- 172.21.15.81:0/480397759 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5ca0041020 con 0x7f5cb00f5410 2023-11-15T06:32:05.456 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:05.448+0000 7f5caf7fe700 1 --2- 172.21.15.81:0/480397759 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f5c9c02ff60 0x7f5c9c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:32:05.456 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:05.449+0000 7f5caf7fe700 1 --2- 172.21.15.81:0/480397759 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f5c9c02ff60 0x7f5c9c032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f5c980096a0 tx=0x7f5c98006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:32:05.458 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:05.453+0000 7f5cacff9700 1 -- 172.21.15.81:0/480397759 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5ca0014420 con 0x7f5cb00f5410 2023-11-15T06:32:05.658 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:05.656+0000 7f5cb66f4700 1 -- 172.21.15.81:0/480397759 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5cb0045ad0 con 0x7f5cb00f5410 2023-11-15T06:32:05.659 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:05.657+0000 7f5cacff9700 1 -- 172.21.15.81:0/480397759 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5ca001d7f0 con 0x7f5cb00f5410 2023-11-15T06:32:05.659 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:32:05.660 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:32:05.660 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:05.658+0000 7f5cb66f4700 1 -- 172.21.15.81:0/480397759 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f5c9c02ff60 msgr2=0x7f5c9c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:05.661 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:05.658+0000 7f5cb66f4700 1 --2- 172.21.15.81:0/480397759 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f5c9c02ff60 0x7f5c9c032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f5c980096a0 tx=0x7f5c98006b40).stop 2023-11-15T06:32:05.661 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:05.658+0000 7f5cb66f4700 1 -- 172.21.15.81:0/480397759 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5cb00f5410 msgr2=0x7f5cb0109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:05.661 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:05.658+0000 7f5cb66f4700 1 --2- 172.21.15.81:0/480397759 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5cb00f5410 0x7f5cb0109450 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f5ca0013510 tx=0x7f5ca000b040).stop 2023-11-15T06:32:05.662 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:05.658+0000 7f5cb66f4700 1 -- 172.21.15.81:0/480397759 shutdown_connections 2023-11-15T06:32:05.662 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:05.658+0000 7f5cb66f4700 1 --2- 172.21.15.81:0/480397759 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f5c9c02ff60 0x7f5c9c032420 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:05.662 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:05.658+0000 7f5cb66f4700 1 --2- 172.21.15.81:0/480397759 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5cb00f5410 0x7f5cb0109450 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:05.662 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:05.658+0000 7f5cb66f4700 1 -- 172.21.15.81:0/480397759 >> 172.21.15.81:0/480397759 conn(0x7f5cb00f0b70 msgr2=0x7f5cb00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:32:05.663 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:05.659+0000 7f5cb66f4700 1 -- 172.21.15.81:0/480397759 shutdown_connections 2023-11-15T06:32:05.663 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:05.659+0000 7f5cb66f4700 1 -- 172.21.15.81:0/480397759 wait complete. 2023-11-15T06:32:05.664 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:32:06.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:05 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:32:05.657760+0000 mon.smithi062 (mon.0) 258 : audit [DBG] from='client.? 172.21.15.81:0/480397759' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:32:07.079 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:32:07.080 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:32:07.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:06 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:32:05.836722+0000 mgr.smithi062.rwurtk (mgr.14164) 192 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:32:08.379 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:08.377+0000 7f1ae97e5700 1 -- 172.21.15.81:0/2330480606 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ae40f6510 msgr2=0x7f1ae40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:08.379 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:08.377+0000 7f1ae97e5700 1 --2- 172.21.15.81:0/2330480606 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ae40f6510 0x7f1ae40f6930 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f1ad4009fb0 tx=0x7f1ad4009b30).stop 2023-11-15T06:32:08.379 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:08.377+0000 7f1ae97e5700 1 -- 172.21.15.81:0/2330480606 shutdown_connections 2023-11-15T06:32:08.379 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:08.377+0000 7f1ae97e5700 1 --2- 172.21.15.81:0/2330480606 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ae40f6510 0x7f1ae40f6930 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:08.380 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:08.377+0000 7f1ae97e5700 1 -- 172.21.15.81:0/2330480606 >> 172.21.15.81:0/2330480606 conn(0x7f1ae40f1c70 msgr2=0x7f1ae40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:32:08.380 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:08.377+0000 7f1ae97e5700 1 -- 172.21.15.81:0/2330480606 shutdown_connections 2023-11-15T06:32:08.380 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:08.377+0000 7f1ae97e5700 1 -- 172.21.15.81:0/2330480606 wait complete. 2023-11-15T06:32:08.382 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:08.377+0000 7f1ae97e5700 1 Processor -- start 2023-11-15T06:32:08.382 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:08.377+0000 7f1ae97e5700 1 -- start start 2023-11-15T06:32:08.382 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:08.377+0000 7f1ae97e5700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ae40f6510 0x7f1ae4108920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:32:08.383 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:08.377+0000 7f1ae97e5700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae4108e10 con 0x7f1ae40f6510 2023-11-15T06:32:08.383 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:08.378+0000 7f1ae2ffd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ae40f6510 0x7f1ae4108920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:32:08.383 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:08.378+0000 7f1ae2ffd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ae40f6510 0x7f1ae4108920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:45918/0 (socket says 172.21.15.81:45918) 2023-11-15T06:32:08.383 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:08.378+0000 7f1ae2ffd700 1 -- 172.21.15.81:0/3408983788 learned_addr learned my addr 172.21.15.81:0/3408983788 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:32:08.383 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:08.378+0000 7f1ae2ffd700 1 -- 172.21.15.81:0/3408983788 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad4005040 con 0x7f1ae40f6510 2023-11-15T06:32:08.384 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:08.378+0000 7f1ae2ffd700 1 --2- 172.21.15.81:0/3408983788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ae40f6510 0x7f1ae4108920 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f1ad4013510 tx=0x7f1ad400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:32:08.384 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:08.378+0000 7f1acb7fe700 1 -- 172.21.15.81:0/3408983788 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ad4009320 con 0x7f1ae40f6510 2023-11-15T06:32:08.384 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:08.379+0000 7f1ae97e5700 1 -- 172.21.15.81:0/3408983788 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ae4109010 con 0x7f1ae40f6510 2023-11-15T06:32:08.384 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:08.379+0000 7f1acb7fe700 1 -- 172.21.15.81:0/3408983788 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1ad4013f00 con 0x7f1ae40f6510 2023-11-15T06:32:08.384 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:08.379+0000 7f1acb7fe700 1 -- 172.21.15.81:0/3408983788 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ad4014d90 con 0x7f1ae40f6510 2023-11-15T06:32:08.385 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:08.379+0000 7f1ae97e5700 1 -- 172.21.15.81:0/3408983788 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ae41094a0 con 0x7f1ae40f6510 2023-11-15T06:32:08.385 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:08.379+0000 7f1acb7fe700 1 -- 172.21.15.81:0/3408983788 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1ad40095f0 con 0x7f1ae40f6510 2023-11-15T06:32:08.385 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:08.379+0000 7f1ae97e5700 1 -- 172.21.15.81:0/3408983788 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ae4045ad0 con 0x7f1ae40f6510 2023-11-15T06:32:08.385 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:08.380+0000 7f1acb7fe700 1 --2- 172.21.15.81:0/3408983788 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f1acc02ff60 0x7f1acc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:32:08.385 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:08.380+0000 7f1acb7fe700 1 -- 172.21.15.81:0/3408983788 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1ad4040fc0 con 0x7f1ae40f6510 2023-11-15T06:32:08.385 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:08.380+0000 7f1ae27fc700 1 --2- 172.21.15.81:0/3408983788 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f1acc02ff60 0x7f1acc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:32:08.386 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:08.380+0000 7f1ae27fc700 1 --2- 172.21.15.81:0/3408983788 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f1acc02ff60 0x7f1acc032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f1ad80096a0 tx=0x7f1ad8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:32:08.386 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:08.384+0000 7f1acb7fe700 1 -- 172.21.15.81:0/3408983788 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1ad4014420 con 0x7f1ae40f6510 2023-11-15T06:32:08.590 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:08.588+0000 7f1ae97e5700 1 -- 172.21.15.81:0/3408983788 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1ae410a2d0 con 0x7f1ae40f6510 2023-11-15T06:32:08.591 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:08.589+0000 7f1acb7fe700 1 -- 172.21.15.81:0/3408983788 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1ad4010230 con 0x7f1ae40f6510 2023-11-15T06:32:08.592 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:32:08.592 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:32:08.593 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:08.590+0000 7f1ae97e5700 1 -- 172.21.15.81:0/3408983788 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f1acc02ff60 msgr2=0x7f1acc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:08.593 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:08.590+0000 7f1ae97e5700 1 --2- 172.21.15.81:0/3408983788 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f1acc02ff60 0x7f1acc032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f1ad80096a0 tx=0x7f1ad8006b40).stop 2023-11-15T06:32:08.593 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:08.591+0000 7f1ae97e5700 1 -- 172.21.15.81:0/3408983788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ae40f6510 msgr2=0x7f1ae4108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:08.593 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:08.591+0000 7f1ae97e5700 1 --2- 172.21.15.81:0/3408983788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ae40f6510 0x7f1ae4108920 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f1ad4013510 tx=0x7f1ad400b040).stop 2023-11-15T06:32:08.594 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:08.591+0000 7f1ae97e5700 1 -- 172.21.15.81:0/3408983788 shutdown_connections 2023-11-15T06:32:08.594 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:08.591+0000 7f1ae97e5700 1 --2- 172.21.15.81:0/3408983788 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f1acc02ff60 0x7f1acc032420 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:08.594 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:08.591+0000 7f1ae97e5700 1 --2- 172.21.15.81:0/3408983788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ae40f6510 0x7f1ae4108920 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:08.594 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:08.591+0000 7f1ae97e5700 1 -- 172.21.15.81:0/3408983788 >> 172.21.15.81:0/3408983788 conn(0x7f1ae40f1c70 msgr2=0x7f1ae40f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:32:08.594 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:08.591+0000 7f1ae97e5700 1 -- 172.21.15.81:0/3408983788 shutdown_connections 2023-11-15T06:32:08.595 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:08.591+0000 7f1ae97e5700 1 -- 172.21.15.81:0/3408983788 wait complete. 2023-11-15T06:32:08.595 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:32:08.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:08 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:32:07.837058+0000 mgr.smithi062.rwurtk (mgr.14164) 193 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:32:08.919 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:08 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:32:08.589881+0000 mon.smithi062 (mon.0) 259 : audit [DBG] from='client.? 172.21.15.81:0/3408983788' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:32:09.990 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:32:09.991 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:32:11.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:10 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:32:09.837482+0000 mgr.smithi062.rwurtk (mgr.14164) 194 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:32:11.213 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:11.209+0000 7f875f41c700 1 -- 172.21.15.81:0/549382047 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87580f6510 msgr2=0x7f87580f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:11.213 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:11.209+0000 7f875f41c700 1 --2- 172.21.15.81:0/549382047 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87580f6510 0x7f87580f6930 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f8748009fb0 tx=0x7f8748009b30).stop 2023-11-15T06:32:11.213 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:11.209+0000 7f875f41c700 1 -- 172.21.15.81:0/549382047 shutdown_connections 2023-11-15T06:32:11.214 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:11.209+0000 7f875f41c700 1 --2- 172.21.15.81:0/549382047 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87580f6510 0x7f87580f6930 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:11.214 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:11.209+0000 7f875f41c700 1 -- 172.21.15.81:0/549382047 >> 172.21.15.81:0/549382047 conn(0x7f87580f1c70 msgr2=0x7f87580f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:32:11.214 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:11.209+0000 7f875f41c700 1 -- 172.21.15.81:0/549382047 shutdown_connections 2023-11-15T06:32:11.214 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:11.209+0000 7f875f41c700 1 -- 172.21.15.81:0/549382047 wait complete. 2023-11-15T06:32:11.215 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:11.210+0000 7f875f41c700 1 Processor -- start 2023-11-15T06:32:11.215 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:11.210+0000 7f875f41c700 1 -- start start 2023-11-15T06:32:11.215 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:11.210+0000 7f875f41c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87580f6510 0x7f8758106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:32:11.215 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:11.210+0000 7f875f41c700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8758106c20 con 0x7f87580f6510 2023-11-15T06:32:11.215 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:11.210+0000 7f875d1b8700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87580f6510 0x7f8758106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:32:11.216 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:11.210+0000 7f875d1b8700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87580f6510 0x7f8758106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:45934/0 (socket says 172.21.15.81:45934) 2023-11-15T06:32:11.216 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:11.210+0000 7f875d1b8700 1 -- 172.21.15.81:0/1833038962 learned_addr learned my addr 172.21.15.81:0/1833038962 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:32:11.216 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:11.211+0000 7f875d1b8700 1 -- 172.21.15.81:0/1833038962 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8748005040 con 0x7f87580f6510 2023-11-15T06:32:11.217 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:11.211+0000 7f875d1b8700 1 --2- 172.21.15.81:0/1833038962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87580f6510 0x7f8758106730 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f8748013510 tx=0x7f874800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:32:11.217 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:11.211+0000 7f874dffb700 1 -- 172.21.15.81:0/1833038962 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8748009320 con 0x7f87580f6510 2023-11-15T06:32:11.217 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:11.211+0000 7f874dffb700 1 -- 172.21.15.81:0/1833038962 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8748013f00 con 0x7f87580f6510 2023-11-15T06:32:11.217 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:11.211+0000 7f875f41c700 1 -- 172.21.15.81:0/1833038962 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8758106e20 con 0x7f87580f6510 2023-11-15T06:32:11.217 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:11.211+0000 7f874dffb700 1 -- 172.21.15.81:0/1833038962 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8748014d90 con 0x7f87580f6510 2023-11-15T06:32:11.218 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:11.211+0000 7f875f41c700 1 -- 172.21.15.81:0/1833038962 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87581072b0 con 0x7f87580f6510 2023-11-15T06:32:11.219 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:11.212+0000 7f875f41c700 1 -- 172.21.15.81:0/1833038962 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8758045ad0 con 0x7f87580f6510 2023-11-15T06:32:11.220 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:11.212+0000 7f874dffb700 1 -- 172.21.15.81:0/1833038962 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f87480095f0 con 0x7f87580f6510 2023-11-15T06:32:11.220 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:11.212+0000 7f874dffb700 1 --2- 172.21.15.81:0/1833038962 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f874402ff60 0x7f8744032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:32:11.220 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:11.212+0000 7f874dffb700 1 -- 172.21.15.81:0/1833038962 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8748041000 con 0x7f87580f6510 2023-11-15T06:32:11.220 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:11.212+0000 7f875c9b7700 1 --2- 172.21.15.81:0/1833038962 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f874402ff60 0x7f8744032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:32:11.221 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:11.213+0000 7f875c9b7700 1 --2- 172.21.15.81:0/1833038962 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f874402ff60 0x7f8744032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f87540096a0 tx=0x7f8754006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:32:11.221 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:11.217+0000 7f874dffb700 1 -- 172.21.15.81:0/1833038962 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8748014420 con 0x7f87580f6510 2023-11-15T06:32:11.427 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:11.425+0000 7f875f41c700 1 -- 172.21.15.81:0/1833038962 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f87580fbf60 con 0x7f87580f6510 2023-11-15T06:32:11.427 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:11.425+0000 7f874dffb700 1 -- 172.21.15.81:0/1833038962 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f874801d7f0 con 0x7f87580f6510 2023-11-15T06:32:11.428 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:32:11.428 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:32:11.429 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:11.427+0000 7f875f41c700 1 -- 172.21.15.81:0/1833038962 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f874402ff60 msgr2=0x7f8744032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:11.429 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:11.427+0000 7f875f41c700 1 --2- 172.21.15.81:0/1833038962 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f874402ff60 0x7f8744032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f87540096a0 tx=0x7f8754006b40).stop 2023-11-15T06:32:11.429 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:11.427+0000 7f875f41c700 1 -- 172.21.15.81:0/1833038962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87580f6510 msgr2=0x7f8758106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:11.430 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:11.427+0000 7f875f41c700 1 --2- 172.21.15.81:0/1833038962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87580f6510 0x7f8758106730 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f8748013510 tx=0x7f874800b040).stop 2023-11-15T06:32:11.430 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:11.427+0000 7f875f41c700 1 -- 172.21.15.81:0/1833038962 shutdown_connections 2023-11-15T06:32:11.430 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:11.427+0000 7f875f41c700 1 --2- 172.21.15.81:0/1833038962 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f874402ff60 0x7f8744032420 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:11.430 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:11.427+0000 7f875f41c700 1 --2- 172.21.15.81:0/1833038962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87580f6510 0x7f8758106730 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:11.430 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:11.427+0000 7f875f41c700 1 -- 172.21.15.81:0/1833038962 >> 172.21.15.81:0/1833038962 conn(0x7f87580f1c70 msgr2=0x7f87580f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:32:11.431 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:11.427+0000 7f875f41c700 1 -- 172.21.15.81:0/1833038962 shutdown_connections 2023-11-15T06:32:11.431 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:11.427+0000 7f875f41c700 1 -- 172.21.15.81:0/1833038962 wait complete. 2023-11-15T06:32:11.431 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:32:12.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:11 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:32:11.426333+0000 mon.smithi062 (mon.0) 260 : audit [DBG] from='client.? 172.21.15.81:0/1833038962' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:32:12.850 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:32:12.851 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:32:13.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:12 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:32:11.837769+0000 mgr.smithi062.rwurtk (mgr.14164) 195 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:32:14.118 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:14.116+0000 7f476a790700 1 -- 172.21.15.81:0/1509972422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47640f51a0 msgr2=0x7f47640f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:14.119 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:14.116+0000 7f476a790700 1 --2- 172.21.15.81:0/1509972422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47640f51a0 0x7f47640f55c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f4760009fb0 tx=0x7f4760009b30).stop 2023-11-15T06:32:14.119 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:14.116+0000 7f476a790700 1 -- 172.21.15.81:0/1509972422 shutdown_connections 2023-11-15T06:32:14.119 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:14.116+0000 7f476a790700 1 --2- 172.21.15.81:0/1509972422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47640f51a0 0x7f47640f55c0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:14.119 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:14.116+0000 7f476a790700 1 -- 172.21.15.81:0/1509972422 >> 172.21.15.81:0/1509972422 conn(0x7f47640f09a0 msgr2=0x7f47640f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:32:14.120 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:14.116+0000 7f476a790700 1 -- 172.21.15.81:0/1509972422 shutdown_connections 2023-11-15T06:32:14.120 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:14.116+0000 7f476a790700 1 -- 172.21.15.81:0/1509972422 wait complete. 2023-11-15T06:32:14.120 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:14.116+0000 7f476a790700 1 Processor -- start 2023-11-15T06:32:14.120 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:14.116+0000 7f476a790700 1 -- start start 2023-11-15T06:32:14.120 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:14.116+0000 7f476a790700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47640f51a0 0x7f47641064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:32:14.121 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:14.116+0000 7f476a790700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4764106990 con 0x7f47640f51a0 2023-11-15T06:32:14.122 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:14.117+0000 7f476978e700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47640f51a0 0x7f47641064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:32:14.122 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:14.117+0000 7f476978e700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47640f51a0 0x7f47641064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:33930/0 (socket says 172.21.15.81:33930) 2023-11-15T06:32:14.123 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:14.117+0000 7f476978e700 1 -- 172.21.15.81:0/2403605722 learned_addr learned my addr 172.21.15.81:0/2403605722 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:32:14.123 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:14.117+0000 7f476978e700 1 -- 172.21.15.81:0/2403605722 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4760005040 con 0x7f47640f51a0 2023-11-15T06:32:14.123 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:14.117+0000 7f476978e700 1 --2- 172.21.15.81:0/2403605722 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47640f51a0 0x7f47641064a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f4760013510 tx=0x7f476000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:32:14.123 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:14.117+0000 7f4759ffb700 1 -- 172.21.15.81:0/2403605722 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4760009320 con 0x7f47640f51a0 2023-11-15T06:32:14.123 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:14.117+0000 7f476a790700 1 -- 172.21.15.81:0/2403605722 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4764106b90 con 0x7f47640f51a0 2023-11-15T06:32:14.124 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:14.118+0000 7f4759ffb700 1 -- 172.21.15.81:0/2403605722 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4760013f00 con 0x7f47640f51a0 2023-11-15T06:32:14.124 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:14.118+0000 7f4759ffb700 1 -- 172.21.15.81:0/2403605722 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4760014dc0 con 0x7f47640f51a0 2023-11-15T06:32:14.124 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:14.118+0000 7f476a790700 1 -- 172.21.15.81:0/2403605722 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4764107020 con 0x7f47640f51a0 2023-11-15T06:32:14.124 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:14.119+0000 7f4759ffb700 1 -- 172.21.15.81:0/2403605722 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4760014420 con 0x7f47640f51a0 2023-11-15T06:32:14.124 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:14.119+0000 7f476a790700 1 -- 172.21.15.81:0/2403605722 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4764045ad0 con 0x7f47640f51a0 2023-11-15T06:32:14.125 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:14.119+0000 7f4759ffb700 1 --2- 172.21.15.81:0/2403605722 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f475002ff60 0x7f4750032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:32:14.125 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:14.119+0000 7f4759ffb700 1 -- 172.21.15.81:0/2403605722 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4760041020 con 0x7f47640f51a0 2023-11-15T06:32:14.125 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:14.119+0000 7f4768f8d700 1 --2- 172.21.15.81:0/2403605722 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f475002ff60 0x7f4750032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:32:14.125 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:14.119+0000 7f4768f8d700 1 --2- 172.21.15.81:0/2403605722 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f475002ff60 0x7f4750032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f47540096a0 tx=0x7f4754006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:32:14.126 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:14.123+0000 7f4759ffb700 1 -- 172.21.15.81:0/2403605722 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f47600095f0 con 0x7f47640f51a0 2023-11-15T06:32:14.330 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:14.328+0000 7f476a790700 1 -- 172.21.15.81:0/2403605722 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4764107e70 con 0x7f47640f51a0 2023-11-15T06:32:14.331 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:14.329+0000 7f4759ffb700 1 -- 172.21.15.81:0/2403605722 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f476001cb10 con 0x7f47640f51a0 2023-11-15T06:32:14.331 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:32:14.332 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:32:14.333 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:14.330+0000 7f476a790700 1 -- 172.21.15.81:0/2403605722 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f475002ff60 msgr2=0x7f4750032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:14.333 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:14.330+0000 7f476a790700 1 --2- 172.21.15.81:0/2403605722 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f475002ff60 0x7f4750032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f47540096a0 tx=0x7f4754006b40).stop 2023-11-15T06:32:14.333 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:14.331+0000 7f476a790700 1 -- 172.21.15.81:0/2403605722 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47640f51a0 msgr2=0x7f47641064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:14.333 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:14.331+0000 7f476a790700 1 --2- 172.21.15.81:0/2403605722 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47640f51a0 0x7f47641064a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f4760013510 tx=0x7f476000b040).stop 2023-11-15T06:32:14.334 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:14.331+0000 7f476a790700 1 -- 172.21.15.81:0/2403605722 shutdown_connections 2023-11-15T06:32:14.334 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:14.331+0000 7f476a790700 1 --2- 172.21.15.81:0/2403605722 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f475002ff60 0x7f4750032420 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:14.334 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:14.331+0000 7f476a790700 1 --2- 172.21.15.81:0/2403605722 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47640f51a0 0x7f47641064a0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:14.334 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:14.331+0000 7f476a790700 1 -- 172.21.15.81:0/2403605722 >> 172.21.15.81:0/2403605722 conn(0x7f47640f09a0 msgr2=0x7f476405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:32:14.334 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:14.331+0000 7f476a790700 1 -- 172.21.15.81:0/2403605722 shutdown_connections 2023-11-15T06:32:14.335 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:14.331+0000 7f476a790700 1 -- 172.21.15.81:0/2403605722 wait complete. 2023-11-15T06:32:14.335 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:32:15.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:14 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:32:13.838085+0000 mgr.smithi062.rwurtk (mgr.14164) 196 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:32:15.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:14 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:32:14.329895+0000 mon.smithi062 (mon.0) 261 : audit [DBG] from='client.? 172.21.15.81:0/2403605722' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:32:15.715 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:32:15.716 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:32:16.867 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:16.863+0000 7fa9e4a3f700 1 -- 172.21.15.81:0/3721189977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa9e00f5410 msgr2=0x7fa9e00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:16.867 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:16.863+0000 7fa9e4a3f700 1 --2- 172.21.15.81:0/3721189977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa9e00f5410 0x7fa9e00f5830 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7fa9c8009fb0 tx=0x7fa9c8009b30).stop 2023-11-15T06:32:16.868 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:16.864+0000 7fa9e4a3f700 1 -- 172.21.15.81:0/3721189977 shutdown_connections 2023-11-15T06:32:16.868 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:16.864+0000 7fa9e4a3f700 1 --2- 172.21.15.81:0/3721189977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa9e00f5410 0x7fa9e00f5830 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:16.868 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:16.864+0000 7fa9e4a3f700 1 -- 172.21.15.81:0/3721189977 >> 172.21.15.81:0/3721189977 conn(0x7fa9e00f0b70 msgr2=0x7fa9e00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:32:16.868 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:16.864+0000 7fa9e4a3f700 1 -- 172.21.15.81:0/3721189977 shutdown_connections 2023-11-15T06:32:16.868 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:16.864+0000 7fa9e4a3f700 1 -- 172.21.15.81:0/3721189977 wait complete. 2023-11-15T06:32:16.869 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:16.864+0000 7fa9e4a3f700 1 Processor -- start 2023-11-15T06:32:16.869 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:16.864+0000 7fa9e4a3f700 1 -- start start 2023-11-15T06:32:16.869 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:16.864+0000 7fa9e4a3f700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa9e00f5410 0x7fa9e0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:32:16.869 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:16.864+0000 7fa9e4a3f700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e0106c20 con 0x7fa9e00f5410 2023-11-15T06:32:16.869 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:16.864+0000 7fa9de59c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa9e00f5410 0x7fa9e0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:32:16.870 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:16.864+0000 7fa9de59c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa9e00f5410 0x7fa9e0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:33940/0 (socket says 172.21.15.81:33940) 2023-11-15T06:32:16.870 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:16.864+0000 7fa9de59c700 1 -- 172.21.15.81:0/924122626 learned_addr learned my addr 172.21.15.81:0/924122626 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:32:16.870 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:16.865+0000 7fa9de59c700 1 -- 172.21.15.81:0/924122626 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c8005040 con 0x7fa9e00f5410 2023-11-15T06:32:16.870 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:16.865+0000 7fa9de59c700 1 --2- 172.21.15.81:0/924122626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa9e00f5410 0x7fa9e0106730 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fa9c8013510 tx=0x7fa9c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:32:16.870 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:16.865+0000 7fa9d6ffd700 1 -- 172.21.15.81:0/924122626 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9c8009320 con 0x7fa9e00f5410 2023-11-15T06:32:16.871 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:16.865+0000 7fa9e4a3f700 1 -- 172.21.15.81:0/924122626 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9e0106e20 con 0x7fa9e00f5410 2023-11-15T06:32:16.871 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:16.865+0000 7fa9d6ffd700 1 -- 172.21.15.81:0/924122626 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa9c8013f00 con 0x7fa9e00f5410 2023-11-15T06:32:16.871 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:16.865+0000 7fa9e4a3f700 1 -- 172.21.15.81:0/924122626 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9e01072b0 con 0x7fa9e00f5410 2023-11-15T06:32:16.871 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:16.865+0000 7fa9d6ffd700 1 -- 172.21.15.81:0/924122626 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9c8014d90 con 0x7fa9e00f5410 2023-11-15T06:32:16.872 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:16.866+0000 7fa9d6ffd700 1 -- 172.21.15.81:0/924122626 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa9c80095f0 con 0x7fa9e00f5410 2023-11-15T06:32:16.873 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:16.866+0000 7fa9e4a3f700 1 -- 172.21.15.81:0/924122626 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9e0045ad0 con 0x7fa9e00f5410 2023-11-15T06:32:16.873 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:16.866+0000 7fa9d6ffd700 1 --2- 172.21.15.81:0/924122626 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fa9cc02ff60 0x7fa9cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:32:16.873 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:16.866+0000 7fa9d6ffd700 1 -- 172.21.15.81:0/924122626 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa9c8040fd0 con 0x7fa9e00f5410 2023-11-15T06:32:16.873 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:16.866+0000 7fa9ddd9b700 1 --2- 172.21.15.81:0/924122626 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fa9cc02ff60 0x7fa9cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:32:16.873 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:16.867+0000 7fa9ddd9b700 1 --2- 172.21.15.81:0/924122626 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fa9cc02ff60 0x7fa9cc032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fa9d00096a0 tx=0x7fa9d0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:32:16.874 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:16.871+0000 7fa9d6ffd700 1 -- 172.21.15.81:0/924122626 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa9c8014420 con 0x7fa9e00f5410 2023-11-15T06:32:17.078 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:17.076+0000 7fa9e4a3f700 1 -- 172.21.15.81:0/924122626 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa9e00fae60 con 0x7fa9e00f5410 2023-11-15T06:32:17.078 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:17.076+0000 7fa9d6ffd700 1 -- 172.21.15.81:0/924122626 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa9c8010230 con 0x7fa9e00f5410 2023-11-15T06:32:17.079 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:32:17.079 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:32:17.080 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:17.078+0000 7fa9e4a3f700 1 -- 172.21.15.81:0/924122626 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fa9cc02ff60 msgr2=0x7fa9cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:17.080 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:17.078+0000 7fa9e4a3f700 1 --2- 172.21.15.81:0/924122626 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fa9cc02ff60 0x7fa9cc032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fa9d00096a0 tx=0x7fa9d0006b40).stop 2023-11-15T06:32:17.080 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:17.078+0000 7fa9e4a3f700 1 -- 172.21.15.81:0/924122626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa9e00f5410 msgr2=0x7fa9e0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:17.080 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:17.078+0000 7fa9e4a3f700 1 --2- 172.21.15.81:0/924122626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa9e00f5410 0x7fa9e0106730 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fa9c8013510 tx=0x7fa9c800b040).stop 2023-11-15T06:32:17.081 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:17.078+0000 7fa9e4a3f700 1 -- 172.21.15.81:0/924122626 shutdown_connections 2023-11-15T06:32:17.081 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:17.078+0000 7fa9e4a3f700 1 --2- 172.21.15.81:0/924122626 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fa9cc02ff60 0x7fa9cc032420 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:17.081 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:17.078+0000 7fa9e4a3f700 1 --2- 172.21.15.81:0/924122626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa9e00f5410 0x7fa9e0106730 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:17.081 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:17.078+0000 7fa9e4a3f700 1 -- 172.21.15.81:0/924122626 >> 172.21.15.81:0/924122626 conn(0x7fa9e00f0b70 msgr2=0x7fa9e00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:32:17.081 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:17.078+0000 7fa9e4a3f700 1 -- 172.21.15.81:0/924122626 shutdown_connections 2023-11-15T06:32:17.082 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:17.078+0000 7fa9e4a3f700 1 -- 172.21.15.81:0/924122626 wait complete. 2023-11-15T06:32:17.082 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:32:17.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:16 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:32:15.838441+0000 mgr.smithi062.rwurtk (mgr.14164) 197 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:32:18.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:17 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:32:17.077120+0000 mon.smithi062 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.81:0/924122626' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:32:18.455 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:32:18.455 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:32:19.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:32:17.838813+0000 mgr.smithi062.rwurtk (mgr.14164) 198 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:32:19.676 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:19.673+0000 7f7a189cd700 1 -- 172.21.15.81:0/930755013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a140f62a0 msgr2=0x7f7a140f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:19.677 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:19.673+0000 7f7a189cd700 1 --2- 172.21.15.81:0/930755013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a140f62a0 0x7f7a140f66c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f7a04009fb0 tx=0x7f7a04009b30).stop 2023-11-15T06:32:19.677 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:19.673+0000 7f7a189cd700 1 -- 172.21.15.81:0/930755013 shutdown_connections 2023-11-15T06:32:19.677 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:19.673+0000 7f7a189cd700 1 --2- 172.21.15.81:0/930755013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a140f62a0 0x7f7a140f66c0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:19.677 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:19.673+0000 7f7a189cd700 1 -- 172.21.15.81:0/930755013 >> 172.21.15.81:0/930755013 conn(0x7f7a140f1aa0 msgr2=0x7f7a140f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:32:19.678 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:19.673+0000 7f7a189cd700 1 -- 172.21.15.81:0/930755013 shutdown_connections 2023-11-15T06:32:19.678 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:19.673+0000 7f7a189cd700 1 -- 172.21.15.81:0/930755013 wait complete. 2023-11-15T06:32:19.678 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:19.673+0000 7f7a189cd700 1 Processor -- start 2023-11-15T06:32:19.678 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:19.673+0000 7f7a189cd700 1 -- start start 2023-11-15T06:32:19.678 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:19.673+0000 7f7a189cd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a140f62a0 0x7f7a141064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:32:19.679 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:19.673+0000 7f7a189cd700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a141069a0 con 0x7f7a140f62a0 2023-11-15T06:32:19.681 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:19.674+0000 7f7a137fe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a140f62a0 0x7f7a141064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:32:19.681 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:19.674+0000 7f7a137fe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a140f62a0 0x7f7a141064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:33950/0 (socket says 172.21.15.81:33950) 2023-11-15T06:32:19.682 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:19.674+0000 7f7a137fe700 1 -- 172.21.15.81:0/3837412033 learned_addr learned my addr 172.21.15.81:0/3837412033 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:32:19.682 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:19.674+0000 7f7a137fe700 1 -- 172.21.15.81:0/3837412033 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a04005040 con 0x7f7a140f62a0 2023-11-15T06:32:19.682 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:19.674+0000 7f7a137fe700 1 --2- 172.21.15.81:0/3837412033 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a140f62a0 0x7f7a141064b0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f7a04013510 tx=0x7f7a0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:32:19.682 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:19.675+0000 7f79fbfff700 1 -- 172.21.15.81:0/3837412033 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a04009320 con 0x7f7a140f62a0 2023-11-15T06:32:19.682 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:19.675+0000 7f7a189cd700 1 -- 172.21.15.81:0/3837412033 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a14106ba0 con 0x7f7a140f62a0 2023-11-15T06:32:19.683 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:19.675+0000 7f79fbfff700 1 -- 172.21.15.81:0/3837412033 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7a04013f00 con 0x7f7a140f62a0 2023-11-15T06:32:19.683 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:19.675+0000 7f79fbfff700 1 -- 172.21.15.81:0/3837412033 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a04014dc0 con 0x7f7a140f62a0 2023-11-15T06:32:19.683 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:19.675+0000 7f7a189cd700 1 -- 172.21.15.81:0/3837412033 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a14107030 con 0x7f7a140f62a0 2023-11-15T06:32:19.683 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:19.675+0000 7f79fbfff700 1 -- 172.21.15.81:0/3837412033 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7a04014420 con 0x7f7a140f62a0 2023-11-15T06:32:19.683 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:19.675+0000 7f7a189cd700 1 -- 172.21.15.81:0/3837412033 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a14045ad0 con 0x7f7a140f62a0 2023-11-15T06:32:19.684 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:19.676+0000 7f79fbfff700 1 --2- 172.21.15.81:0/3837412033 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f79fc02ff60 0x7f79fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:32:19.684 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:19.676+0000 7f79fbfff700 1 -- 172.21.15.81:0/3837412033 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7a040410e0 con 0x7f7a140f62a0 2023-11-15T06:32:19.684 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:19.676+0000 7f7a12ffd700 1 --2- 172.21.15.81:0/3837412033 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f79fc02ff60 0x7f79fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:32:19.684 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:19.676+0000 7f7a12ffd700 1 --2- 172.21.15.81:0/3837412033 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f79fc02ff60 0x7f79fc032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f7a080096a0 tx=0x7f7a08006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:32:19.685 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:19.680+0000 7f79fbfff700 1 -- 172.21.15.81:0/3837412033 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7a040095f0 con 0x7f7a140f62a0 2023-11-15T06:32:19.888 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:19.885+0000 7f7a189cd700 1 -- 172.21.15.81:0/3837412033 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7a14107e80 con 0x7f7a140f62a0 2023-11-15T06:32:19.889 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:19.886+0000 7f79fbfff700 1 -- 172.21.15.81:0/3837412033 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7a0401cb10 con 0x7f7a140f62a0 2023-11-15T06:32:19.889 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:32:19.890 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:32:19.890 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:19.887+0000 7f7a189cd700 1 -- 172.21.15.81:0/3837412033 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f79fc02ff60 msgr2=0x7f79fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:19.891 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:19.887+0000 7f7a189cd700 1 --2- 172.21.15.81:0/3837412033 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f79fc02ff60 0x7f79fc032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f7a080096a0 tx=0x7f7a08006b40).stop 2023-11-15T06:32:19.891 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:19.887+0000 7f7a189cd700 1 -- 172.21.15.81:0/3837412033 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a140f62a0 msgr2=0x7f7a141064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:19.891 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:19.887+0000 7f7a189cd700 1 --2- 172.21.15.81:0/3837412033 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a140f62a0 0x7f7a141064b0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f7a04013510 tx=0x7f7a0400b040).stop 2023-11-15T06:32:19.891 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:19.887+0000 7f7a189cd700 1 -- 172.21.15.81:0/3837412033 shutdown_connections 2023-11-15T06:32:19.892 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:19.887+0000 7f7a189cd700 1 --2- 172.21.15.81:0/3837412033 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f79fc02ff60 0x7f79fc032420 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:19.892 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:19.887+0000 7f7a189cd700 1 --2- 172.21.15.81:0/3837412033 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a140f62a0 0x7f7a141064b0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:19.892 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:19.887+0000 7f7a189cd700 1 -- 172.21.15.81:0/3837412033 >> 172.21.15.81:0/3837412033 conn(0x7f7a140f1aa0 msgr2=0x7f7a140f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:32:19.892 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:19.888+0000 7f7a189cd700 1 -- 172.21.15.81:0/3837412033 shutdown_connections 2023-11-15T06:32:19.892 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:19.888+0000 7f7a189cd700 1 -- 172.21.15.81:0/3837412033 wait complete. 2023-11-15T06:32:19.893 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:32:20.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:19 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:32:19.887832+0000 mon.smithi062 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.81:0/3837412033' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:32:21.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:20 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:32:19.839166+0000 mgr.smithi062.rwurtk (mgr.14164) 199 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:32:21.323 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:32:21.324 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:32:22.555 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:22.549+0000 7f4edad4d700 1 -- 172.21.15.81:0/3191648476 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ed40f6510 msgr2=0x7f4ed40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:22.556 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:22.549+0000 7f4edad4d700 1 --2- 172.21.15.81:0/3191648476 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ed40f6510 0x7f4ed40f6930 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f4ec8009fb0 tx=0x7f4ec8009b30).stop 2023-11-15T06:32:22.556 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:22.549+0000 7f4edad4d700 1 -- 172.21.15.81:0/3191648476 shutdown_connections 2023-11-15T06:32:22.556 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:22.549+0000 7f4edad4d700 1 --2- 172.21.15.81:0/3191648476 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ed40f6510 0x7f4ed40f6930 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:22.557 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:22.549+0000 7f4edad4d700 1 -- 172.21.15.81:0/3191648476 >> 172.21.15.81:0/3191648476 conn(0x7f4ed40f1c70 msgr2=0x7f4ed40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:32:22.557 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:22.549+0000 7f4edad4d700 1 -- 172.21.15.81:0/3191648476 shutdown_connections 2023-11-15T06:32:22.557 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:22.549+0000 7f4edad4d700 1 -- 172.21.15.81:0/3191648476 wait complete. 2023-11-15T06:32:22.557 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:22.550+0000 7f4edad4d700 1 Processor -- start 2023-11-15T06:32:22.557 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:22.550+0000 7f4edad4d700 1 -- start start 2023-11-15T06:32:22.558 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:22.550+0000 7f4edad4d700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ed40f6510 0x7f4ed4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:32:22.558 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:22.550+0000 7f4edad4d700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed4106c20 con 0x7f4ed40f6510 2023-11-15T06:32:22.558 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:22.550+0000 7f4ed8ae9700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ed40f6510 0x7f4ed4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:32:22.558 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:22.550+0000 7f4ed8ae9700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ed40f6510 0x7f4ed4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:56170/0 (socket says 172.21.15.81:56170) 2023-11-15T06:32:22.558 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:22.550+0000 7f4ed8ae9700 1 -- 172.21.15.81:0/2043505980 learned_addr learned my addr 172.21.15.81:0/2043505980 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:32:22.559 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:22.551+0000 7f4ed8ae9700 1 -- 172.21.15.81:0/2043505980 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ec8005040 con 0x7f4ed40f6510 2023-11-15T06:32:22.559 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:22.551+0000 7f4ed8ae9700 1 --2- 172.21.15.81:0/2043505980 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ed40f6510 0x7f4ed4106730 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f4ec8013510 tx=0x7f4ec800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:32:22.559 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:22.551+0000 7f4ed17fa700 1 -- 172.21.15.81:0/2043505980 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ec8009320 con 0x7f4ed40f6510 2023-11-15T06:32:22.559 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:22.551+0000 7f4edad4d700 1 -- 172.21.15.81:0/2043505980 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ed4106e20 con 0x7f4ed40f6510 2023-11-15T06:32:22.559 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:22.551+0000 7f4ed17fa700 1 -- 172.21.15.81:0/2043505980 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4ec8013f00 con 0x7f4ed40f6510 2023-11-15T06:32:22.560 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:22.551+0000 7f4ed17fa700 1 -- 172.21.15.81:0/2043505980 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ec8014d90 con 0x7f4ed40f6510 2023-11-15T06:32:22.560 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:22.551+0000 7f4edad4d700 1 -- 172.21.15.81:0/2043505980 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ed41072b0 con 0x7f4ed40f6510 2023-11-15T06:32:22.560 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:22.552+0000 7f4ed17fa700 1 -- 172.21.15.81:0/2043505980 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4ec8014420 con 0x7f4ed40f6510 2023-11-15T06:32:22.560 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:22.552+0000 7f4edad4d700 1 -- 172.21.15.81:0/2043505980 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ed4045ad0 con 0x7f4ed40f6510 2023-11-15T06:32:22.560 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:22.552+0000 7f4ed17fa700 1 --2- 172.21.15.81:0/2043505980 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f4ec402ff60 0x7f4ec4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:32:22.560 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:22.552+0000 7f4ed17fa700 1 -- 172.21.15.81:0/2043505980 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4ec8040ff0 con 0x7f4ed40f6510 2023-11-15T06:32:22.561 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:22.552+0000 7f4ed3fff700 1 --2- 172.21.15.81:0/2043505980 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f4ec402ff60 0x7f4ec4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:32:22.561 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:22.553+0000 7f4ed3fff700 1 --2- 172.21.15.81:0/2043505980 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f4ec402ff60 0x7f4ec4032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f4ec00096a0 tx=0x7f4ec0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:32:22.561 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:22.557+0000 7f4ed17fa700 1 -- 172.21.15.81:0/2043505980 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4ec80095f0 con 0x7f4ed40f6510 2023-11-15T06:32:22.765 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:22.762+0000 7f4edad4d700 1 -- 172.21.15.81:0/2043505980 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4ed40fbf60 con 0x7f4ed40f6510 2023-11-15T06:32:22.765 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:22.762+0000 7f4ed17fa700 1 -- 172.21.15.81:0/2043505980 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4ec801db70 con 0x7f4ed40f6510 2023-11-15T06:32:22.766 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:32:22.766 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:32:22.767 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:22.764+0000 7f4edad4d700 1 -- 172.21.15.81:0/2043505980 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f4ec402ff60 msgr2=0x7f4ec4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:22.767 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:22.764+0000 7f4edad4d700 1 --2- 172.21.15.81:0/2043505980 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f4ec402ff60 0x7f4ec4032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f4ec00096a0 tx=0x7f4ec0006b40).stop 2023-11-15T06:32:22.767 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:22.764+0000 7f4edad4d700 1 -- 172.21.15.81:0/2043505980 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ed40f6510 msgr2=0x7f4ed4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:22.768 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:22.764+0000 7f4edad4d700 1 --2- 172.21.15.81:0/2043505980 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ed40f6510 0x7f4ed4106730 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f4ec8013510 tx=0x7f4ec800b040).stop 2023-11-15T06:32:22.768 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:22.764+0000 7f4edad4d700 1 -- 172.21.15.81:0/2043505980 shutdown_connections 2023-11-15T06:32:22.768 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:22.764+0000 7f4edad4d700 1 --2- 172.21.15.81:0/2043505980 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f4ec402ff60 0x7f4ec4032420 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:22.769 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:22.764+0000 7f4edad4d700 1 --2- 172.21.15.81:0/2043505980 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ed40f6510 0x7f4ed4106730 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:22.769 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:22.764+0000 7f4edad4d700 1 -- 172.21.15.81:0/2043505980 >> 172.21.15.81:0/2043505980 conn(0x7f4ed40f1c70 msgr2=0x7f4ed40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:32:22.769 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:22.764+0000 7f4edad4d700 1 -- 172.21.15.81:0/2043505980 shutdown_connections 2023-11-15T06:32:22.770 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:22.764+0000 7f4edad4d700 1 -- 172.21.15.81:0/2043505980 wait complete. 2023-11-15T06:32:22.770 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:32:23.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:22 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:32:21.839459+0000 mgr.smithi062.rwurtk (mgr.14164) 200 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:32:23.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:22 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:32:22.764295+0000 mon.smithi062 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.81:0/2043505980' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:32:24.168 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:32:24.169 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:32:25.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:24 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:32:23.839759+0000 mgr.smithi062.rwurtk (mgr.14164) 201 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:32:25.357 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:25.354+0000 7eff7500c700 1 -- 172.21.15.81:0/1106252175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff700f5410 msgr2=0x7eff700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:25.358 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:25.354+0000 7eff7500c700 1 --2- 172.21.15.81:0/1106252175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff700f5410 0x7eff700f5830 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7eff60009fb0 tx=0x7eff60009b30).stop 2023-11-15T06:32:25.358 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:25.354+0000 7eff7500c700 1 -- 172.21.15.81:0/1106252175 shutdown_connections 2023-11-15T06:32:25.358 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:25.354+0000 7eff7500c700 1 --2- 172.21.15.81:0/1106252175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff700f5410 0x7eff700f5830 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:25.359 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:25.354+0000 7eff7500c700 1 -- 172.21.15.81:0/1106252175 >> 172.21.15.81:0/1106252175 conn(0x7eff700f0b70 msgr2=0x7eff700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:32:25.359 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:25.354+0000 7eff7500c700 1 -- 172.21.15.81:0/1106252175 shutdown_connections 2023-11-15T06:32:25.359 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:25.354+0000 7eff7500c700 1 -- 172.21.15.81:0/1106252175 wait complete. 2023-11-15T06:32:25.361 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:25.355+0000 7eff7500c700 1 Processor -- start 2023-11-15T06:32:25.361 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:25.355+0000 7eff7500c700 1 -- start start 2023-11-15T06:32:25.361 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:25.355+0000 7eff7500c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff700f5410 0x7eff701093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:32:25.362 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:25.355+0000 7eff7500c700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff70060280 con 0x7eff700f5410 2023-11-15T06:32:25.362 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:25.355+0000 7eff6ed9d700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff700f5410 0x7eff701093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:32:25.362 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:25.355+0000 7eff6ed9d700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff700f5410 0x7eff701093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:56178/0 (socket says 172.21.15.81:56178) 2023-11-15T06:32:25.362 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:25.355+0000 7eff6ed9d700 1 -- 172.21.15.81:0/3406337177 learned_addr learned my addr 172.21.15.81:0/3406337177 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:32:25.362 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:25.356+0000 7eff6ed9d700 1 -- 172.21.15.81:0/3406337177 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff60005040 con 0x7eff700f5410 2023-11-15T06:32:25.363 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:25.356+0000 7eff6ed9d700 1 --2- 172.21.15.81:0/3406337177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff700f5410 0x7eff701093f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7eff60014000 tx=0x7eff6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:32:25.363 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:25.356+0000 7eff577fe700 1 -- 172.21.15.81:0/3406337177 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff600095f0 con 0x7eff700f5410 2023-11-15T06:32:25.363 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:25.356+0000 7eff7500c700 1 -- 172.21.15.81:0/3406337177 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff70060480 con 0x7eff700f5410 2023-11-15T06:32:25.364 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:25.356+0000 7eff577fe700 1 -- 172.21.15.81:0/3406337177 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7eff60007e70 con 0x7eff700f5410 2023-11-15T06:32:25.364 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:25.356+0000 7eff577fe700 1 -- 172.21.15.81:0/3406337177 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff60014ea0 con 0x7eff700f5410 2023-11-15T06:32:25.364 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:25.356+0000 7eff7500c700 1 -- 172.21.15.81:0/3406337177 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff70109b20 con 0x7eff700f5410 2023-11-15T06:32:25.364 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:25.357+0000 7eff577fe700 1 -- 172.21.15.81:0/3406337177 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7eff60025070 con 0x7eff700f5410 2023-11-15T06:32:25.364 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:25.357+0000 7eff7500c700 1 -- 172.21.15.81:0/3406337177 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff70045ad0 con 0x7eff700f5410 2023-11-15T06:32:25.365 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:25.357+0000 7eff577fe700 1 --2- 172.21.15.81:0/3406337177 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7eff5802ff60 0x7eff58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:32:25.365 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:25.357+0000 7eff577fe700 1 -- 172.21.15.81:0/3406337177 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7eff60040970 con 0x7eff700f5410 2023-11-15T06:32:25.365 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:25.357+0000 7eff6e59c700 1 --2- 172.21.15.81:0/3406337177 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7eff5802ff60 0x7eff58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:32:25.365 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:25.358+0000 7eff6e59c700 1 --2- 172.21.15.81:0/3406337177 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7eff5802ff60 0x7eff58032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7eff640096a0 tx=0x7eff64006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:32:25.366 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:25.362+0000 7eff577fe700 1 -- 172.21.15.81:0/3406337177 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7eff600079c0 con 0x7eff700f5410 2023-11-15T06:32:25.571 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:25.568+0000 7eff7500c700 1 -- 172.21.15.81:0/3406337177 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7eff7010a340 con 0x7eff700f5410 2023-11-15T06:32:25.572 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:25.568+0000 7eff577fe700 1 -- 172.21.15.81:0/3406337177 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7eff6000f300 con 0x7eff700f5410 2023-11-15T06:32:25.572 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:32:25.573 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:32:25.573 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:25.570+0000 7eff7500c700 1 -- 172.21.15.81:0/3406337177 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7eff5802ff60 msgr2=0x7eff58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:25.574 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:25.570+0000 7eff7500c700 1 --2- 172.21.15.81:0/3406337177 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7eff5802ff60 0x7eff58032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7eff640096a0 tx=0x7eff64006b40).stop 2023-11-15T06:32:25.574 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:25.570+0000 7eff7500c700 1 -- 172.21.15.81:0/3406337177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff700f5410 msgr2=0x7eff701093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:25.574 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:25.570+0000 7eff7500c700 1 --2- 172.21.15.81:0/3406337177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff700f5410 0x7eff701093f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7eff60014000 tx=0x7eff6000b040).stop 2023-11-15T06:32:25.574 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:25.570+0000 7eff7500c700 1 -- 172.21.15.81:0/3406337177 shutdown_connections 2023-11-15T06:32:25.575 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:25.570+0000 7eff7500c700 1 --2- 172.21.15.81:0/3406337177 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7eff5802ff60 0x7eff58032420 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:25.575 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:25.570+0000 7eff7500c700 1 --2- 172.21.15.81:0/3406337177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff700f5410 0x7eff701093f0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:25.575 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:25.570+0000 7eff7500c700 1 -- 172.21.15.81:0/3406337177 >> 172.21.15.81:0/3406337177 conn(0x7eff700f0b70 msgr2=0x7eff700f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:32:25.575 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:25.570+0000 7eff7500c700 1 -- 172.21.15.81:0/3406337177 shutdown_connections 2023-11-15T06:32:25.575 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:25.570+0000 7eff7500c700 1 -- 172.21.15.81:0/3406337177 wait complete. 2023-11-15T06:32:25.576 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:32:26.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:25 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:32:25.570492+0000 mon.smithi062 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.81:0/3406337177' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:32:26.971 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:32:26.971 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:32:27.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:26 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:32:25.840110+0000 mgr.smithi062.rwurtk (mgr.14164) 202 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:32:28.167 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:28.163+0000 7ff26380f700 1 -- 172.21.15.81:0/2439241560 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff25c0f5410 msgr2=0x7ff25c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:28.167 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:28.163+0000 7ff26380f700 1 --2- 172.21.15.81:0/2439241560 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff25c0f5410 0x7ff25c0f5830 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7ff250009fb0 tx=0x7ff250009b30).stop 2023-11-15T06:32:28.168 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:28.163+0000 7ff26380f700 1 -- 172.21.15.81:0/2439241560 shutdown_connections 2023-11-15T06:32:28.168 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:28.163+0000 7ff26380f700 1 --2- 172.21.15.81:0/2439241560 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff25c0f5410 0x7ff25c0f5830 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:28.168 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:28.163+0000 7ff26380f700 1 -- 172.21.15.81:0/2439241560 >> 172.21.15.81:0/2439241560 conn(0x7ff25c0f0b70 msgr2=0x7ff25c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:32:28.168 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:28.164+0000 7ff26380f700 1 -- 172.21.15.81:0/2439241560 shutdown_connections 2023-11-15T06:32:28.168 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:28.164+0000 7ff26380f700 1 -- 172.21.15.81:0/2439241560 wait complete. 2023-11-15T06:32:28.169 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:28.164+0000 7ff26380f700 1 Processor -- start 2023-11-15T06:32:28.169 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:28.164+0000 7ff26380f700 1 -- start start 2023-11-15T06:32:28.171 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:28.164+0000 7ff26380f700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff25c0f5410 0x7ff25c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:32:28.171 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:28.165+0000 7ff26380f700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff25c105b20 con 0x7ff25c0f5410 2023-11-15T06:32:28.171 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:28.165+0000 7ff2615ab700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff25c0f5410 0x7ff25c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:32:28.171 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:28.165+0000 7ff2615ab700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff25c0f5410 0x7ff25c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:56194/0 (socket says 172.21.15.81:56194) 2023-11-15T06:32:28.172 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:28.165+0000 7ff2615ab700 1 -- 172.21.15.81:0/3363127369 learned_addr learned my addr 172.21.15.81:0/3363127369 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:32:28.172 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:28.165+0000 7ff2615ab700 1 -- 172.21.15.81:0/3363127369 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff250005040 con 0x7ff25c0f5410 2023-11-15T06:32:28.172 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:28.165+0000 7ff2615ab700 1 --2- 172.21.15.81:0/3363127369 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff25c0f5410 0x7ff25c105630 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7ff250013510 tx=0x7ff25000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:32:28.172 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:28.166+0000 7ff24dffb700 1 -- 172.21.15.81:0/3363127369 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff250009320 con 0x7ff25c0f5410 2023-11-15T06:32:28.172 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:28.166+0000 7ff26380f700 1 -- 172.21.15.81:0/3363127369 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff25c105d20 con 0x7ff25c0f5410 2023-11-15T06:32:28.173 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:28.166+0000 7ff24dffb700 1 -- 172.21.15.81:0/3363127369 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff250013f00 con 0x7ff25c0f5410 2023-11-15T06:32:28.173 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:28.166+0000 7ff24dffb700 1 -- 172.21.15.81:0/3363127369 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff250014d90 con 0x7ff25c0f5410 2023-11-15T06:32:28.173 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:28.166+0000 7ff26380f700 1 -- 172.21.15.81:0/3363127369 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff25c1061b0 con 0x7ff25c0f5410 2023-11-15T06:32:28.173 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:28.167+0000 7ff24dffb700 1 -- 172.21.15.81:0/3363127369 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff2500263a0 con 0x7ff25c0f5410 2023-11-15T06:32:28.173 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:28.167+0000 7ff26380f700 1 -- 172.21.15.81:0/3363127369 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff25c045ad0 con 0x7ff25c0f5410 2023-11-15T06:32:28.174 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:28.167+0000 7ff24dffb700 1 --2- 172.21.15.81:0/3363127369 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff24802ff60 0x7ff248032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:32:28.174 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:28.167+0000 7ff24dffb700 1 -- 172.21.15.81:0/3363127369 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff250040fc0 con 0x7ff25c0f5410 2023-11-15T06:32:28.174 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:28.167+0000 7ff260daa700 1 --2- 172.21.15.81:0/3363127369 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff24802ff60 0x7ff248032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:32:28.174 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:28.168+0000 7ff260daa700 1 --2- 172.21.15.81:0/3363127369 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff24802ff60 0x7ff248032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7ff2580096a0 tx=0x7ff258006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:32:28.175 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:28.172+0000 7ff24dffb700 1 -- 172.21.15.81:0/3363127369 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff2500126f0 con 0x7ff25c0f5410 2023-11-15T06:32:28.380 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:28.376+0000 7ff26380f700 1 -- 172.21.15.81:0/3363127369 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff25c0fae60 con 0x7ff25c0f5410 2023-11-15T06:32:28.380 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:28.377+0000 7ff24dffb700 1 -- 172.21.15.81:0/3363127369 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff250012330 con 0x7ff25c0f5410 2023-11-15T06:32:28.381 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:32:28.381 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:32:28.382 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:28.379+0000 7ff26380f700 1 -- 172.21.15.81:0/3363127369 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff24802ff60 msgr2=0x7ff248032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:28.382 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:28.379+0000 7ff26380f700 1 --2- 172.21.15.81:0/3363127369 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff24802ff60 0x7ff248032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7ff2580096a0 tx=0x7ff258006b40).stop 2023-11-15T06:32:28.382 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:28.379+0000 7ff26380f700 1 -- 172.21.15.81:0/3363127369 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff25c0f5410 msgr2=0x7ff25c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:28.383 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:28.379+0000 7ff26380f700 1 --2- 172.21.15.81:0/3363127369 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff25c0f5410 0x7ff25c105630 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7ff250013510 tx=0x7ff25000b040).stop 2023-11-15T06:32:28.383 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:28.379+0000 7ff26380f700 1 -- 172.21.15.81:0/3363127369 shutdown_connections 2023-11-15T06:32:28.383 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:28.379+0000 7ff26380f700 1 --2- 172.21.15.81:0/3363127369 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff24802ff60 0x7ff248032420 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:28.383 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:28.379+0000 7ff26380f700 1 --2- 172.21.15.81:0/3363127369 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff25c0f5410 0x7ff25c105630 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:28.383 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:28.379+0000 7ff26380f700 1 -- 172.21.15.81:0/3363127369 >> 172.21.15.81:0/3363127369 conn(0x7ff25c0f0b70 msgr2=0x7ff25c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:32:28.383 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:28.379+0000 7ff26380f700 1 -- 172.21.15.81:0/3363127369 shutdown_connections 2023-11-15T06:32:28.384 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:28.379+0000 7ff26380f700 1 -- 172.21.15.81:0/3363127369 wait complete. 2023-11-15T06:32:28.384 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:32:28.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:28 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:32:27.840492+0000 mgr.smithi062.rwurtk (mgr.14164) 203 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:32:28.919 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:28 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:32:28.379024+0000 mon.smithi062 (mon.0) 266 : audit [DBG] from='client.? 172.21.15.81:0/3363127369' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:32:29.805 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:32:29.805 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:32:31.093 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:31.089+0000 7f58cf7de700 1 -- 172.21.15.81:0/2317328125 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f58c80f6510 msgr2=0x7f58c80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:31.094 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:31.089+0000 7f58cf7de700 1 --2- 172.21.15.81:0/2317328125 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f58c80f6510 0x7f58c80f6930 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f58b8009fb0 tx=0x7f58b8009b30).stop 2023-11-15T06:32:31.094 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:31.089+0000 7f58cf7de700 1 -- 172.21.15.81:0/2317328125 shutdown_connections 2023-11-15T06:32:31.094 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:31.089+0000 7f58cf7de700 1 --2- 172.21.15.81:0/2317328125 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f58c80f6510 0x7f58c80f6930 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:31.094 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:31.089+0000 7f58cf7de700 1 -- 172.21.15.81:0/2317328125 >> 172.21.15.81:0/2317328125 conn(0x7f58c80f1c70 msgr2=0x7f58c80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:32:31.095 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:31.089+0000 7f58cf7de700 1 -- 172.21.15.81:0/2317328125 shutdown_connections 2023-11-15T06:32:31.095 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:31.090+0000 7f58cf7de700 1 -- 172.21.15.81:0/2317328125 wait complete. 2023-11-15T06:32:31.095 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:31.090+0000 7f58cf7de700 1 Processor -- start 2023-11-15T06:32:31.095 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:31.090+0000 7f58cf7de700 1 -- start start 2023-11-15T06:32:31.095 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:31.090+0000 7f58cf7de700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f58c80f6510 0x7f58c8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:32:31.096 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:31.090+0000 7f58cf7de700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58c8106c20 con 0x7f58c80f6510 2023-11-15T06:32:31.096 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:31.090+0000 7f58cd57a700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f58c80f6510 0x7f58c8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:32:31.096 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:31.090+0000 7f58cd57a700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f58c80f6510 0x7f58c8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:56216/0 (socket says 172.21.15.81:56216) 2023-11-15T06:32:31.096 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:31.090+0000 7f58cd57a700 1 -- 172.21.15.81:0/1297941888 learned_addr learned my addr 172.21.15.81:0/1297941888 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:32:31.096 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:31.091+0000 7f58cd57a700 1 -- 172.21.15.81:0/1297941888 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58b8005040 con 0x7f58c80f6510 2023-11-15T06:32:31.096 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:31.091+0000 7f58cd57a700 1 --2- 172.21.15.81:0/1297941888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f58c80f6510 0x7f58c8106730 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f58b8013510 tx=0x7f58b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:32:31.098 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:31.091+0000 7f58bdffb700 1 -- 172.21.15.81:0/1297941888 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58b8009320 con 0x7f58c80f6510 2023-11-15T06:32:31.098 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:31.091+0000 7f58bdffb700 1 -- 172.21.15.81:0/1297941888 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f58b8013f00 con 0x7f58c80f6510 2023-11-15T06:32:31.098 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:31.091+0000 7f58cf7de700 1 -- 172.21.15.81:0/1297941888 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58c8106e20 con 0x7f58c80f6510 2023-11-15T06:32:31.099 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:31.091+0000 7f58bdffb700 1 -- 172.21.15.81:0/1297941888 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58b8014d90 con 0x7f58c80f6510 2023-11-15T06:32:31.099 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:31.091+0000 7f58cf7de700 1 -- 172.21.15.81:0/1297941888 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58c81072b0 con 0x7f58c80f6510 2023-11-15T06:32:31.099 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:31.092+0000 7f58bdffb700 1 -- 172.21.15.81:0/1297941888 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f58b80095f0 con 0x7f58c80f6510 2023-11-15T06:32:31.099 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:31.092+0000 7f58cf7de700 1 -- 172.21.15.81:0/1297941888 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58c8045ad0 con 0x7f58c80f6510 2023-11-15T06:32:31.099 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:31.092+0000 7f58bdffb700 1 --2- 172.21.15.81:0/1297941888 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f58b402ff60 0x7f58b4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:32:31.100 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:31.092+0000 7f58bdffb700 1 -- 172.21.15.81:0/1297941888 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f58b8041000 con 0x7f58c80f6510 2023-11-15T06:32:31.100 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:31.092+0000 7f58ccd79700 1 --2- 172.21.15.81:0/1297941888 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f58b402ff60 0x7f58b4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:32:31.100 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:31.093+0000 7f58ccd79700 1 --2- 172.21.15.81:0/1297941888 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f58b402ff60 0x7f58b4032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f58c40096a0 tx=0x7f58c4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:32:31.100 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:31.097+0000 7f58bdffb700 1 -- 172.21.15.81:0/1297941888 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f58b801d7f0 con 0x7f58c80f6510 2023-11-15T06:32:31.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:30 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:32:29.840796+0000 mgr.smithi062.rwurtk (mgr.14164) 204 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:32:31.304 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:31.300+0000 7f58cf7de700 1 -- 172.21.15.81:0/1297941888 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f58c80fbf60 con 0x7f58c80f6510 2023-11-15T06:32:31.304 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:31.301+0000 7f58bdffb700 1 -- 172.21.15.81:0/1297941888 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f58b8010230 con 0x7f58c80f6510 2023-11-15T06:32:31.305 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:32:31.305 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:32:31.306 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:31.302+0000 7f58cf7de700 1 -- 172.21.15.81:0/1297941888 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f58b402ff60 msgr2=0x7f58b4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:31.306 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:31.302+0000 7f58cf7de700 1 --2- 172.21.15.81:0/1297941888 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f58b402ff60 0x7f58b4032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f58c40096a0 tx=0x7f58c4006b40).stop 2023-11-15T06:32:31.307 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:31.302+0000 7f58cf7de700 1 -- 172.21.15.81:0/1297941888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f58c80f6510 msgr2=0x7f58c8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:31.307 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:31.302+0000 7f58cf7de700 1 --2- 172.21.15.81:0/1297941888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f58c80f6510 0x7f58c8106730 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f58b8013510 tx=0x7f58b800b040).stop 2023-11-15T06:32:31.307 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:31.302+0000 7f58cf7de700 1 -- 172.21.15.81:0/1297941888 shutdown_connections 2023-11-15T06:32:31.307 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:31.302+0000 7f58cf7de700 1 --2- 172.21.15.81:0/1297941888 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f58b402ff60 0x7f58b4032420 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:31.307 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:31.302+0000 7f58cf7de700 1 --2- 172.21.15.81:0/1297941888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f58c80f6510 0x7f58c8106730 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:31.307 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:31.302+0000 7f58cf7de700 1 -- 172.21.15.81:0/1297941888 >> 172.21.15.81:0/1297941888 conn(0x7f58c80f1c70 msgr2=0x7f58c80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:32:31.308 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:31.302+0000 7f58cf7de700 1 -- 172.21.15.81:0/1297941888 shutdown_connections 2023-11-15T06:32:31.308 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:31.302+0000 7f58cf7de700 1 -- 172.21.15.81:0/1297941888 wait complete. 2023-11-15T06:32:31.308 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:32:32.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:31 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:32:31.302624+0000 mon.smithi062 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.81:0/1297941888' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:32:32.693 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:32:32.694 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:32:33.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:32 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:32:31.841149+0000 mgr.smithi062.rwurtk (mgr.14164) 205 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:32:33.968 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:33.963+0000 7efc6a720700 1 -- 172.21.15.81:0/1984230643 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efc640f51a0 msgr2=0x7efc640f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:33.969 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:33.963+0000 7efc6a720700 1 --2- 172.21.15.81:0/1984230643 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efc640f51a0 0x7efc640f55c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7efc54009fb0 tx=0x7efc54009b30).stop 2023-11-15T06:32:33.969 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:33.963+0000 7efc6a720700 1 -- 172.21.15.81:0/1984230643 shutdown_connections 2023-11-15T06:32:33.969 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:33.963+0000 7efc6a720700 1 --2- 172.21.15.81:0/1984230643 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efc640f51a0 0x7efc640f55c0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:33.969 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:33.963+0000 7efc6a720700 1 -- 172.21.15.81:0/1984230643 >> 172.21.15.81:0/1984230643 conn(0x7efc640f09a0 msgr2=0x7efc640f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:32:33.970 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:33.963+0000 7efc6a720700 1 -- 172.21.15.81:0/1984230643 shutdown_connections 2023-11-15T06:32:33.970 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:33.963+0000 7efc6a720700 1 -- 172.21.15.81:0/1984230643 wait complete. 2023-11-15T06:32:33.970 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:33.964+0000 7efc6a720700 1 Processor -- start 2023-11-15T06:32:33.970 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:33.964+0000 7efc6a720700 1 -- start start 2023-11-15T06:32:33.970 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:33.964+0000 7efc6a720700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efc640f51a0 0x7efc641053e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:32:33.971 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:33.964+0000 7efc6a720700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc641058d0 con 0x7efc640f51a0 2023-11-15T06:32:33.971 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:33.965+0000 7efc6971e700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efc640f51a0 0x7efc641053e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:32:33.971 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:33.965+0000 7efc6971e700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efc640f51a0 0x7efc641053e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:49614/0 (socket says 172.21.15.81:49614) 2023-11-15T06:32:33.971 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:33.965+0000 7efc6971e700 1 -- 172.21.15.81:0/1311064267 learned_addr learned my addr 172.21.15.81:0/1311064267 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:32:33.971 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:33.965+0000 7efc6971e700 1 -- 172.21.15.81:0/1311064267 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc54005040 con 0x7efc640f51a0 2023-11-15T06:32:33.972 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:33.965+0000 7efc6971e700 1 --2- 172.21.15.81:0/1311064267 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efc640f51a0 0x7efc641053e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7efc54013510 tx=0x7efc5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:32:33.972 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:33.965+0000 7efc59ffb700 1 -- 172.21.15.81:0/1311064267 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc54009320 con 0x7efc640f51a0 2023-11-15T06:32:33.972 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:33.965+0000 7efc6a720700 1 -- 172.21.15.81:0/1311064267 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc64105ad0 con 0x7efc640f51a0 2023-11-15T06:32:33.972 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:33.965+0000 7efc59ffb700 1 -- 172.21.15.81:0/1311064267 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efc54013f00 con 0x7efc640f51a0 2023-11-15T06:32:33.972 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:33.965+0000 7efc59ffb700 1 -- 172.21.15.81:0/1311064267 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc54014dc0 con 0x7efc640f51a0 2023-11-15T06:32:33.972 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:33.966+0000 7efc6a720700 1 -- 172.21.15.81:0/1311064267 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc64105f60 con 0x7efc640f51a0 2023-11-15T06:32:33.974 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:33.966+0000 7efc59ffb700 1 -- 172.21.15.81:0/1311064267 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efc540095f0 con 0x7efc640f51a0 2023-11-15T06:32:33.974 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:33.966+0000 7efc6a720700 1 -- 172.21.15.81:0/1311064267 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc64045ad0 con 0x7efc640f51a0 2023-11-15T06:32:33.974 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:33.966+0000 7efc59ffb700 1 --2- 172.21.15.81:0/1311064267 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7efc5002ff60 0x7efc50032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:32:33.974 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:33.967+0000 7efc59ffb700 1 -- 172.21.15.81:0/1311064267 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efc54041100 con 0x7efc640f51a0 2023-11-15T06:32:33.974 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:33.967+0000 7efc68f1d700 1 --2- 172.21.15.81:0/1311064267 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7efc5002ff60 0x7efc50032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:32:33.975 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:33.967+0000 7efc68f1d700 1 --2- 172.21.15.81:0/1311064267 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7efc5002ff60 0x7efc50032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7efc600096a0 tx=0x7efc60006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:32:33.975 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:33.971+0000 7efc59ffb700 1 -- 172.21.15.81:0/1311064267 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efc54014420 con 0x7efc640f51a0 2023-11-15T06:32:34.176 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:34.173+0000 7efc6a720700 1 -- 172.21.15.81:0/1311064267 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efc64106e10 con 0x7efc640f51a0 2023-11-15T06:32:34.177 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:34.173+0000 7efc59ffb700 1 -- 172.21.15.81:0/1311064267 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efc54010230 con 0x7efc640f51a0 2023-11-15T06:32:34.178 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:32:34.178 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:32:34.179 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:34.175+0000 7efc6a720700 1 -- 172.21.15.81:0/1311064267 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7efc5002ff60 msgr2=0x7efc50032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:34.179 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:34.175+0000 7efc6a720700 1 --2- 172.21.15.81:0/1311064267 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7efc5002ff60 0x7efc50032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7efc600096a0 tx=0x7efc60006b40).stop 2023-11-15T06:32:34.179 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:34.175+0000 7efc6a720700 1 -- 172.21.15.81:0/1311064267 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efc640f51a0 msgr2=0x7efc641053e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:34.180 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:34.175+0000 7efc6a720700 1 --2- 172.21.15.81:0/1311064267 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efc640f51a0 0x7efc641053e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7efc54013510 tx=0x7efc5400b040).stop 2023-11-15T06:32:34.180 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:34.175+0000 7efc6a720700 1 -- 172.21.15.81:0/1311064267 shutdown_connections 2023-11-15T06:32:34.180 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:34.175+0000 7efc6a720700 1 --2- 172.21.15.81:0/1311064267 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7efc5002ff60 0x7efc50032420 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:34.180 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:34.175+0000 7efc6a720700 1 --2- 172.21.15.81:0/1311064267 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efc640f51a0 0x7efc641053e0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:34.180 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:34.175+0000 7efc6a720700 1 -- 172.21.15.81:0/1311064267 >> 172.21.15.81:0/1311064267 conn(0x7efc640f09a0 msgr2=0x7efc640f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:32:34.180 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:34.175+0000 7efc6a720700 1 -- 172.21.15.81:0/1311064267 shutdown_connections 2023-11-15T06:32:34.181 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:34.175+0000 7efc6a720700 1 -- 172.21.15.81:0/1311064267 wait complete. 2023-11-15T06:32:34.181 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:32:35.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:34 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:32:33.841524+0000 mgr.smithi062.rwurtk (mgr.14164) 206 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:32:35.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:34 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:32:34.175253+0000 mon.smithi062 (mon.0) 268 : audit [DBG] from='client.? 172.21.15.81:0/1311064267' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:32:35.605 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:32:35.606 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:32:36.874 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:36.872+0000 7fdf659df700 1 -- 172.21.15.81:0/2703490978 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf600f47c0 msgr2=0x7fdf600f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:36.875 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:36.872+0000 7fdf659df700 1 --2- 172.21.15.81:0/2703490978 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf600f47c0 0x7fdf600f4be0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7fdf54009fb0 tx=0x7fdf54009b30).stop 2023-11-15T06:32:36.875 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:36.872+0000 7fdf659df700 1 -- 172.21.15.81:0/2703490978 shutdown_connections 2023-11-15T06:32:36.875 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:36.872+0000 7fdf659df700 1 --2- 172.21.15.81:0/2703490978 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf600f47c0 0x7fdf600f4be0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:36.876 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:36.872+0000 7fdf659df700 1 -- 172.21.15.81:0/2703490978 >> 172.21.15.81:0/2703490978 conn(0x7fdf600f02e0 msgr2=0x7fdf600f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:32:36.876 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:36.872+0000 7fdf659df700 1 -- 172.21.15.81:0/2703490978 shutdown_connections 2023-11-15T06:32:36.876 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:36.872+0000 7fdf659df700 1 -- 172.21.15.81:0/2703490978 wait complete. 2023-11-15T06:32:36.879 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:36.873+0000 7fdf659df700 1 Processor -- start 2023-11-15T06:32:36.879 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:36.873+0000 7fdf659df700 1 -- start start 2023-11-15T06:32:36.880 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:36.873+0000 7fdf659df700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf600f47c0 0x7fdf60106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:32:36.880 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:36.873+0000 7fdf659df700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf60106bf0 con 0x7fdf600f47c0 2023-11-15T06:32:36.880 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:36.873+0000 7fdf5effd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf600f47c0 0x7fdf60106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:32:36.880 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:36.873+0000 7fdf5effd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf600f47c0 0x7fdf60106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:49638/0 (socket says 172.21.15.81:49638) 2023-11-15T06:32:36.880 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:36.873+0000 7fdf5effd700 1 -- 172.21.15.81:0/2515555788 learned_addr learned my addr 172.21.15.81:0/2515555788 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:32:36.881 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:36.874+0000 7fdf5effd700 1 -- 172.21.15.81:0/2515555788 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf54005040 con 0x7fdf600f47c0 2023-11-15T06:32:36.881 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:36.874+0000 7fdf5effd700 1 --2- 172.21.15.81:0/2515555788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf600f47c0 0x7fdf60106700 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fdf54013510 tx=0x7fdf5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:32:36.881 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:36.874+0000 7fdf477fe700 1 -- 172.21.15.81:0/2515555788 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf54009320 con 0x7fdf600f47c0 2023-11-15T06:32:36.882 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:36.874+0000 7fdf659df700 1 -- 172.21.15.81:0/2515555788 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf60106df0 con 0x7fdf600f47c0 2023-11-15T06:32:36.882 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:36.874+0000 7fdf477fe700 1 -- 172.21.15.81:0/2515555788 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdf54013f00 con 0x7fdf600f47c0 2023-11-15T06:32:36.882 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:36.874+0000 7fdf477fe700 1 -- 172.21.15.81:0/2515555788 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf54014d90 con 0x7fdf600f47c0 2023-11-15T06:32:36.882 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:36.874+0000 7fdf659df700 1 -- 172.21.15.81:0/2515555788 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf60107280 con 0x7fdf600f47c0 2023-11-15T06:32:36.882 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:36.875+0000 7fdf477fe700 1 -- 172.21.15.81:0/2515555788 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdf54014420 con 0x7fdf600f47c0 2023-11-15T06:32:36.886 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:36.875+0000 7fdf659df700 1 -- 172.21.15.81:0/2515555788 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf60045ad0 con 0x7fdf600f47c0 2023-11-15T06:32:36.886 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:36.875+0000 7fdf477fe700 1 --2- 172.21.15.81:0/2515555788 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fdf4802ff60 0x7fdf48032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:32:36.887 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:36.875+0000 7fdf477fe700 1 -- 172.21.15.81:0/2515555788 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdf54041020 con 0x7fdf600f47c0 2023-11-15T06:32:36.887 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:36.875+0000 7fdf5e7fc700 1 --2- 172.21.15.81:0/2515555788 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fdf4802ff60 0x7fdf48032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:32:36.887 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:36.875+0000 7fdf5e7fc700 1 --2- 172.21.15.81:0/2515555788 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fdf4802ff60 0x7fdf48032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fdf500096a0 tx=0x7fdf50006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:32:36.888 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:36.879+0000 7fdf477fe700 1 -- 172.21.15.81:0/2515555788 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdf540095f0 con 0x7fdf600f47c0 2023-11-15T06:32:37.083 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:37.081+0000 7fdf659df700 1 -- 172.21.15.81:0/2515555788 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdf601080d0 con 0x7fdf600f47c0 2023-11-15T06:32:37.083 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:37.081+0000 7fdf477fe700 1 -- 172.21.15.81:0/2515555788 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdf5401db70 con 0x7fdf600f47c0 2023-11-15T06:32:37.084 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:32:37.084 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:32:37.085 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:37.083+0000 7fdf659df700 1 -- 172.21.15.81:0/2515555788 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fdf4802ff60 msgr2=0x7fdf48032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:37.085 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:37.083+0000 7fdf659df700 1 --2- 172.21.15.81:0/2515555788 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fdf4802ff60 0x7fdf48032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fdf500096a0 tx=0x7fdf50006b40).stop 2023-11-15T06:32:37.085 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:37.083+0000 7fdf659df700 1 -- 172.21.15.81:0/2515555788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf600f47c0 msgr2=0x7fdf60106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:37.086 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:37.083+0000 7fdf659df700 1 --2- 172.21.15.81:0/2515555788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf600f47c0 0x7fdf60106700 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fdf54013510 tx=0x7fdf5400b040).stop 2023-11-15T06:32:37.086 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:37.083+0000 7fdf659df700 1 -- 172.21.15.81:0/2515555788 shutdown_connections 2023-11-15T06:32:37.086 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:37.083+0000 7fdf659df700 1 --2- 172.21.15.81:0/2515555788 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fdf4802ff60 0x7fdf48032420 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:37.086 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:37.083+0000 7fdf659df700 1 --2- 172.21.15.81:0/2515555788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf600f47c0 0x7fdf60106700 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:37.086 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:37.083+0000 7fdf659df700 1 -- 172.21.15.81:0/2515555788 >> 172.21.15.81:0/2515555788 conn(0x7fdf600f02e0 msgr2=0x7fdf600f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:32:37.087 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:37.083+0000 7fdf659df700 1 -- 172.21.15.81:0/2515555788 shutdown_connections 2023-11-15T06:32:37.087 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:37.083+0000 7fdf659df700 1 -- 172.21.15.81:0/2515555788 wait complete. 2023-11-15T06:32:37.087 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:32:37.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:36 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:32:35.841916+0000 mgr.smithi062.rwurtk (mgr.14164) 207 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:32:38.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:37 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:32:37.082105+0000 mon.smithi062 (mon.0) 269 : audit [DBG] from='client.? 172.21.15.81:0/2515555788' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:32:38.457 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:32:38.458 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:32:39.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:38 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:32:37.842303+0000 mgr.smithi062.rwurtk (mgr.14164) 208 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:32:39.717 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:39.713+0000 7f8f7ec2f700 1 -- 172.21.15.81:0/2016546097 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f78060280 msgr2=0x7f8f780fb4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:39.718 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:39.713+0000 7f8f7ec2f700 1 --2- 172.21.15.81:0/2016546097 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f78060280 0x7f8f780fb4f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f8f6c009fb0 tx=0x7f8f6c009b30).stop 2023-11-15T06:32:39.718 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:39.713+0000 7f8f7ec2f700 1 -- 172.21.15.81:0/2016546097 shutdown_connections 2023-11-15T06:32:39.718 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:39.713+0000 7f8f7ec2f700 1 --2- 172.21.15.81:0/2016546097 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f78060280 0x7f8f780fb4f0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:39.719 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:39.713+0000 7f8f7ec2f700 1 -- 172.21.15.81:0/2016546097 >> 172.21.15.81:0/2016546097 conn(0x7f8f780f0b70 msgr2=0x7f8f780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:32:39.719 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:39.713+0000 7f8f7ec2f700 1 -- 172.21.15.81:0/2016546097 shutdown_connections 2023-11-15T06:32:39.719 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:39.713+0000 7f8f7ec2f700 1 -- 172.21.15.81:0/2016546097 wait complete. 2023-11-15T06:32:39.720 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:39.714+0000 7f8f7ec2f700 1 Processor -- start 2023-11-15T06:32:39.720 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:39.714+0000 7f8f7ec2f700 1 -- start start 2023-11-15T06:32:39.720 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:39.714+0000 7f8f7ec2f700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f78060280 0x7f8f781088e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:32:39.721 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:39.714+0000 7f8f7ec2f700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f78108dd0 con 0x7f8f78060280 2023-11-15T06:32:39.721 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:39.714+0000 7f8f7c9cb700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f78060280 0x7f8f781088e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:32:39.722 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:39.714+0000 7f8f7c9cb700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f78060280 0x7f8f781088e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:49656/0 (socket says 172.21.15.81:49656) 2023-11-15T06:32:39.722 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:39.714+0000 7f8f7c9cb700 1 -- 172.21.15.81:0/3793201766 learned_addr learned my addr 172.21.15.81:0/3793201766 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:32:39.722 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:39.714+0000 7f8f7c9cb700 1 -- 172.21.15.81:0/3793201766 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f6c005040 con 0x7f8f78060280 2023-11-15T06:32:39.723 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:39.715+0000 7f8f7c9cb700 1 --2- 172.21.15.81:0/3793201766 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f78060280 0x7f8f781088e0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f8f6c013510 tx=0x7f8f6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:32:39.723 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:39.715+0000 7f8f757fa700 1 -- 172.21.15.81:0/3793201766 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f6c009320 con 0x7f8f78060280 2023-11-15T06:32:39.723 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:39.715+0000 7f8f7ec2f700 1 -- 172.21.15.81:0/3793201766 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f78108fd0 con 0x7f8f78060280 2023-11-15T06:32:39.723 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:39.715+0000 7f8f757fa700 1 -- 172.21.15.81:0/3793201766 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8f6c013f00 con 0x7f8f78060280 2023-11-15T06:32:39.724 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:39.715+0000 7f8f757fa700 1 -- 172.21.15.81:0/3793201766 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f6c014dc0 con 0x7f8f78060280 2023-11-15T06:32:39.724 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:39.715+0000 7f8f7ec2f700 1 -- 172.21.15.81:0/3793201766 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f78109460 con 0x7f8f78060280 2023-11-15T06:32:39.726 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:39.716+0000 7f8f757fa700 1 -- 172.21.15.81:0/3793201766 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8f6c014420 con 0x7f8f78060280 2023-11-15T06:32:39.726 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:39.716+0000 7f8f7ec2f700 1 -- 172.21.15.81:0/3793201766 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f78045ad0 con 0x7f8f78060280 2023-11-15T06:32:39.726 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:39.716+0000 7f8f757fa700 1 --2- 172.21.15.81:0/3793201766 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f8f6802ff60 0x7f8f68032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:32:39.726 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:39.716+0000 7f8f757fa700 1 -- 172.21.15.81:0/3793201766 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8f6c041050 con 0x7f8f78060280 2023-11-15T06:32:39.727 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:39.716+0000 7f8f77fff700 1 --2- 172.21.15.81:0/3793201766 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f8f6802ff60 0x7f8f68032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:32:39.727 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:39.716+0000 7f8f77fff700 1 --2- 172.21.15.81:0/3793201766 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f8f6802ff60 0x7f8f68032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f8f640096a0 tx=0x7f8f64006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:32:39.727 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:39.721+0000 7f8f757fa700 1 -- 172.21.15.81:0/3793201766 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8f6c0095f0 con 0x7f8f78060280 2023-11-15T06:32:39.926 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:39.924+0000 7f8f7ec2f700 1 -- 172.21.15.81:0/3793201766 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8f7810a310 con 0x7f8f78060280 2023-11-15T06:32:39.927 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:39.925+0000 7f8f757fa700 1 -- 172.21.15.81:0/3793201766 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8f6c01cb10 con 0x7f8f78060280 2023-11-15T06:32:39.927 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:32:39.928 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:32:39.929 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:39.926+0000 7f8f7ec2f700 1 -- 172.21.15.81:0/3793201766 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f8f6802ff60 msgr2=0x7f8f68032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:39.929 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:39.926+0000 7f8f7ec2f700 1 --2- 172.21.15.81:0/3793201766 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f8f6802ff60 0x7f8f68032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f8f640096a0 tx=0x7f8f64006b40).stop 2023-11-15T06:32:39.929 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:39.926+0000 7f8f7ec2f700 1 -- 172.21.15.81:0/3793201766 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f78060280 msgr2=0x7f8f781088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:39.929 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:39.926+0000 7f8f7ec2f700 1 --2- 172.21.15.81:0/3793201766 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f78060280 0x7f8f781088e0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f8f6c013510 tx=0x7f8f6c00b040).stop 2023-11-15T06:32:39.930 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:39.926+0000 7f8f7ec2f700 1 -- 172.21.15.81:0/3793201766 shutdown_connections 2023-11-15T06:32:39.930 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:39.926+0000 7f8f7ec2f700 1 --2- 172.21.15.81:0/3793201766 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f8f6802ff60 0x7f8f68032420 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:39.930 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:39.926+0000 7f8f7ec2f700 1 --2- 172.21.15.81:0/3793201766 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f78060280 0x7f8f781088e0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:39.931 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:39.926+0000 7f8f7ec2f700 1 -- 172.21.15.81:0/3793201766 >> 172.21.15.81:0/3793201766 conn(0x7f8f780f0b70 msgr2=0x7f8f780f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:32:39.931 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:39.926+0000 7f8f7ec2f700 1 -- 172.21.15.81:0/3793201766 shutdown_connections 2023-11-15T06:32:39.931 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:39.926+0000 7f8f7ec2f700 1 -- 172.21.15.81:0/3793201766 wait complete. 2023-11-15T06:32:39.931 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:32:41.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:40 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:32:39.842569+0000 mgr.smithi062.rwurtk (mgr.14164) 209 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:32:41.169 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:40 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:32:39.925565+0000 mon.smithi062 (mon.0) 270 : audit [DBG] from='client.? 172.21.15.81:0/3793201766' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:32:41.281 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:32:41.282 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:32:42.486 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:42.483+0000 7f9da6775700 1 -- 172.21.15.81:0/2881753352 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9da00f6510 msgr2=0x7f9da00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:42.487 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:42.483+0000 7f9da6775700 1 --2- 172.21.15.81:0/2881753352 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9da00f6510 0x7f9da00f6930 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f9d94009fb0 tx=0x7f9d94009b30).stop 2023-11-15T06:32:42.487 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:42.483+0000 7f9da6775700 1 -- 172.21.15.81:0/2881753352 shutdown_connections 2023-11-15T06:32:42.487 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:42.483+0000 7f9da6775700 1 --2- 172.21.15.81:0/2881753352 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9da00f6510 0x7f9da00f6930 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:42.487 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:42.483+0000 7f9da6775700 1 -- 172.21.15.81:0/2881753352 >> 172.21.15.81:0/2881753352 conn(0x7f9da00f1c70 msgr2=0x7f9da00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:32:42.487 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:42.484+0000 7f9da6775700 1 -- 172.21.15.81:0/2881753352 shutdown_connections 2023-11-15T06:32:42.487 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:42.484+0000 7f9da6775700 1 -- 172.21.15.81:0/2881753352 wait complete. 2023-11-15T06:32:42.488 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:42.484+0000 7f9da6775700 1 Processor -- start 2023-11-15T06:32:42.488 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:42.484+0000 7f9da6775700 1 -- start start 2023-11-15T06:32:42.490 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:42.485+0000 7f9da6775700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9da00f6510 0x7f9da0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:32:42.490 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:42.485+0000 7f9da6775700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da0106c20 con 0x7f9da00f6510 2023-11-15T06:32:42.490 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:42.485+0000 7f9d9ffff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9da00f6510 0x7f9da0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:32:42.490 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:42.485+0000 7f9d9ffff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9da00f6510 0x7f9da0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:47770/0 (socket says 172.21.15.81:47770) 2023-11-15T06:32:42.490 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:42.485+0000 7f9d9ffff700 1 -- 172.21.15.81:0/2007664951 learned_addr learned my addr 172.21.15.81:0/2007664951 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:32:42.491 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:42.485+0000 7f9d9ffff700 1 -- 172.21.15.81:0/2007664951 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d94005040 con 0x7f9da00f6510 2023-11-15T06:32:42.491 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:42.485+0000 7f9d9ffff700 1 --2- 172.21.15.81:0/2007664951 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9da00f6510 0x7f9da0106730 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f9d94013510 tx=0x7f9d9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:32:42.491 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:42.486+0000 7f9d9cff9700 1 -- 172.21.15.81:0/2007664951 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d94009320 con 0x7f9da00f6510 2023-11-15T06:32:42.491 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:42.486+0000 7f9da6775700 1 -- 172.21.15.81:0/2007664951 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9da0106e20 con 0x7f9da00f6510 2023-11-15T06:32:42.491 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:42.486+0000 7f9d9cff9700 1 -- 172.21.15.81:0/2007664951 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9d94013f00 con 0x7f9da00f6510 2023-11-15T06:32:42.492 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:42.486+0000 7f9d9cff9700 1 -- 172.21.15.81:0/2007664951 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d94014d90 con 0x7f9da00f6510 2023-11-15T06:32:42.492 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:42.486+0000 7f9da6775700 1 -- 172.21.15.81:0/2007664951 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9da01072b0 con 0x7f9da00f6510 2023-11-15T06:32:42.492 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:42.487+0000 7f9d9cff9700 1 -- 172.21.15.81:0/2007664951 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9d940263a0 con 0x7f9da00f6510 2023-11-15T06:32:42.492 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:42.487+0000 7f9da6775700 1 -- 172.21.15.81:0/2007664951 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9da0045ad0 con 0x7f9da00f6510 2023-11-15T06:32:42.492 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:42.487+0000 7f9d9cff9700 1 --2- 172.21.15.81:0/2007664951 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f9d8802ff60 0x7f9d88032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:32:42.493 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:42.487+0000 7f9d9cff9700 1 -- 172.21.15.81:0/2007664951 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9d94041020 con 0x7f9da00f6510 2023-11-15T06:32:42.493 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:42.487+0000 7f9d9f7fe700 1 --2- 172.21.15.81:0/2007664951 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f9d8802ff60 0x7f9d88032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:32:42.493 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:42.487+0000 7f9d9f7fe700 1 --2- 172.21.15.81:0/2007664951 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f9d8802ff60 0x7f9d88032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f9d900096a0 tx=0x7f9d90006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:32:42.494 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:42.491+0000 7f9d9cff9700 1 -- 172.21.15.81:0/2007664951 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9d940126f0 con 0x7f9da00f6510 2023-11-15T06:32:42.699 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:42.697+0000 7f9da6775700 1 -- 172.21.15.81:0/2007664951 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9da00fbf60 con 0x7f9da00f6510 2023-11-15T06:32:42.699 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:42.697+0000 7f9d9cff9700 1 -- 172.21.15.81:0/2007664951 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9d94012330 con 0x7f9da00f6510 2023-11-15T06:32:42.700 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:32:42.700 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:32:42.701 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:42.699+0000 7f9da6775700 1 -- 172.21.15.81:0/2007664951 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f9d8802ff60 msgr2=0x7f9d88032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:42.701 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:42.699+0000 7f9da6775700 1 --2- 172.21.15.81:0/2007664951 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f9d8802ff60 0x7f9d88032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f9d900096a0 tx=0x7f9d90006b40).stop 2023-11-15T06:32:42.701 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:42.699+0000 7f9da6775700 1 -- 172.21.15.81:0/2007664951 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9da00f6510 msgr2=0x7f9da0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:42.701 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:42.699+0000 7f9da6775700 1 --2- 172.21.15.81:0/2007664951 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9da00f6510 0x7f9da0106730 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f9d94013510 tx=0x7f9d9400b040).stop 2023-11-15T06:32:42.702 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:42.699+0000 7f9da6775700 1 -- 172.21.15.81:0/2007664951 shutdown_connections 2023-11-15T06:32:42.702 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:42.699+0000 7f9da6775700 1 --2- 172.21.15.81:0/2007664951 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f9d8802ff60 0x7f9d88032420 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:42.702 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:42.699+0000 7f9da6775700 1 --2- 172.21.15.81:0/2007664951 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9da00f6510 0x7f9da0106730 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:42.702 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:42.699+0000 7f9da6775700 1 -- 172.21.15.81:0/2007664951 >> 172.21.15.81:0/2007664951 conn(0x7f9da00f1c70 msgr2=0x7f9da00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:32:42.702 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:42.699+0000 7f9da6775700 1 -- 172.21.15.81:0/2007664951 shutdown_connections 2023-11-15T06:32:42.703 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:42.699+0000 7f9da6775700 1 -- 172.21.15.81:0/2007664951 wait complete. 2023-11-15T06:32:42.703 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:32:43.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:42 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:32:41.842896+0000 mgr.smithi062.rwurtk (mgr.14164) 210 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:32:43.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:42 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:32:42.698256+0000 mon.smithi062 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.81:0/2007664951' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:32:44.103 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:32:44.104 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:32:45.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:44 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:32:43.843248+0000 mgr.smithi062.rwurtk (mgr.14164) 211 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:32:45.408 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:45.406+0000 7efe330ac700 1 -- 172.21.15.81:0/2954683198 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe2c0f6510 msgr2=0x7efe2c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:45.409 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:45.406+0000 7efe330ac700 1 --2- 172.21.15.81:0/2954683198 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe2c0f6510 0x7efe2c0f6930 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7efe20009fb0 tx=0x7efe20009b30).stop 2023-11-15T06:32:45.409 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:45.406+0000 7efe330ac700 1 -- 172.21.15.81:0/2954683198 shutdown_connections 2023-11-15T06:32:45.409 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:45.406+0000 7efe330ac700 1 --2- 172.21.15.81:0/2954683198 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe2c0f6510 0x7efe2c0f6930 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:45.409 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:45.406+0000 7efe330ac700 1 -- 172.21.15.81:0/2954683198 >> 172.21.15.81:0/2954683198 conn(0x7efe2c0f1c70 msgr2=0x7efe2c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:32:45.410 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:45.406+0000 7efe330ac700 1 -- 172.21.15.81:0/2954683198 shutdown_connections 2023-11-15T06:32:45.410 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:45.406+0000 7efe330ac700 1 -- 172.21.15.81:0/2954683198 wait complete. 2023-11-15T06:32:45.410 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:45.406+0000 7efe330ac700 1 Processor -- start 2023-11-15T06:32:45.410 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:45.406+0000 7efe330ac700 1 -- start start 2023-11-15T06:32:45.412 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:45.407+0000 7efe330ac700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe2c0f6510 0x7efe2c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:32:45.412 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:45.407+0000 7efe330ac700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe2c106c20 con 0x7efe2c0f6510 2023-11-15T06:32:45.412 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:45.407+0000 7efe30e48700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe2c0f6510 0x7efe2c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:32:45.413 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:45.407+0000 7efe30e48700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe2c0f6510 0x7efe2c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:47790/0 (socket says 172.21.15.81:47790) 2023-11-15T06:32:45.413 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:45.407+0000 7efe30e48700 1 -- 172.21.15.81:0/495138114 learned_addr learned my addr 172.21.15.81:0/495138114 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:32:45.413 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:45.407+0000 7efe30e48700 1 -- 172.21.15.81:0/495138114 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe20005040 con 0x7efe2c0f6510 2023-11-15T06:32:45.413 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:45.407+0000 7efe30e48700 1 --2- 172.21.15.81:0/495138114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe2c0f6510 0x7efe2c106730 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7efe20013510 tx=0x7efe2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:32:45.415 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:45.408+0000 7efe297fa700 1 -- 172.21.15.81:0/495138114 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe20009320 con 0x7efe2c0f6510 2023-11-15T06:32:45.418 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:45.408+0000 7efe330ac700 1 -- 172.21.15.81:0/495138114 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe2c106e20 con 0x7efe2c0f6510 2023-11-15T06:32:45.418 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:45.408+0000 7efe297fa700 1 -- 172.21.15.81:0/495138114 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efe20013f00 con 0x7efe2c0f6510 2023-11-15T06:32:45.418 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:45.408+0000 7efe297fa700 1 -- 172.21.15.81:0/495138114 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe20014d90 con 0x7efe2c0f6510 2023-11-15T06:32:45.418 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:45.408+0000 7efe330ac700 1 -- 172.21.15.81:0/495138114 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe2c1072b0 con 0x7efe2c0f6510 2023-11-15T06:32:45.419 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:45.408+0000 7efe297fa700 1 -- 172.21.15.81:0/495138114 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efe20014420 con 0x7efe2c0f6510 2023-11-15T06:32:45.419 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:45.408+0000 7efe330ac700 1 -- 172.21.15.81:0/495138114 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe2c045ad0 con 0x7efe2c0f6510 2023-11-15T06:32:45.419 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:45.409+0000 7efe297fa700 1 --2- 172.21.15.81:0/495138114 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7efe1c034370 0x7efe1c036830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:32:45.419 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:45.409+0000 7efe297fa700 1 -- 172.21.15.81:0/495138114 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efe20041070 con 0x7efe2c0f6510 2023-11-15T06:32:45.419 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:45.409+0000 7efe2bfff700 1 --2- 172.21.15.81:0/495138114 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7efe1c034370 0x7efe1c036830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:32:45.420 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:45.409+0000 7efe2bfff700 1 --2- 172.21.15.81:0/495138114 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7efe1c034370 0x7efe1c036830 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7efe180096a0 tx=0x7efe18006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:32:45.420 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:45.413+0000 7efe297fa700 1 -- 172.21.15.81:0/495138114 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efe200095f0 con 0x7efe2c0f6510 2023-11-15T06:32:45.618 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:45.616+0000 7efe330ac700 1 -- 172.21.15.81:0/495138114 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efe2c0fbf60 con 0x7efe2c0f6510 2023-11-15T06:32:45.619 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:45.616+0000 7efe297fa700 1 -- 172.21.15.81:0/495138114 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efe2001db70 con 0x7efe2c0f6510 2023-11-15T06:32:45.619 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:32:45.619 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:32:45.620 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:45.618+0000 7efe330ac700 1 -- 172.21.15.81:0/495138114 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7efe1c034370 msgr2=0x7efe1c036830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:45.620 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:45.618+0000 7efe330ac700 1 --2- 172.21.15.81:0/495138114 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7efe1c034370 0x7efe1c036830 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7efe180096a0 tx=0x7efe18006b40).stop 2023-11-15T06:32:45.621 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:45.618+0000 7efe330ac700 1 -- 172.21.15.81:0/495138114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe2c0f6510 msgr2=0x7efe2c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:45.621 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:45.618+0000 7efe330ac700 1 --2- 172.21.15.81:0/495138114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe2c0f6510 0x7efe2c106730 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7efe20013510 tx=0x7efe2000b040).stop 2023-11-15T06:32:45.621 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:45.618+0000 7efe330ac700 1 -- 172.21.15.81:0/495138114 shutdown_connections 2023-11-15T06:32:45.621 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:45.618+0000 7efe330ac700 1 --2- 172.21.15.81:0/495138114 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7efe1c034370 0x7efe1c036830 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:45.621 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:45.618+0000 7efe330ac700 1 --2- 172.21.15.81:0/495138114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe2c0f6510 0x7efe2c106730 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:45.622 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:45.618+0000 7efe330ac700 1 -- 172.21.15.81:0/495138114 >> 172.21.15.81:0/495138114 conn(0x7efe2c0f1c70 msgr2=0x7efe2c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:32:45.622 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:45.618+0000 7efe330ac700 1 -- 172.21.15.81:0/495138114 shutdown_connections 2023-11-15T06:32:45.622 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:45.618+0000 7efe330ac700 1 -- 172.21.15.81:0/495138114 wait complete. 2023-11-15T06:32:45.622 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:32:46.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:45 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:32:45.617355+0000 mon.smithi062 (mon.0) 272 : audit [DBG] from='client.? 172.21.15.81:0/495138114' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:32:47.032 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:32:47.032 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:32:47.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:46 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:32:45.843633+0000 mgr.smithi062.rwurtk (mgr.14164) 212 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:32:48.253 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:48.249+0000 7ff1cc833700 1 -- 172.21.15.81:0/4222904455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1c8060280 msgr2=0x7ff1c80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:48.253 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:48.249+0000 7ff1cc833700 1 --2- 172.21.15.81:0/4222904455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1c8060280 0x7ff1c80f90f0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7ff1b0009fb0 tx=0x7ff1b0009b30).stop 2023-11-15T06:32:48.253 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:48.249+0000 7ff1cc833700 1 -- 172.21.15.81:0/4222904455 shutdown_connections 2023-11-15T06:32:48.254 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:48.249+0000 7ff1cc833700 1 --2- 172.21.15.81:0/4222904455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1c8060280 0x7ff1c80f90f0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:48.254 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:48.249+0000 7ff1cc833700 1 -- 172.21.15.81:0/4222904455 >> 172.21.15.81:0/4222904455 conn(0x7ff1c80f0b70 msgr2=0x7ff1c80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:32:48.254 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:48.249+0000 7ff1cc833700 1 -- 172.21.15.81:0/4222904455 shutdown_connections 2023-11-15T06:32:48.254 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:48.249+0000 7ff1cc833700 1 -- 172.21.15.81:0/4222904455 wait complete. 2023-11-15T06:32:48.255 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:48.249+0000 7ff1cc833700 1 Processor -- start 2023-11-15T06:32:48.255 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:48.249+0000 7ff1cc833700 1 -- start start 2023-11-15T06:32:48.255 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:48.249+0000 7ff1cc833700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1c8060280 0x7ff1c81066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:32:48.255 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:48.249+0000 7ff1cc833700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c8106bd0 con 0x7ff1c8060280 2023-11-15T06:32:48.256 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:48.250+0000 7ff1c659c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1c8060280 0x7ff1c81066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:32:48.257 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:48.250+0000 7ff1c659c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1c8060280 0x7ff1c81066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:47806/0 (socket says 172.21.15.81:47806) 2023-11-15T06:32:48.257 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:48.250+0000 7ff1c659c700 1 -- 172.21.15.81:0/4033893241 learned_addr learned my addr 172.21.15.81:0/4033893241 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:32:48.257 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:48.250+0000 7ff1c659c700 1 -- 172.21.15.81:0/4033893241 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1b0005040 con 0x7ff1c8060280 2023-11-15T06:32:48.258 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:48.250+0000 7ff1c659c700 1 --2- 172.21.15.81:0/4033893241 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1c8060280 0x7ff1c81066e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7ff1b0009830 tx=0x7ff1b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:32:48.258 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:48.250+0000 7ff1beffd700 1 -- 172.21.15.81:0/4033893241 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1b0009320 con 0x7ff1c8060280 2023-11-15T06:32:48.258 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:48.250+0000 7ff1cc833700 1 -- 172.21.15.81:0/4033893241 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1c8106dd0 con 0x7ff1c8060280 2023-11-15T06:32:48.259 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:48.250+0000 7ff1beffd700 1 -- 172.21.15.81:0/4033893241 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff1b00136c0 con 0x7ff1c8060280 2023-11-15T06:32:48.259 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:48.250+0000 7ff1beffd700 1 -- 172.21.15.81:0/4033893241 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1b001ee90 con 0x7ff1c8060280 2023-11-15T06:32:48.259 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:48.250+0000 7ff1cc833700 1 -- 172.21.15.81:0/4033893241 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1c8107260 con 0x7ff1c8060280 2023-11-15T06:32:48.261 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:48.251+0000 7ff1beffd700 1 -- 172.21.15.81:0/4033893241 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff1b001e420 con 0x7ff1c8060280 2023-11-15T06:32:48.261 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:48.251+0000 7ff1cc833700 1 -- 172.21.15.81:0/4033893241 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1c8045ad0 con 0x7ff1c8060280 2023-11-15T06:32:48.261 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:48.251+0000 7ff1beffd700 1 --2- 172.21.15.81:0/4033893241 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff1b402ff10 0x7ff1b40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:32:48.262 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:48.251+0000 7ff1beffd700 1 -- 172.21.15.81:0/4033893241 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff1b00410b0 con 0x7ff1c8060280 2023-11-15T06:32:48.262 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:48.252+0000 7ff1c5d9b700 1 --2- 172.21.15.81:0/4033893241 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff1b402ff10 0x7ff1b40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:32:48.262 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:48.252+0000 7ff1c5d9b700 1 --2- 172.21.15.81:0/4033893241 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff1b402ff10 0x7ff1b40323d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7ff1b80096a0 tx=0x7ff1b8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:32:48.263 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:48.256+0000 7ff1beffd700 1 -- 172.21.15.81:0/4033893241 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff1b00095f0 con 0x7ff1c8060280 2023-11-15T06:32:48.462 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:48.459+0000 7ff1cc833700 1 -- 172.21.15.81:0/4033893241 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff1c8108110 con 0x7ff1c8060280 2023-11-15T06:32:48.463 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:48.460+0000 7ff1beffd700 1 -- 172.21.15.81:0/4033893241 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff1b002c2d0 con 0x7ff1c8060280 2023-11-15T06:32:48.463 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:32:48.463 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:32:48.464 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:48.461+0000 7ff1cc833700 1 -- 172.21.15.81:0/4033893241 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff1b402ff10 msgr2=0x7ff1b40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:48.465 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:48.461+0000 7ff1cc833700 1 --2- 172.21.15.81:0/4033893241 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff1b402ff10 0x7ff1b40323d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7ff1b80096a0 tx=0x7ff1b8006b40).stop 2023-11-15T06:32:48.465 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:48.461+0000 7ff1cc833700 1 -- 172.21.15.81:0/4033893241 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1c8060280 msgr2=0x7ff1c81066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:48.465 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:48.461+0000 7ff1cc833700 1 --2- 172.21.15.81:0/4033893241 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1c8060280 0x7ff1c81066e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7ff1b0009830 tx=0x7ff1b000b040).stop 2023-11-15T06:32:48.465 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:48.462+0000 7ff1cc833700 1 -- 172.21.15.81:0/4033893241 shutdown_connections 2023-11-15T06:32:48.465 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:48.462+0000 7ff1cc833700 1 --2- 172.21.15.81:0/4033893241 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff1b402ff10 0x7ff1b40323d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:48.466 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:48.462+0000 7ff1cc833700 1 --2- 172.21.15.81:0/4033893241 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1c8060280 0x7ff1c81066e0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:48.466 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:48.462+0000 7ff1cc833700 1 -- 172.21.15.81:0/4033893241 >> 172.21.15.81:0/4033893241 conn(0x7ff1c80f0b70 msgr2=0x7ff1c80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:32:48.466 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:48.462+0000 7ff1cc833700 1 -- 172.21.15.81:0/4033893241 shutdown_connections 2023-11-15T06:32:48.466 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:48.462+0000 7ff1cc833700 1 -- 172.21.15.81:0/4033893241 wait complete. 2023-11-15T06:32:48.467 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:32:48.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:48 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:32:47.843860+0000 mgr.smithi062.rwurtk (mgr.14164) 213 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:32:48.919 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:48 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:32:48.460890+0000 mon.smithi062 (mon.0) 273 : audit [DBG] from='client.? 172.21.15.81:0/4033893241' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:32:49.865 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:32:49.866 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:32:51.132 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:51.129+0000 7f355fea1700 1 -- 172.21.15.81:0/1850809207 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35580f6510 msgr2=0x7f35580f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:51.133 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:51.129+0000 7f355fea1700 1 --2- 172.21.15.81:0/1850809207 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35580f6510 0x7f35580f6930 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f3548009fb0 tx=0x7f3548009b30).stop 2023-11-15T06:32:51.133 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:51.129+0000 7f355fea1700 1 -- 172.21.15.81:0/1850809207 shutdown_connections 2023-11-15T06:32:51.133 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:51.129+0000 7f355fea1700 1 --2- 172.21.15.81:0/1850809207 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35580f6510 0x7f35580f6930 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:51.133 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:51.129+0000 7f355fea1700 1 -- 172.21.15.81:0/1850809207 >> 172.21.15.81:0/1850809207 conn(0x7f35580f1c70 msgr2=0x7f35580f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:32:51.134 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:51.129+0000 7f355fea1700 1 -- 172.21.15.81:0/1850809207 shutdown_connections 2023-11-15T06:32:51.134 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:51.129+0000 7f355fea1700 1 -- 172.21.15.81:0/1850809207 wait complete. 2023-11-15T06:32:51.134 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:51.130+0000 7f355fea1700 1 Processor -- start 2023-11-15T06:32:51.134 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:51.130+0000 7f355fea1700 1 -- start start 2023-11-15T06:32:51.135 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:51.130+0000 7f355fea1700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35580f6510 0x7f3558108910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:32:51.135 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:51.130+0000 7f355fea1700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3558108e00 con 0x7f35580f6510 2023-11-15T06:32:51.135 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:51.130+0000 7f355dc3d700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35580f6510 0x7f3558108910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:32:51.135 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:51.130+0000 7f355dc3d700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35580f6510 0x7f3558108910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:47838/0 (socket says 172.21.15.81:47838) 2023-11-15T06:32:51.135 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:51.130+0000 7f355dc3d700 1 -- 172.21.15.81:0/1332976878 learned_addr learned my addr 172.21.15.81:0/1332976878 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:32:51.136 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:51.130+0000 7f355dc3d700 1 -- 172.21.15.81:0/1332976878 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3548005040 con 0x7f35580f6510 2023-11-15T06:32:51.136 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:51.131+0000 7f355dc3d700 1 --2- 172.21.15.81:0/1332976878 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35580f6510 0x7f3558108910 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f3548013510 tx=0x7f354800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:32:51.137 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:51.131+0000 7f354e7fc700 1 -- 172.21.15.81:0/1332976878 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3548009320 con 0x7f35580f6510 2023-11-15T06:32:51.137 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:51.131+0000 7f355fea1700 1 -- 172.21.15.81:0/1332976878 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3558109000 con 0x7f35580f6510 2023-11-15T06:32:51.138 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:51.131+0000 7f354e7fc700 1 -- 172.21.15.81:0/1332976878 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3548013f00 con 0x7f35580f6510 2023-11-15T06:32:51.138 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:51.131+0000 7f354e7fc700 1 -- 172.21.15.81:0/1332976878 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3548014d90 con 0x7f35580f6510 2023-11-15T06:32:51.138 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:51.131+0000 7f355fea1700 1 -- 172.21.15.81:0/1332976878 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3558109490 con 0x7f35580f6510 2023-11-15T06:32:51.138 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:51.132+0000 7f354e7fc700 1 -- 172.21.15.81:0/1332976878 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f35480095f0 con 0x7f35580f6510 2023-11-15T06:32:51.138 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:51.132+0000 7f355fea1700 1 -- 172.21.15.81:0/1332976878 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3558045ad0 con 0x7f35580f6510 2023-11-15T06:32:51.139 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:51.132+0000 7f354e7fc700 1 --2- 172.21.15.81:0/1332976878 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f354402ff60 0x7f3544032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:32:51.139 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:51.132+0000 7f354e7fc700 1 -- 172.21.15.81:0/1332976878 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3548041000 con 0x7f35580f6510 2023-11-15T06:32:51.139 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:51.132+0000 7f355d43c700 1 --2- 172.21.15.81:0/1332976878 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f354402ff60 0x7f3544032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:32:51.140 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:51.132+0000 7f355d43c700 1 --2- 172.21.15.81:0/1332976878 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f354402ff60 0x7f3544032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f35540096a0 tx=0x7f3554006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:32:51.140 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:51.136+0000 7f354e7fc700 1 -- 172.21.15.81:0/1332976878 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3548014420 con 0x7f35580f6510 2023-11-15T06:32:51.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:50 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:32:49.844151+0000 mgr.smithi062.rwurtk (mgr.14164) 214 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:32:51.343 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:51.341+0000 7f355fea1700 1 -- 172.21.15.81:0/1332976878 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f355810a340 con 0x7f35580f6510 2023-11-15T06:32:51.344 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:51.341+0000 7f354e7fc700 1 -- 172.21.15.81:0/1332976878 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3548010230 con 0x7f35580f6510 2023-11-15T06:32:51.344 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:32:51.344 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:32:51.345 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:51.343+0000 7f355fea1700 1 -- 172.21.15.81:0/1332976878 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f354402ff60 msgr2=0x7f3544032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:51.346 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:51.343+0000 7f355fea1700 1 --2- 172.21.15.81:0/1332976878 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f354402ff60 0x7f3544032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f35540096a0 tx=0x7f3554006b40).stop 2023-11-15T06:32:51.346 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:51.343+0000 7f355fea1700 1 -- 172.21.15.81:0/1332976878 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35580f6510 msgr2=0x7f3558108910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:51.346 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:51.343+0000 7f355fea1700 1 --2- 172.21.15.81:0/1332976878 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35580f6510 0x7f3558108910 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f3548013510 tx=0x7f354800b040).stop 2023-11-15T06:32:51.346 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:51.343+0000 7f355fea1700 1 -- 172.21.15.81:0/1332976878 shutdown_connections 2023-11-15T06:32:51.346 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:51.343+0000 7f355fea1700 1 --2- 172.21.15.81:0/1332976878 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f354402ff60 0x7f3544032420 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:51.347 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:51.343+0000 7f355fea1700 1 --2- 172.21.15.81:0/1332976878 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35580f6510 0x7f3558108910 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:51.347 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:51.343+0000 7f355fea1700 1 -- 172.21.15.81:0/1332976878 >> 172.21.15.81:0/1332976878 conn(0x7f35580f1c70 msgr2=0x7f35580f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:32:51.347 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:51.343+0000 7f355fea1700 1 -- 172.21.15.81:0/1332976878 shutdown_connections 2023-11-15T06:32:51.347 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:51.343+0000 7f355fea1700 1 -- 172.21.15.81:0/1332976878 wait complete. 2023-11-15T06:32:51.348 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:32:52.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:51 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:32:51.342325+0000 mon.smithi062 (mon.0) 274 : audit [DBG] from='client.? 172.21.15.81:0/1332976878' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:32:52.728 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:32:52.729 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:32:53.169 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:52 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:32:51.844456+0000 mgr.smithi062.rwurtk (mgr.14164) 215 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:32:53.927 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:53.922+0000 7f1a8051f700 1 -- 172.21.15.81:0/2562073896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a780f5410 msgr2=0x7f1a780f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:53.928 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:53.922+0000 7f1a8051f700 1 --2- 172.21.15.81:0/2562073896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a780f5410 0x7f1a780f5830 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f1a74009fb0 tx=0x7f1a74009b30).stop 2023-11-15T06:32:53.928 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:53.922+0000 7f1a8051f700 1 -- 172.21.15.81:0/2562073896 shutdown_connections 2023-11-15T06:32:53.928 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:53.922+0000 7f1a8051f700 1 --2- 172.21.15.81:0/2562073896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a780f5410 0x7f1a780f5830 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:53.928 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:53.922+0000 7f1a8051f700 1 -- 172.21.15.81:0/2562073896 >> 172.21.15.81:0/2562073896 conn(0x7f1a780f0b70 msgr2=0x7f1a780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:32:53.929 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:53.922+0000 7f1a8051f700 1 -- 172.21.15.81:0/2562073896 shutdown_connections 2023-11-15T06:32:53.929 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:53.922+0000 7f1a8051f700 1 -- 172.21.15.81:0/2562073896 wait complete. 2023-11-15T06:32:53.929 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:53.923+0000 7f1a8051f700 1 Processor -- start 2023-11-15T06:32:53.929 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:53.923+0000 7f1a8051f700 1 -- start start 2023-11-15T06:32:53.929 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:53.923+0000 7f1a8051f700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a780f5410 0x7f1a78105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:32:53.930 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:53.923+0000 7f1a8051f700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a78105b20 con 0x7f1a780f5410 2023-11-15T06:32:53.930 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:53.923+0000 7f1a7e2bb700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a780f5410 0x7f1a78105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:32:53.930 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:53.923+0000 7f1a7e2bb700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a780f5410 0x7f1a78105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:45530/0 (socket says 172.21.15.81:45530) 2023-11-15T06:32:53.930 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:53.923+0000 7f1a7e2bb700 1 -- 172.21.15.81:0/2878667948 learned_addr learned my addr 172.21.15.81:0/2878667948 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:32:53.930 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:53.924+0000 7f1a7e2bb700 1 -- 172.21.15.81:0/2878667948 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a74005040 con 0x7f1a780f5410 2023-11-15T06:32:53.931 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:53.924+0000 7f1a7e2bb700 1 --2- 172.21.15.81:0/2878667948 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a780f5410 0x7f1a78105630 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f1a74013510 tx=0x7f1a7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:32:53.931 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:53.924+0000 7f1a6effd700 1 -- 172.21.15.81:0/2878667948 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a74009320 con 0x7f1a780f5410 2023-11-15T06:32:53.931 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:53.924+0000 7f1a8051f700 1 -- 172.21.15.81:0/2878667948 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a78105d20 con 0x7f1a780f5410 2023-11-15T06:32:53.931 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:53.924+0000 7f1a6effd700 1 -- 172.21.15.81:0/2878667948 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1a74013f00 con 0x7f1a780f5410 2023-11-15T06:32:53.931 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:53.924+0000 7f1a6effd700 1 -- 172.21.15.81:0/2878667948 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a74014d90 con 0x7f1a780f5410 2023-11-15T06:32:53.932 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:53.924+0000 7f1a8051f700 1 -- 172.21.15.81:0/2878667948 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a781061b0 con 0x7f1a780f5410 2023-11-15T06:32:53.932 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:53.925+0000 7f1a6effd700 1 -- 172.21.15.81:0/2878667948 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1a740095f0 con 0x7f1a780f5410 2023-11-15T06:32:53.932 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:53.925+0000 7f1a8051f700 1 -- 172.21.15.81:0/2878667948 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a78045ad0 con 0x7f1a780f5410 2023-11-15T06:32:53.932 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:53.925+0000 7f1a6effd700 1 --2- 172.21.15.81:0/2878667948 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f1a6402ff60 0x7f1a64032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:32:53.932 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:53.925+0000 7f1a6effd700 1 -- 172.21.15.81:0/2878667948 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1a74040fe0 con 0x7f1a780f5410 2023-11-15T06:32:53.933 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:53.925+0000 7f1a7daba700 1 --2- 172.21.15.81:0/2878667948 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f1a6402ff60 0x7f1a64032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:32:53.933 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:53.925+0000 7f1a7daba700 1 --2- 172.21.15.81:0/2878667948 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f1a6402ff60 0x7f1a64032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f1a680096a0 tx=0x7f1a68006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:32:53.933 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:53.929+0000 7f1a6effd700 1 -- 172.21.15.81:0/2878667948 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1a7401d7f0 con 0x7f1a780f5410 2023-11-15T06:32:54.135 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:54.133+0000 7f1a8051f700 1 -- 172.21.15.81:0/2878667948 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1a780fae60 con 0x7f1a780f5410 2023-11-15T06:32:54.135 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:54.133+0000 7f1a6effd700 1 -- 172.21.15.81:0/2878667948 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1a74010230 con 0x7f1a780f5410 2023-11-15T06:32:54.136 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:32:54.137 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:32:54.138 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:54.134+0000 7f1a8051f700 1 -- 172.21.15.81:0/2878667948 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f1a6402ff60 msgr2=0x7f1a64032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:54.138 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:54.134+0000 7f1a8051f700 1 --2- 172.21.15.81:0/2878667948 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f1a6402ff60 0x7f1a64032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f1a680096a0 tx=0x7f1a68006b40).stop 2023-11-15T06:32:54.142 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:54.134+0000 7f1a8051f700 1 -- 172.21.15.81:0/2878667948 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a780f5410 msgr2=0x7f1a78105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:54.142 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:54.134+0000 7f1a8051f700 1 --2- 172.21.15.81:0/2878667948 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a780f5410 0x7f1a78105630 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f1a74013510 tx=0x7f1a7400b040).stop 2023-11-15T06:32:54.142 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:54.135+0000 7f1a8051f700 1 -- 172.21.15.81:0/2878667948 shutdown_connections 2023-11-15T06:32:54.143 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:54.135+0000 7f1a8051f700 1 --2- 172.21.15.81:0/2878667948 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f1a6402ff60 0x7f1a64032420 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:54.143 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:54.135+0000 7f1a8051f700 1 --2- 172.21.15.81:0/2878667948 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1a780f5410 0x7f1a78105630 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:54.143 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:54.135+0000 7f1a8051f700 1 -- 172.21.15.81:0/2878667948 >> 172.21.15.81:0/2878667948 conn(0x7f1a780f0b70 msgr2=0x7f1a780f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:32:54.143 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:54.135+0000 7f1a8051f700 1 -- 172.21.15.81:0/2878667948 shutdown_connections 2023-11-15T06:32:54.144 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:54.135+0000 7f1a8051f700 1 -- 172.21.15.81:0/2878667948 wait complete. 2023-11-15T06:32:54.144 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:32:55.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:54 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:32:53.844746+0000 mgr.smithi062.rwurtk (mgr.14164) 216 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:32:55.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:54 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:32:53.923484+0000 mon.smithi062 (mon.0) 275 : audit [INF] from='mgr.14164 172.21.15.62:0/147479783' entity='mgr.smithi062.rwurtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi062.rwurtk/trash_purge_schedule"}]: dispatch 2023-11-15T06:32:55.169 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:54 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:32:53.923704+0000 mon.smithi062 (mon.0) 276 : audit [INF] from='mgr.14164 172.21.15.62:0/147479783' entity='mgr.smithi062.rwurtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi062.rwurtk/mirror_snapshot_schedule"}]: dispatch 2023-11-15T06:32:55.169 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:54 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:32:54.134094+0000 mon.smithi062 (mon.0) 277 : audit [DBG] from='client.? 172.21.15.81:0/2878667948' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:32:55.555 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:32:55.555 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:32:56.839 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:56.837+0000 7f0e2485c700 1 -- 172.21.15.81:0/2241480849 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e200f62a0 msgr2=0x7f0e200f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:56.839 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:56.837+0000 7f0e2485c700 1 --2- 172.21.15.81:0/2241480849 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e200f62a0 0x7f0e200f66c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f0e10009fb0 tx=0x7f0e10009b30).stop 2023-11-15T06:32:56.840 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:56.837+0000 7f0e2485c700 1 -- 172.21.15.81:0/2241480849 shutdown_connections 2023-11-15T06:32:56.840 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:56.837+0000 7f0e2485c700 1 --2- 172.21.15.81:0/2241480849 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e200f62a0 0x7f0e200f66c0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:56.840 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:56.837+0000 7f0e2485c700 1 -- 172.21.15.81:0/2241480849 >> 172.21.15.81:0/2241480849 conn(0x7f0e200f1aa0 msgr2=0x7f0e200f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:32:56.840 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:56.837+0000 7f0e2485c700 1 -- 172.21.15.81:0/2241480849 shutdown_connections 2023-11-15T06:32:56.841 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:56.837+0000 7f0e2485c700 1 -- 172.21.15.81:0/2241480849 wait complete. 2023-11-15T06:32:56.843 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:56.837+0000 7f0e2485c700 1 Processor -- start 2023-11-15T06:32:56.843 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:56.837+0000 7f0e2485c700 1 -- start start 2023-11-15T06:32:56.843 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:56.838+0000 7f0e2485c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e200f62a0 0x7f0e201064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:32:56.844 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:56.838+0000 7f0e2485c700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e201069a0 con 0x7f0e200f62a0 2023-11-15T06:32:56.844 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:56.838+0000 7f0e1ed9d700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e200f62a0 0x7f0e201064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:32:56.844 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:56.838+0000 7f0e1ed9d700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e200f62a0 0x7f0e201064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:45550/0 (socket says 172.21.15.81:45550) 2023-11-15T06:32:56.844 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:56.838+0000 7f0e1ed9d700 1 -- 172.21.15.81:0/1650473880 learned_addr learned my addr 172.21.15.81:0/1650473880 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:32:56.844 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:56.838+0000 7f0e1ed9d700 1 -- 172.21.15.81:0/1650473880 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e10005040 con 0x7f0e200f62a0 2023-11-15T06:32:56.845 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:56.838+0000 7f0e1ed9d700 1 --2- 172.21.15.81:0/1650473880 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e200f62a0 0x7f0e201064b0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f0e10013510 tx=0x7f0e1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:32:56.845 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:56.838+0000 7f0e077fe700 1 -- 172.21.15.81:0/1650473880 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e10009320 con 0x7f0e200f62a0 2023-11-15T06:32:56.845 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:56.838+0000 7f0e2485c700 1 -- 172.21.15.81:0/1650473880 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e20106ba0 con 0x7f0e200f62a0 2023-11-15T06:32:56.845 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:56.838+0000 7f0e077fe700 1 -- 172.21.15.81:0/1650473880 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0e10013f00 con 0x7f0e200f62a0 2023-11-15T06:32:56.845 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:56.838+0000 7f0e077fe700 1 -- 172.21.15.81:0/1650473880 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e10014dc0 con 0x7f0e200f62a0 2023-11-15T06:32:56.846 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:56.838+0000 7f0e2485c700 1 -- 172.21.15.81:0/1650473880 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e20107030 con 0x7f0e200f62a0 2023-11-15T06:32:56.846 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:56.839+0000 7f0e077fe700 1 -- 172.21.15.81:0/1650473880 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0e1001d3d0 con 0x7f0e200f62a0 2023-11-15T06:32:56.846 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:56.839+0000 7f0e077fe700 1 --2- 172.21.15.81:0/1650473880 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f0e0802ff60 0x7f0e08032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:32:56.846 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:56.839+0000 7f0e077fe700 1 -- 172.21.15.81:0/1650473880 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0e10041000 con 0x7f0e200f62a0 2023-11-15T06:32:56.846 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:56.839+0000 7f0e2485c700 1 -- 172.21.15.81:0/1650473880 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e20045ad0 con 0x7f0e200f62a0 2023-11-15T06:32:56.847 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:56.840+0000 7f0e1e59c700 1 --2- 172.21.15.81:0/1650473880 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f0e0802ff60 0x7f0e08032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:32:56.847 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:56.840+0000 7f0e1e59c700 1 --2- 172.21.15.81:0/1650473880 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f0e0802ff60 0x7f0e08032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f0e140096a0 tx=0x7f0e14006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:32:56.847 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:56.844+0000 7f0e077fe700 1 -- 172.21.15.81:0/1650473880 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0e10026490 con 0x7f0e200f62a0 2023-11-15T06:32:57.048 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:57.046+0000 7f0e2485c700 1 -- 172.21.15.81:0/1650473880 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0e20107d50 con 0x7f0e200f62a0 2023-11-15T06:32:57.049 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:57.047+0000 7f0e077fe700 1 -- 172.21.15.81:0/1650473880 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0e100126f0 con 0x7f0e200f62a0 2023-11-15T06:32:57.049 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:32:57.050 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:32:57.051 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:57.049+0000 7f0e2485c700 1 -- 172.21.15.81:0/1650473880 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f0e0802ff60 msgr2=0x7f0e08032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:57.051 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:57.049+0000 7f0e2485c700 1 --2- 172.21.15.81:0/1650473880 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f0e0802ff60 0x7f0e08032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f0e140096a0 tx=0x7f0e14006b40).stop 2023-11-15T06:32:57.051 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:57.049+0000 7f0e2485c700 1 -- 172.21.15.81:0/1650473880 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e200f62a0 msgr2=0x7f0e201064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:57.051 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:57.049+0000 7f0e2485c700 1 --2- 172.21.15.81:0/1650473880 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e200f62a0 0x7f0e201064b0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f0e10013510 tx=0x7f0e1000b040).stop 2023-11-15T06:32:57.052 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:57.049+0000 7f0e2485c700 1 -- 172.21.15.81:0/1650473880 shutdown_connections 2023-11-15T06:32:57.052 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:57.049+0000 7f0e2485c700 1 --2- 172.21.15.81:0/1650473880 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f0e0802ff60 0x7f0e08032420 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:57.052 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:57.049+0000 7f0e2485c700 1 --2- 172.21.15.81:0/1650473880 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e200f62a0 0x7f0e201064b0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:57.052 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:57.049+0000 7f0e2485c700 1 -- 172.21.15.81:0/1650473880 >> 172.21.15.81:0/1650473880 conn(0x7f0e200f1aa0 msgr2=0x7f0e200f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:32:57.052 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:57.049+0000 7f0e2485c700 1 -- 172.21.15.81:0/1650473880 shutdown_connections 2023-11-15T06:32:57.053 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:57.049+0000 7f0e2485c700 1 -- 172.21.15.81:0/1650473880 wait complete. 2023-11-15T06:32:57.053 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:32:57.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:56 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:32:55.845063+0000 mgr.smithi062.rwurtk (mgr.14164) 217 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:32:58.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:57 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:32:57.048015+0000 mon.smithi062 (mon.0) 278 : audit [DBG] from='client.? 172.21.15.81:0/1650473880' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:32:58.447 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:32:58.447 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:32:59.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:58 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:32:57.845388+0000 mgr.smithi062.rwurtk (mgr.14164) 218 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:32:59.751 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:59.697+0000 7f16ec7fc700 1 -- 172.21.15.81:0/3806276498 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16e40f6510 msgr2=0x7f16e40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:59.752 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:59.697+0000 7f16ec7fc700 1 --2- 172.21.15.81:0/3806276498 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16e40f6510 0x7f16e40f6930 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f16d4009fb0 tx=0x7f16d4009b30).stop 2023-11-15T06:32:59.752 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:59.697+0000 7f16ec7fc700 1 -- 172.21.15.81:0/3806276498 shutdown_connections 2023-11-15T06:32:59.752 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:59.697+0000 7f16ec7fc700 1 --2- 172.21.15.81:0/3806276498 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16e40f6510 0x7f16e40f6930 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:59.752 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:59.697+0000 7f16ec7fc700 1 -- 172.21.15.81:0/3806276498 >> 172.21.15.81:0/3806276498 conn(0x7f16e40f1c70 msgr2=0x7f16e40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:32:59.753 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:59.697+0000 7f16ec7fc700 1 -- 172.21.15.81:0/3806276498 shutdown_connections 2023-11-15T06:32:59.753 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:59.697+0000 7f16ec7fc700 1 -- 172.21.15.81:0/3806276498 wait complete. 2023-11-15T06:32:59.753 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:59.698+0000 7f16ec7fc700 1 Processor -- start 2023-11-15T06:32:59.753 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:59.698+0000 7f16ec7fc700 1 -- start start 2023-11-15T06:32:59.753 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:59.698+0000 7f16ec7fc700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16e40f6510 0x7f16e4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:32:59.754 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:59.698+0000 7f16ec7fc700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16e4106c20 con 0x7f16e40f6510 2023-11-15T06:32:59.754 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:59.698+0000 7f16ea598700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16e40f6510 0x7f16e4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:32:59.754 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:59.698+0000 7f16ea598700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16e40f6510 0x7f16e4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:45574/0 (socket says 172.21.15.81:45574) 2023-11-15T06:32:59.754 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:59.698+0000 7f16ea598700 1 -- 172.21.15.81:0/2937450078 learned_addr learned my addr 172.21.15.81:0/2937450078 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:32:59.755 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:59.699+0000 7f16ea598700 1 -- 172.21.15.81:0/2937450078 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16d4005040 con 0x7f16e40f6510 2023-11-15T06:32:59.755 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:59.699+0000 7f16ea598700 1 --2- 172.21.15.81:0/2937450078 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16e40f6510 0x7f16e4106730 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f16d4013510 tx=0x7f16d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:32:59.755 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:59.699+0000 7f16daffd700 1 -- 172.21.15.81:0/2937450078 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16d4009320 con 0x7f16e40f6510 2023-11-15T06:32:59.755 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:59.699+0000 7f16daffd700 1 -- 172.21.15.81:0/2937450078 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f16d4013f00 con 0x7f16e40f6510 2023-11-15T06:32:59.755 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:59.699+0000 7f16ec7fc700 1 -- 172.21.15.81:0/2937450078 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16e4106e20 con 0x7f16e40f6510 2023-11-15T06:32:59.755 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:59.699+0000 7f16daffd700 1 -- 172.21.15.81:0/2937450078 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16d4014d90 con 0x7f16e40f6510 2023-11-15T06:32:59.756 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:59.699+0000 7f16ec7fc700 1 -- 172.21.15.81:0/2937450078 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16e41072b0 con 0x7f16e40f6510 2023-11-15T06:32:59.756 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:59.700+0000 7f16daffd700 1 -- 172.21.15.81:0/2937450078 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f16d40263a0 con 0x7f16e40f6510 2023-11-15T06:32:59.756 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:59.700+0000 7f16daffd700 1 --2- 172.21.15.81:0/2937450078 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f16d002ffb0 0x7f16d0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:32:59.756 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:59.700+0000 7f16ec7fc700 1 -- 172.21.15.81:0/2937450078 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16e4045ad0 con 0x7f16e40f6510 2023-11-15T06:32:59.756 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:59.700+0000 7f16daffd700 1 -- 172.21.15.81:0/2937450078 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f16d40410d0 con 0x7f16e40f6510 2023-11-15T06:32:59.757 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:59.700+0000 7f16e9d97700 1 --2- 172.21.15.81:0/2937450078 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f16d002ffb0 0x7f16d0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:32:59.757 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:59.701+0000 7f16e9d97700 1 --2- 172.21.15.81:0/2937450078 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f16d002ffb0 0x7f16d0032470 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f16e00096a0 tx=0x7f16e0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:32:59.757 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:59.705+0000 7f16daffd700 1 -- 172.21.15.81:0/2937450078 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f16d406d2b0 con 0x7f16e40f6510 2023-11-15T06:32:59.910 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:59.908+0000 7f16ec7fc700 1 -- 172.21.15.81:0/2937450078 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f16e40fc050 con 0x7f16e40f6510 2023-11-15T06:32:59.911 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:59.908+0000 7f16daffd700 1 -- 172.21.15.81:0/2937450078 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f16d40126f0 con 0x7f16e40f6510 2023-11-15T06:32:59.911 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:32:59.911 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:32:59.912 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:59.910+0000 7f16ec7fc700 1 -- 172.21.15.81:0/2937450078 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f16d002ffb0 msgr2=0x7f16d0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:59.912 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:59.910+0000 7f16ec7fc700 1 --2- 172.21.15.81:0/2937450078 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f16d002ffb0 0x7f16d0032470 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f16e00096a0 tx=0x7f16e0006b40).stop 2023-11-15T06:32:59.913 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:59.910+0000 7f16ec7fc700 1 -- 172.21.15.81:0/2937450078 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16e40f6510 msgr2=0x7f16e4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:32:59.913 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:59.910+0000 7f16ec7fc700 1 --2- 172.21.15.81:0/2937450078 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16e40f6510 0x7f16e4106730 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f16d4013510 tx=0x7f16d400b040).stop 2023-11-15T06:32:59.913 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:59.910+0000 7f16ec7fc700 1 -- 172.21.15.81:0/2937450078 shutdown_connections 2023-11-15T06:32:59.913 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:59.910+0000 7f16ec7fc700 1 --2- 172.21.15.81:0/2937450078 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f16d002ffb0 0x7f16d0032470 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:59.913 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:59.910+0000 7f16ec7fc700 1 --2- 172.21.15.81:0/2937450078 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16e40f6510 0x7f16e4106730 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:32:59.914 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:59.910+0000 7f16ec7fc700 1 -- 172.21.15.81:0/2937450078 >> 172.21.15.81:0/2937450078 conn(0x7f16e40f1c70 msgr2=0x7f16e40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:32:59.914 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:59.910+0000 7f16ec7fc700 1 -- 172.21.15.81:0/2937450078 shutdown_connections 2023-11-15T06:32:59.914 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:32:59.910+0000 7f16ec7fc700 1 -- 172.21.15.81:0/2937450078 wait complete. 2023-11-15T06:32:59.915 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:33:00.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:32:59 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:32:59.909480+0000 mon.smithi062 (mon.0) 279 : audit [DBG] from='client.? 172.21.15.81:0/2937450078' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:33:01.345 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:33:01.346 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:33:01.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:00 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:32:59.845751+0000 mgr.smithi062.rwurtk (mgr.14164) 219 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:33:02.624 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:02.621+0000 7f862fa7a700 1 -- 172.21.15.81:0/3704860779 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86280f6510 msgr2=0x7f86280f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:02.625 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:02.621+0000 7f862fa7a700 1 --2- 172.21.15.81:0/3704860779 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86280f6510 0x7f86280f6930 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f8618009fb0 tx=0x7f8618009b30).stop 2023-11-15T06:33:02.625 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:02.622+0000 7f862fa7a700 1 -- 172.21.15.81:0/3704860779 shutdown_connections 2023-11-15T06:33:02.625 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:02.622+0000 7f862fa7a700 1 --2- 172.21.15.81:0/3704860779 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86280f6510 0x7f86280f6930 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:02.625 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:02.622+0000 7f862fa7a700 1 -- 172.21.15.81:0/3704860779 >> 172.21.15.81:0/3704860779 conn(0x7f86280f1c70 msgr2=0x7f86280f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:33:02.625 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:02.622+0000 7f862fa7a700 1 -- 172.21.15.81:0/3704860779 shutdown_connections 2023-11-15T06:33:02.626 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:02.622+0000 7f862fa7a700 1 -- 172.21.15.81:0/3704860779 wait complete. 2023-11-15T06:33:02.626 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:02.622+0000 7f862fa7a700 1 Processor -- start 2023-11-15T06:33:02.626 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:02.622+0000 7f862fa7a700 1 -- start start 2023-11-15T06:33:02.627 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:02.622+0000 7f862fa7a700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86281067b0 0x7f8628106bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:33:02.627 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:02.622+0000 7f862fa7a700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86281070c0 con 0x7f86281067b0 2023-11-15T06:33:02.630 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:02.623+0000 7f862d816700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86281067b0 0x7f8628106bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:33:02.630 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:02.623+0000 7f862d816700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86281067b0 0x7f8628106bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:57894/0 (socket says 172.21.15.81:57894) 2023-11-15T06:33:02.630 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:02.623+0000 7f862d816700 1 -- 172.21.15.81:0/2250527581 learned_addr learned my addr 172.21.15.81:0/2250527581 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:33:02.631 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:02.623+0000 7f862d816700 1 -- 172.21.15.81:0/2250527581 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8618005040 con 0x7f86281067b0 2023-11-15T06:33:02.631 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:02.623+0000 7f862d816700 1 --2- 172.21.15.81:0/2250527581 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86281067b0 0x7f8628106bd0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f8618009f80 tx=0x7f8618009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:33:02.631 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:02.623+0000 7f861e7fc700 1 -- 172.21.15.81:0/2250527581 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86180136c0 con 0x7f86281067b0 2023-11-15T06:33:02.632 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:02.623+0000 7f861e7fc700 1 -- 172.21.15.81:0/2250527581 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f86180095f0 con 0x7f86281067b0 2023-11-15T06:33:02.632 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:02.623+0000 7f862fa7a700 1 -- 172.21.15.81:0/2250527581 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86281072c0 con 0x7f86281067b0 2023-11-15T06:33:02.633 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:02.623+0000 7f861e7fc700 1 -- 172.21.15.81:0/2250527581 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8618014420 con 0x7f86281067b0 2023-11-15T06:33:02.633 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:02.623+0000 7f862fa7a700 1 -- 172.21.15.81:0/2250527581 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8628107e10 con 0x7f86281067b0 2023-11-15T06:33:02.633 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:02.624+0000 7f862fa7a700 1 -- 172.21.15.81:0/2250527581 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8628059100 con 0x7f86281067b0 2023-11-15T06:33:02.634 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:02.624+0000 7f861e7fc700 1 -- 172.21.15.81:0/2250527581 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8618025060 con 0x7f86281067b0 2023-11-15T06:33:02.634 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:02.624+0000 7f861e7fc700 1 --2- 172.21.15.81:0/2250527581 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f861402ff60 0x7f8614032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:33:02.634 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:02.624+0000 7f861e7fc700 1 -- 172.21.15.81:0/2250527581 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8618040e60 con 0x7f86281067b0 2023-11-15T06:33:02.635 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:02.625+0000 7f862d015700 1 --2- 172.21.15.81:0/2250527581 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f861402ff60 0x7f8614032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:33:02.635 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:02.625+0000 7f862d015700 1 --2- 172.21.15.81:0/2250527581 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f861402ff60 0x7f8614032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f86240096a0 tx=0x7f8624006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:33:02.636 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:02.629+0000 7f861e7fc700 1 -- 172.21.15.81:0/2250527581 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f861801cb60 con 0x7f86281067b0 2023-11-15T06:33:02.834 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:02.831+0000 7f862fa7a700 1 -- 172.21.15.81:0/2250527581 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8628045ad0 con 0x7f86281067b0 2023-11-15T06:33:02.834 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:02.832+0000 7f861e7fc700 1 -- 172.21.15.81:0/2250527581 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f861800f280 con 0x7f86281067b0 2023-11-15T06:33:02.835 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:33:02.835 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:33:02.836 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:02.834+0000 7f862fa7a700 1 -- 172.21.15.81:0/2250527581 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f861402ff60 msgr2=0x7f8614032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:02.836 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:02.834+0000 7f862fa7a700 1 --2- 172.21.15.81:0/2250527581 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f861402ff60 0x7f8614032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f86240096a0 tx=0x7f8624006b40).stop 2023-11-15T06:33:02.836 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:02.834+0000 7f862fa7a700 1 -- 172.21.15.81:0/2250527581 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86281067b0 msgr2=0x7f8628106bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:02.836 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:02.834+0000 7f862fa7a700 1 --2- 172.21.15.81:0/2250527581 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86281067b0 0x7f8628106bd0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f8618009f80 tx=0x7f8618009b30).stop 2023-11-15T06:33:02.837 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:02.834+0000 7f862fa7a700 1 -- 172.21.15.81:0/2250527581 shutdown_connections 2023-11-15T06:33:02.837 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:02.834+0000 7f862fa7a700 1 --2- 172.21.15.81:0/2250527581 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f861402ff60 0x7f8614032420 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:02.837 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:02.834+0000 7f862fa7a700 1 --2- 172.21.15.81:0/2250527581 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86281067b0 0x7f8628106bd0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:02.837 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:02.834+0000 7f862fa7a700 1 -- 172.21.15.81:0/2250527581 >> 172.21.15.81:0/2250527581 conn(0x7f86280f1c70 msgr2=0x7f86280f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:33:02.837 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:02.834+0000 7f862fa7a700 1 -- 172.21.15.81:0/2250527581 shutdown_connections 2023-11-15T06:33:02.838 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:02.834+0000 7f862fa7a700 1 -- 172.21.15.81:0/2250527581 wait complete. 2023-11-15T06:33:02.838 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:33:03.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:33:01.846103+0000 mgr.smithi062.rwurtk (mgr.14164) 220 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:33:03.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:33:02.833122+0000 mon.smithi062 (mon.0) 280 : audit [DBG] from='client.? 172.21.15.81:0/2250527581' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:33:04.245 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:33:04.246 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:33:05.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:04 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:33:03.846505+0000 mgr.smithi062.rwurtk (mgr.14164) 221 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:33:05.483 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:05.479+0000 7fd19754d700 1 -- 172.21.15.81:0/3101984533 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1900f6510 msgr2=0x7fd1900f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:05.483 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:05.479+0000 7fd19754d700 1 --2- 172.21.15.81:0/3101984533 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1900f6510 0x7fd1900f6930 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7fd180009fb0 tx=0x7fd180009b30).stop 2023-11-15T06:33:05.483 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:05.479+0000 7fd19754d700 1 -- 172.21.15.81:0/3101984533 shutdown_connections 2023-11-15T06:33:05.484 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:05.479+0000 7fd19754d700 1 --2- 172.21.15.81:0/3101984533 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1900f6510 0x7fd1900f6930 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:05.484 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:05.479+0000 7fd19754d700 1 -- 172.21.15.81:0/3101984533 >> 172.21.15.81:0/3101984533 conn(0x7fd1900f1c70 msgr2=0x7fd1900f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:33:05.484 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:05.479+0000 7fd19754d700 1 -- 172.21.15.81:0/3101984533 shutdown_connections 2023-11-15T06:33:05.484 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:05.479+0000 7fd19754d700 1 -- 172.21.15.81:0/3101984533 wait complete. 2023-11-15T06:33:05.485 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:05.480+0000 7fd19754d700 1 Processor -- start 2023-11-15T06:33:05.485 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:05.480+0000 7fd19754d700 1 -- start start 2023-11-15T06:33:05.485 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:05.480+0000 7fd19754d700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1900f6510 0x7fd190106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:33:05.485 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:05.480+0000 7fd19754d700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd190106c20 con 0x7fd1900f6510 2023-11-15T06:33:05.486 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:05.480+0000 7fd1952e9700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1900f6510 0x7fd190106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:33:05.486 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:05.480+0000 7fd1952e9700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1900f6510 0x7fd190106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:57910/0 (socket says 172.21.15.81:57910) 2023-11-15T06:33:05.486 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:05.480+0000 7fd1952e9700 1 -- 172.21.15.81:0/2030277127 learned_addr learned my addr 172.21.15.81:0/2030277127 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:33:05.486 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:05.480+0000 7fd1952e9700 1 -- 172.21.15.81:0/2030277127 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd180005040 con 0x7fd1900f6510 2023-11-15T06:33:05.486 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:05.481+0000 7fd1952e9700 1 --2- 172.21.15.81:0/2030277127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1900f6510 0x7fd190106730 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7fd180013510 tx=0x7fd18000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:33:05.487 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:05.481+0000 7fd185ffb700 1 -- 172.21.15.81:0/2030277127 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd180009320 con 0x7fd1900f6510 2023-11-15T06:33:05.487 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:05.481+0000 7fd19754d700 1 -- 172.21.15.81:0/2030277127 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd190106e20 con 0x7fd1900f6510 2023-11-15T06:33:05.487 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:05.481+0000 7fd185ffb700 1 -- 172.21.15.81:0/2030277127 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd180013f00 con 0x7fd1900f6510 2023-11-15T06:33:05.487 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:05.481+0000 7fd185ffb700 1 -- 172.21.15.81:0/2030277127 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd180014d90 con 0x7fd1900f6510 2023-11-15T06:33:05.488 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:05.481+0000 7fd19754d700 1 -- 172.21.15.81:0/2030277127 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1901072b0 con 0x7fd1900f6510 2023-11-15T06:33:05.489 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:05.482+0000 7fd185ffb700 1 -- 172.21.15.81:0/2030277127 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd1800095f0 con 0x7fd1900f6510 2023-11-15T06:33:05.489 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:05.482+0000 7fd19754d700 1 -- 172.21.15.81:0/2030277127 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd190045ad0 con 0x7fd1900f6510 2023-11-15T06:33:05.489 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:05.482+0000 7fd185ffb700 1 --2- 172.21.15.81:0/2030277127 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fd17c02ff60 0x7fd17c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:33:05.489 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:05.482+0000 7fd185ffb700 1 -- 172.21.15.81:0/2030277127 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd180040fc0 con 0x7fd1900f6510 2023-11-15T06:33:05.490 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:05.482+0000 7fd194ae8700 1 --2- 172.21.15.81:0/2030277127 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fd17c02ff60 0x7fd17c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:33:05.490 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:05.483+0000 7fd194ae8700 1 --2- 172.21.15.81:0/2030277127 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fd17c02ff60 0x7fd17c032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fd18c0096a0 tx=0x7fd18c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:33:05.490 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:05.487+0000 7fd185ffb700 1 -- 172.21.15.81:0/2030277127 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd180014420 con 0x7fd1900f6510 2023-11-15T06:33:05.697 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:05.691+0000 7fd19754d700 1 -- 172.21.15.81:0/2030277127 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd1900fbf60 con 0x7fd1900f6510 2023-11-15T06:33:05.697 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:05.692+0000 7fd185ffb700 1 -- 172.21.15.81:0/2030277127 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd180010230 con 0x7fd1900f6510 2023-11-15T06:33:05.698 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:33:05.698 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:33:05.699 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:05.694+0000 7fd19754d700 1 -- 172.21.15.81:0/2030277127 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fd17c02ff60 msgr2=0x7fd17c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:05.699 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:05.694+0000 7fd19754d700 1 --2- 172.21.15.81:0/2030277127 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fd17c02ff60 0x7fd17c032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fd18c0096a0 tx=0x7fd18c006b40).stop 2023-11-15T06:33:05.700 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:05.694+0000 7fd19754d700 1 -- 172.21.15.81:0/2030277127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1900f6510 msgr2=0x7fd190106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:05.700 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:05.694+0000 7fd19754d700 1 --2- 172.21.15.81:0/2030277127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1900f6510 0x7fd190106730 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7fd180013510 tx=0x7fd18000b040).stop 2023-11-15T06:33:05.700 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:05.694+0000 7fd19754d700 1 -- 172.21.15.81:0/2030277127 shutdown_connections 2023-11-15T06:33:05.700 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:05.694+0000 7fd19754d700 1 --2- 172.21.15.81:0/2030277127 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fd17c02ff60 0x7fd17c032420 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:05.700 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:05.694+0000 7fd19754d700 1 --2- 172.21.15.81:0/2030277127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd1900f6510 0x7fd190106730 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:05.700 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:05.694+0000 7fd19754d700 1 -- 172.21.15.81:0/2030277127 >> 172.21.15.81:0/2030277127 conn(0x7fd1900f1c70 msgr2=0x7fd1900f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:33:05.701 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:05.694+0000 7fd19754d700 1 -- 172.21.15.81:0/2030277127 shutdown_connections 2023-11-15T06:33:05.701 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:05.694+0000 7fd19754d700 1 -- 172.21.15.81:0/2030277127 wait complete. 2023-11-15T06:33:05.701 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:33:06.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:05 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:33:05.693197+0000 mon.smithi062 (mon.0) 281 : audit [DBG] from='client.? 172.21.15.81:0/2030277127' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:33:07.066 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:33:07.067 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:33:07.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:06 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:33:05.846795+0000 mgr.smithi062.rwurtk (mgr.14164) 222 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:33:08.229 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:08.225+0000 7f4413781700 1 -- 172.21.15.81:0/3037905325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f440c0f5410 msgr2=0x7f440c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:08.229 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:08.225+0000 7f4413781700 1 --2- 172.21.15.81:0/3037905325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f440c0f5410 0x7f440c0f5830 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f43fc009fb0 tx=0x7f43fc009b30).stop 2023-11-15T06:33:08.230 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:08.225+0000 7f4413781700 1 -- 172.21.15.81:0/3037905325 shutdown_connections 2023-11-15T06:33:08.230 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:08.225+0000 7f4413781700 1 --2- 172.21.15.81:0/3037905325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f440c0f5410 0x7f440c0f5830 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:08.230 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:08.225+0000 7f4413781700 1 -- 172.21.15.81:0/3037905325 >> 172.21.15.81:0/3037905325 conn(0x7f440c0f0b70 msgr2=0x7f440c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:33:08.230 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:08.225+0000 7f4413781700 1 -- 172.21.15.81:0/3037905325 shutdown_connections 2023-11-15T06:33:08.230 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:08.225+0000 7f4413781700 1 -- 172.21.15.81:0/3037905325 wait complete. 2023-11-15T06:33:08.231 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:08.226+0000 7f4413781700 1 Processor -- start 2023-11-15T06:33:08.231 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:08.226+0000 7f4413781700 1 -- start start 2023-11-15T06:33:08.231 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:08.226+0000 7f4413781700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f440c0f5410 0x7f440c109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:33:08.231 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:08.226+0000 7f4413781700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f440c060240 con 0x7f440c0f5410 2023-11-15T06:33:08.231 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:08.226+0000 7f441151d700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f440c0f5410 0x7f440c109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:33:08.232 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:08.226+0000 7f441151d700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f440c0f5410 0x7f440c109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:57930/0 (socket says 172.21.15.81:57930) 2023-11-15T06:33:08.232 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:08.226+0000 7f441151d700 1 -- 172.21.15.81:0/3495708597 learned_addr learned my addr 172.21.15.81:0/3495708597 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:33:08.232 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:08.227+0000 7f441151d700 1 -- 172.21.15.81:0/3495708597 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43fc005040 con 0x7f440c0f5410 2023-11-15T06:33:08.232 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:08.227+0000 7f441151d700 1 --2- 172.21.15.81:0/3495708597 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f440c0f5410 0x7f440c109430 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f43fc013510 tx=0x7f43fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:33:08.232 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:08.227+0000 7f4401ffb700 1 -- 172.21.15.81:0/3495708597 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43fc009320 con 0x7f440c0f5410 2023-11-15T06:33:08.233 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:08.227+0000 7f4413781700 1 -- 172.21.15.81:0/3495708597 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f440c060440 con 0x7f440c0f5410 2023-11-15T06:33:08.233 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:08.227+0000 7f4401ffb700 1 -- 172.21.15.81:0/3495708597 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f43fc013f00 con 0x7f440c0f5410 2023-11-15T06:33:08.233 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:08.227+0000 7f4401ffb700 1 -- 172.21.15.81:0/3495708597 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43fc014d90 con 0x7f440c0f5410 2023-11-15T06:33:08.233 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:08.227+0000 7f4413781700 1 -- 172.21.15.81:0/3495708597 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f440c109ae0 con 0x7f440c0f5410 2023-11-15T06:33:08.234 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:08.228+0000 7f4401ffb700 1 -- 172.21.15.81:0/3495708597 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f43fc0095f0 con 0x7f440c0f5410 2023-11-15T06:33:08.235 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:08.228+0000 7f4413781700 1 -- 172.21.15.81:0/3495708597 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f440c045ad0 con 0x7f440c0f5410 2023-11-15T06:33:08.235 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:08.228+0000 7f4401ffb700 1 --2- 172.21.15.81:0/3495708597 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f43f802ff60 0x7f43f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:33:08.235 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:08.228+0000 7f4401ffb700 1 -- 172.21.15.81:0/3495708597 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f43fc041080 con 0x7f440c0f5410 2023-11-15T06:33:08.235 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:08.228+0000 7f4410d1c700 1 --2- 172.21.15.81:0/3495708597 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f43f802ff60 0x7f43f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:33:08.236 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:08.229+0000 7f4410d1c700 1 --2- 172.21.15.81:0/3495708597 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f43f802ff60 0x7f43f8032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f44080096a0 tx=0x7f4408006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:33:08.236 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:08.233+0000 7f4401ffb700 1 -- 172.21.15.81:0/3495708597 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f43fc01d7f0 con 0x7f440c0f5410 2023-11-15T06:33:08.441 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:08.439+0000 7f4413781700 1 -- 172.21.15.81:0/3495708597 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f440c0fae60 con 0x7f440c0f5410 2023-11-15T06:33:08.441 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:08.439+0000 7f4401ffb700 1 -- 172.21.15.81:0/3495708597 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f43fc010230 con 0x7f440c0f5410 2023-11-15T06:33:08.442 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:33:08.442 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:33:08.443 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:08.441+0000 7f4413781700 1 -- 172.21.15.81:0/3495708597 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f43f802ff60 msgr2=0x7f43f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:08.444 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:08.441+0000 7f4413781700 1 --2- 172.21.15.81:0/3495708597 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f43f802ff60 0x7f43f8032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f44080096a0 tx=0x7f4408006b40).stop 2023-11-15T06:33:08.444 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:08.441+0000 7f4413781700 1 -- 172.21.15.81:0/3495708597 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f440c0f5410 msgr2=0x7f440c109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:08.444 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:08.441+0000 7f4413781700 1 --2- 172.21.15.81:0/3495708597 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f440c0f5410 0x7f440c109430 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f43fc013510 tx=0x7f43fc00b040).stop 2023-11-15T06:33:08.444 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:08.441+0000 7f4413781700 1 -- 172.21.15.81:0/3495708597 shutdown_connections 2023-11-15T06:33:08.444 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:08.441+0000 7f4413781700 1 --2- 172.21.15.81:0/3495708597 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f43f802ff60 0x7f43f8032420 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:08.445 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:08.441+0000 7f4413781700 1 --2- 172.21.15.81:0/3495708597 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f440c0f5410 0x7f440c109430 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:08.445 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:08.441+0000 7f4413781700 1 -- 172.21.15.81:0/3495708597 >> 172.21.15.81:0/3495708597 conn(0x7f440c0f0b70 msgr2=0x7f440c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:33:08.445 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:08.441+0000 7f4413781700 1 -- 172.21.15.81:0/3495708597 shutdown_connections 2023-11-15T06:33:08.445 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:08.441+0000 7f4413781700 1 -- 172.21.15.81:0/3495708597 wait complete. 2023-11-15T06:33:08.446 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:33:08.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:08 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:33:07.847083+0000 mgr.smithi062.rwurtk (mgr.14164) 223 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:33:08.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:08 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:33:08.440264+0000 mon.smithi062 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.81:0/3495708597' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:33:09.867 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:33:09.868 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:33:11.119 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:11.115+0000 7fa55e83c700 1 -- 172.21.15.81:0/3408869673 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5580f51a0 msgr2=0x7fa5580f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:11.120 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:11.115+0000 7fa55e83c700 1 --2- 172.21.15.81:0/3408869673 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5580f51a0 0x7fa5580f55c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7fa554009fb0 tx=0x7fa554009b30).stop 2023-11-15T06:33:11.120 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:11.115+0000 7fa55e83c700 1 -- 172.21.15.81:0/3408869673 shutdown_connections 2023-11-15T06:33:11.120 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:11.115+0000 7fa55e83c700 1 --2- 172.21.15.81:0/3408869673 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5580f51a0 0x7fa5580f55c0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:11.120 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:11.115+0000 7fa55e83c700 1 -- 172.21.15.81:0/3408869673 >> 172.21.15.81:0/3408869673 conn(0x7fa5580f09a0 msgr2=0x7fa5580f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:33:11.121 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:11.115+0000 7fa55e83c700 1 -- 172.21.15.81:0/3408869673 shutdown_connections 2023-11-15T06:33:11.121 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:11.115+0000 7fa55e83c700 1 -- 172.21.15.81:0/3408869673 wait complete. 2023-11-15T06:33:11.121 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:11.115+0000 7fa55e83c700 1 Processor -- start 2023-11-15T06:33:11.121 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:11.115+0000 7fa55e83c700 1 -- start start 2023-11-15T06:33:11.121 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:11.115+0000 7fa55e83c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5580f51a0 0x7fa55810b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:33:11.122 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:11.115+0000 7fa55e83c700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa558106fc0 con 0x7fa5580f51a0 2023-11-15T06:33:11.122 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:11.116+0000 7fa55d83a700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5580f51a0 0x7fa55810b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:33:11.122 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:11.116+0000 7fa55d83a700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5580f51a0 0x7fa55810b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:57944/0 (socket says 172.21.15.81:57944) 2023-11-15T06:33:11.122 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:11.116+0000 7fa55d83a700 1 -- 172.21.15.81:0/164995291 learned_addr learned my addr 172.21.15.81:0/164995291 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:33:11.123 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:11.116+0000 7fa55d83a700 1 -- 172.21.15.81:0/164995291 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa554005040 con 0x7fa5580f51a0 2023-11-15T06:33:11.123 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:11.116+0000 7fa55d83a700 1 --2- 172.21.15.81:0/164995291 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5580f51a0 0x7fa55810b5d0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7fa554013510 tx=0x7fa55400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:33:11.123 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:11.116+0000 7fa54e7fc700 1 -- 172.21.15.81:0/164995291 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa554009320 con 0x7fa5580f51a0 2023-11-15T06:33:11.123 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:11.116+0000 7fa55e83c700 1 -- 172.21.15.81:0/164995291 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa558060220 con 0x7fa5580f51a0 2023-11-15T06:33:11.123 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:11.116+0000 7fa54e7fc700 1 -- 172.21.15.81:0/164995291 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa554013f00 con 0x7fa5580f51a0 2023-11-15T06:33:11.124 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:11.116+0000 7fa54e7fc700 1 -- 172.21.15.81:0/164995291 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa554014dc0 con 0x7fa5580f51a0 2023-11-15T06:33:11.124 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:11.116+0000 7fa55e83c700 1 -- 172.21.15.81:0/164995291 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa55810bac0 con 0x7fa5580f51a0 2023-11-15T06:33:11.125 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:11.117+0000 7fa54e7fc700 1 -- 172.21.15.81:0/164995291 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa55401d3d0 con 0x7fa5580f51a0 2023-11-15T06:33:11.125 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:11.117+0000 7fa54e7fc700 1 --2- 172.21.15.81:0/164995291 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fa54402ff60 0x7fa544032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:33:11.125 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:11.117+0000 7fa54e7fc700 1 -- 172.21.15.81:0/164995291 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa5540413c0 con 0x7fa5580f51a0 2023-11-15T06:33:11.125 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:11.117+0000 7fa55e83c700 1 -- 172.21.15.81:0/164995291 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa558045ad0 con 0x7fa5580f51a0 2023-11-15T06:33:11.126 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:11.117+0000 7fa55d039700 1 --2- 172.21.15.81:0/164995291 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fa54402ff60 0x7fa544032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:33:11.126 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:11.118+0000 7fa55d039700 1 --2- 172.21.15.81:0/164995291 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fa54402ff60 0x7fa544032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fa5480096a0 tx=0x7fa548006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:33:11.126 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:11.122+0000 7fa54e7fc700 1 -- 172.21.15.81:0/164995291 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa554012330 con 0x7fa5580f51a0 2023-11-15T06:33:11.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:10 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:33:09.847465+0000 mgr.smithi062.rwurtk (mgr.14164) 224 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:33:11.329 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:11.326+0000 7fa55e83c700 1 -- 172.21.15.81:0/164995291 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa55810c280 con 0x7fa5580f51a0 2023-11-15T06:33:11.330 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:11.327+0000 7fa54e7fc700 1 -- 172.21.15.81:0/164995291 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa55402b030 con 0x7fa5580f51a0 2023-11-15T06:33:11.330 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:33:11.331 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:33:11.331 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:11.328+0000 7fa55e83c700 1 -- 172.21.15.81:0/164995291 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fa54402ff60 msgr2=0x7fa544032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:11.332 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:11.328+0000 7fa55e83c700 1 --2- 172.21.15.81:0/164995291 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fa54402ff60 0x7fa544032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fa5480096a0 tx=0x7fa548006b40).stop 2023-11-15T06:33:11.332 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:11.328+0000 7fa55e83c700 1 -- 172.21.15.81:0/164995291 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5580f51a0 msgr2=0x7fa55810b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:11.332 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:11.328+0000 7fa55e83c700 1 --2- 172.21.15.81:0/164995291 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5580f51a0 0x7fa55810b5d0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7fa554013510 tx=0x7fa55400b040).stop 2023-11-15T06:33:11.332 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:11.328+0000 7fa55e83c700 1 -- 172.21.15.81:0/164995291 shutdown_connections 2023-11-15T06:33:11.333 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:11.328+0000 7fa55e83c700 1 --2- 172.21.15.81:0/164995291 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fa54402ff60 0x7fa544032420 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:11.333 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:11.328+0000 7fa55e83c700 1 --2- 172.21.15.81:0/164995291 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5580f51a0 0x7fa55810b5d0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:11.333 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:11.328+0000 7fa55e83c700 1 -- 172.21.15.81:0/164995291 >> 172.21.15.81:0/164995291 conn(0x7fa5580f09a0 msgr2=0x7fa5580f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:33:11.333 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:11.328+0000 7fa55e83c700 1 -- 172.21.15.81:0/164995291 shutdown_connections 2023-11-15T06:33:11.334 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:11.328+0000 7fa55e83c700 1 -- 172.21.15.81:0/164995291 wait complete. 2023-11-15T06:33:11.334 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:33:12.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:11 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:33:11.328621+0000 mon.smithi062 (mon.0) 283 : audit [DBG] from='client.? 172.21.15.81:0/164995291' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:33:12.712 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:33:12.712 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:33:13.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:12 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:33:11.847820+0000 mgr.smithi062.rwurtk (mgr.14164) 225 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:33:13.923 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:13.918+0000 7f3d72a46700 1 -- 172.21.15.81:0/2473156689 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d6c0f5410 msgr2=0x7f3d6c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:13.925 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:13.918+0000 7f3d72a46700 1 --2- 172.21.15.81:0/2473156689 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d6c0f5410 0x7f3d6c0f5830 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f3d5c009fb0 tx=0x7f3d5c009b30).stop 2023-11-15T06:33:13.926 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:13.918+0000 7f3d72a46700 1 -- 172.21.15.81:0/2473156689 shutdown_connections 2023-11-15T06:33:13.926 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:13.918+0000 7f3d72a46700 1 --2- 172.21.15.81:0/2473156689 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d6c0f5410 0x7f3d6c0f5830 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:13.926 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:13.918+0000 7f3d72a46700 1 -- 172.21.15.81:0/2473156689 >> 172.21.15.81:0/2473156689 conn(0x7f3d6c0f0b70 msgr2=0x7f3d6c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:33:13.927 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:13.918+0000 7f3d72a46700 1 -- 172.21.15.81:0/2473156689 shutdown_connections 2023-11-15T06:33:13.927 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:13.918+0000 7f3d72a46700 1 -- 172.21.15.81:0/2473156689 wait complete. 2023-11-15T06:33:13.927 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:13.919+0000 7f3d72a46700 1 Processor -- start 2023-11-15T06:33:13.927 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:13.919+0000 7f3d72a46700 1 -- start start 2023-11-15T06:33:13.928 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:13.919+0000 7f3d72a46700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d6c0f5410 0x7f3d6c109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:33:13.928 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:13.919+0000 7f3d72a46700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6c060240 con 0x7f3d6c0f5410 2023-11-15T06:33:13.928 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:13.919+0000 7f3d6bfff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d6c0f5410 0x7f3d6c109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:33:13.928 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:13.919+0000 7f3d6bfff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d6c0f5410 0x7f3d6c109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:37882/0 (socket says 172.21.15.81:37882) 2023-11-15T06:33:13.928 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:13.919+0000 7f3d6bfff700 1 -- 172.21.15.81:0/3738686211 learned_addr learned my addr 172.21.15.81:0/3738686211 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:33:13.929 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:13.920+0000 7f3d6bfff700 1 -- 172.21.15.81:0/3738686211 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d5c005040 con 0x7f3d6c0f5410 2023-11-15T06:33:13.930 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:13.920+0000 7f3d6bfff700 1 --2- 172.21.15.81:0/3738686211 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d6c0f5410 0x7f3d6c109430 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f3d5c013510 tx=0x7f3d5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:33:13.930 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:13.920+0000 7f3d68ff9700 1 -- 172.21.15.81:0/3738686211 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d5c009320 con 0x7f3d6c0f5410 2023-11-15T06:33:13.930 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:13.920+0000 7f3d72a46700 1 -- 172.21.15.81:0/3738686211 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d6c060440 con 0x7f3d6c0f5410 2023-11-15T06:33:13.930 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:13.920+0000 7f3d68ff9700 1 -- 172.21.15.81:0/3738686211 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3d5c013f00 con 0x7f3d6c0f5410 2023-11-15T06:33:13.930 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:13.920+0000 7f3d68ff9700 1 -- 172.21.15.81:0/3738686211 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d5c014d90 con 0x7f3d6c0f5410 2023-11-15T06:33:13.931 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:13.920+0000 7f3d72a46700 1 -- 172.21.15.81:0/3738686211 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d6c109ae0 con 0x7f3d6c0f5410 2023-11-15T06:33:13.932 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:13.921+0000 7f3d68ff9700 1 -- 172.21.15.81:0/3738686211 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3d5c0263a0 con 0x7f3d6c0f5410 2023-11-15T06:33:13.932 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:13.921+0000 7f3d72a46700 1 -- 172.21.15.81:0/3738686211 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d6c045ad0 con 0x7f3d6c0f5410 2023-11-15T06:33:13.932 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:13.921+0000 7f3d68ff9700 1 --2- 172.21.15.81:0/3738686211 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f3d5802ff60 0x7f3d58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:33:13.933 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:13.921+0000 7f3d68ff9700 1 -- 172.21.15.81:0/3738686211 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3d5c0410c0 con 0x7f3d6c0f5410 2023-11-15T06:33:13.933 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:13.921+0000 7f3d6b7fe700 1 --2- 172.21.15.81:0/3738686211 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f3d5802ff60 0x7f3d58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:33:13.933 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:13.921+0000 7f3d6b7fe700 1 --2- 172.21.15.81:0/3738686211 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f3d5802ff60 0x7f3d58032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f3d540096a0 tx=0x7f3d54006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:33:13.933 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:13.926+0000 7f3d68ff9700 1 -- 172.21.15.81:0/3738686211 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3d5c0126f0 con 0x7f3d6c0f5410 2023-11-15T06:33:14.134 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:14.130+0000 7f3d72a46700 1 -- 172.21.15.81:0/3738686211 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3d6c0fae60 con 0x7f3d6c0f5410 2023-11-15T06:33:14.134 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:14.131+0000 7f3d68ff9700 1 -- 172.21.15.81:0/3738686211 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3d5c012330 con 0x7f3d6c0f5410 2023-11-15T06:33:14.135 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:33:14.135 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:33:14.136 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:14.133+0000 7f3d72a46700 1 -- 172.21.15.81:0/3738686211 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f3d5802ff60 msgr2=0x7f3d58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:14.136 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:14.133+0000 7f3d72a46700 1 --2- 172.21.15.81:0/3738686211 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f3d5802ff60 0x7f3d58032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f3d540096a0 tx=0x7f3d54006b40).stop 2023-11-15T06:33:14.136 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:14.133+0000 7f3d72a46700 1 -- 172.21.15.81:0/3738686211 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d6c0f5410 msgr2=0x7f3d6c109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:14.136 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:14.133+0000 7f3d72a46700 1 --2- 172.21.15.81:0/3738686211 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d6c0f5410 0x7f3d6c109430 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f3d5c013510 tx=0x7f3d5c00b040).stop 2023-11-15T06:33:14.137 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:14.133+0000 7f3d72a46700 1 -- 172.21.15.81:0/3738686211 shutdown_connections 2023-11-15T06:33:14.137 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:14.133+0000 7f3d72a46700 1 --2- 172.21.15.81:0/3738686211 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f3d5802ff60 0x7f3d58032420 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:14.137 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:14.133+0000 7f3d72a46700 1 --2- 172.21.15.81:0/3738686211 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d6c0f5410 0x7f3d6c109430 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:14.137 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:14.133+0000 7f3d72a46700 1 -- 172.21.15.81:0/3738686211 >> 172.21.15.81:0/3738686211 conn(0x7f3d6c0f0b70 msgr2=0x7f3d6c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:33:14.138 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:14.133+0000 7f3d72a46700 1 -- 172.21.15.81:0/3738686211 shutdown_connections 2023-11-15T06:33:14.138 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:14.133+0000 7f3d72a46700 1 -- 172.21.15.81:0/3738686211 wait complete. 2023-11-15T06:33:14.138 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:33:15.169 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:14 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:33:13.848204+0000 mgr.smithi062.rwurtk (mgr.14164) 226 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:33:15.169 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:14 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:33:14.132999+0000 mon.smithi062 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.81:0/3738686211' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:33:15.523 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:33:15.524 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:33:16.699 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:16.692+0000 7f6094539700 1 -- 172.21.15.81:0/1933017592 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f608c0f6510 msgr2=0x7f608c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:16.699 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:16.692+0000 7f6094539700 1 --2- 172.21.15.81:0/1933017592 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f608c0f6510 0x7f608c0f6930 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f607c009fb0 tx=0x7f607c009b30).stop 2023-11-15T06:33:16.700 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:16.692+0000 7f6094539700 1 -- 172.21.15.81:0/1933017592 shutdown_connections 2023-11-15T06:33:16.700 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:16.692+0000 7f6094539700 1 --2- 172.21.15.81:0/1933017592 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f608c0f6510 0x7f608c0f6930 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:16.700 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:16.692+0000 7f6094539700 1 -- 172.21.15.81:0/1933017592 >> 172.21.15.81:0/1933017592 conn(0x7f608c0f1c70 msgr2=0x7f608c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:33:16.701 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:16.692+0000 7f6094539700 1 -- 172.21.15.81:0/1933017592 shutdown_connections 2023-11-15T06:33:16.701 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:16.692+0000 7f6094539700 1 -- 172.21.15.81:0/1933017592 wait complete. 2023-11-15T06:33:16.701 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:16.693+0000 7f6094539700 1 Processor -- start 2023-11-15T06:33:16.702 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:16.693+0000 7f6094539700 1 -- start start 2023-11-15T06:33:16.702 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:16.693+0000 7f6094539700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f608c0f6510 0x7f608c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:33:16.702 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:16.693+0000 7f6094539700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f608c106c20 con 0x7f608c0f6510 2023-11-15T06:33:16.703 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:16.693+0000 7f60922d5700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f608c0f6510 0x7f608c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:33:16.704 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:16.693+0000 7f60922d5700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f608c0f6510 0x7f608c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:37904/0 (socket says 172.21.15.81:37904) 2023-11-15T06:33:16.704 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:16.693+0000 7f60922d5700 1 -- 172.21.15.81:0/2314445031 learned_addr learned my addr 172.21.15.81:0/2314445031 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:33:16.704 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:16.694+0000 7f60922d5700 1 -- 172.21.15.81:0/2314445031 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f607c005040 con 0x7f608c0f6510 2023-11-15T06:33:16.705 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:16.694+0000 7f60922d5700 1 --2- 172.21.15.81:0/2314445031 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f608c0f6510 0x7f608c106730 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f607c013510 tx=0x7f607c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:33:16.705 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:16.694+0000 7f6082ffd700 1 -- 172.21.15.81:0/2314445031 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f607c009320 con 0x7f608c0f6510 2023-11-15T06:33:16.705 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:16.694+0000 7f6094539700 1 -- 172.21.15.81:0/2314445031 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f608c106e20 con 0x7f608c0f6510 2023-11-15T06:33:16.705 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:16.694+0000 7f6082ffd700 1 -- 172.21.15.81:0/2314445031 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f607c013f00 con 0x7f608c0f6510 2023-11-15T06:33:16.705 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:16.694+0000 7f6082ffd700 1 -- 172.21.15.81:0/2314445031 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f607c014d90 con 0x7f608c0f6510 2023-11-15T06:33:16.706 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:16.694+0000 7f6094539700 1 -- 172.21.15.81:0/2314445031 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f608c1072b0 con 0x7f608c0f6510 2023-11-15T06:33:16.706 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:16.695+0000 7f6082ffd700 1 -- 172.21.15.81:0/2314445031 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f607c014420 con 0x7f608c0f6510 2023-11-15T06:33:16.706 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:16.695+0000 7f6082ffd700 1 --2- 172.21.15.81:0/2314445031 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f607802ff60 0x7f6078032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:33:16.707 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:16.695+0000 7f6082ffd700 1 -- 172.21.15.81:0/2314445031 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f607c041050 con 0x7f608c0f6510 2023-11-15T06:33:16.707 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:16.695+0000 7f6094539700 1 -- 172.21.15.81:0/2314445031 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f608c045ad0 con 0x7f608c0f6510 2023-11-15T06:33:16.707 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:16.695+0000 7f6091ad4700 1 --2- 172.21.15.81:0/2314445031 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f607802ff60 0x7f6078032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:33:16.707 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:16.695+0000 7f6091ad4700 1 --2- 172.21.15.81:0/2314445031 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f607802ff60 0x7f6078032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f60880096a0 tx=0x7f6088006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:33:16.708 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:16.700+0000 7f6082ffd700 1 -- 172.21.15.81:0/2314445031 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f607c0095f0 con 0x7f608c0f6510 2023-11-15T06:33:16.909 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:16.906+0000 7f6094539700 1 -- 172.21.15.81:0/2314445031 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f608c0fc050 con 0x7f608c0f6510 2023-11-15T06:33:16.911 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:16.908+0000 7f6082ffd700 1 -- 172.21.15.81:0/2314445031 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f607c02b020 con 0x7f608c0f6510 2023-11-15T06:33:16.912 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:33:16.912 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:33:16.913 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:16.909+0000 7f6094539700 1 -- 172.21.15.81:0/2314445031 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f607802ff60 msgr2=0x7f6078032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:16.913 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:16.909+0000 7f6094539700 1 --2- 172.21.15.81:0/2314445031 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f607802ff60 0x7f6078032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f60880096a0 tx=0x7f6088006b40).stop 2023-11-15T06:33:16.914 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:16.909+0000 7f6094539700 1 -- 172.21.15.81:0/2314445031 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f608c0f6510 msgr2=0x7f608c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:16.914 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:16.909+0000 7f6094539700 1 --2- 172.21.15.81:0/2314445031 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f608c0f6510 0x7f608c106730 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f607c013510 tx=0x7f607c00b040).stop 2023-11-15T06:33:16.914 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:16.909+0000 7f6094539700 1 -- 172.21.15.81:0/2314445031 shutdown_connections 2023-11-15T06:33:16.914 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:16.909+0000 7f6094539700 1 --2- 172.21.15.81:0/2314445031 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f607802ff60 0x7f6078032420 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:16.915 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:16.909+0000 7f6094539700 1 --2- 172.21.15.81:0/2314445031 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f608c0f6510 0x7f608c106730 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:16.915 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:16.909+0000 7f6094539700 1 -- 172.21.15.81:0/2314445031 >> 172.21.15.81:0/2314445031 conn(0x7f608c0f1c70 msgr2=0x7f608c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:33:16.915 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:16.910+0000 7f6094539700 1 -- 172.21.15.81:0/2314445031 shutdown_connections 2023-11-15T06:33:16.916 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:16.910+0000 7f6094539700 1 -- 172.21.15.81:0/2314445031 wait complete. 2023-11-15T06:33:16.916 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:33:17.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:16 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:33:15.848470+0000 mgr.smithi062.rwurtk (mgr.14164) 227 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:33:18.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:17 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:33:16.909826+0000 mon.smithi062 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.81:0/2314445031' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:33:18.344 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:33:18.345 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:33:19.169 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:33:17.848820+0000 mgr.smithi062.rwurtk (mgr.14164) 228 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:33:19.169 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cephadm 2023-11-15T06:33:17.957202+0000 mgr.smithi062.rwurtk (mgr.14164) 229 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi062 ... 2023-11-15T06:33:19.170 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: Verifying port 9100 ... 2023-11-15T06:33:19.170 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: Non-zero exit code 1 from systemctl start ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7@node-exporter.smithi062 2023-11-15T06:33:19.170 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: systemctl: stderr Job for ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7@node-exporter.smithi062.service failed because a timeout was exceeded. 2023-11-15T06:33:19.171 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: systemctl: stderr See "systemctl status ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7@node-exporter.smithi062.service" and "journalctl -xe" for details. 2023-11-15T06:33:19.171 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: Traceback (most recent call last): 2023-11-15T06:33:19.171 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: File "/var/lib/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-15T06:33:19.172 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: main() 2023-11-15T06:33:19.172 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: File "/var/lib/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-15T06:33:19.173 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: r = ctx.func(ctx) 2023-11-15T06:33:19.173 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: File "/var/lib/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-15T06:33:19.173 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: return func(ctx) 2023-11-15T06:33:19.174 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: File "/var/lib/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-11-15T06:33:19.174 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: ports=daemon_ports) 2023-11-15T06:33:19.174 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: File "/var/lib/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2023-11-15T06:33:19.174 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: c, osd_fsid=osd_fsid, ports=ports) 2023-11-15T06:33:19.175 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: File "/var/lib/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2023-11-15T06:33:19.175 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-11-15T06:33:19.175 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: File "/var/lib/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-11-15T06:33:19.175 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-11-15T06:33:19.176 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: RuntimeError: Failed command: systemctl start ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7@node-exporter.smithi062 2023-11-15T06:33:19.176 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: Traceback (most recent call last): 2023-11-15T06:33:19.176 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-11-15T06:33:19.176 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: yield (conn, connr) 2023-11-15T06:33:19.177 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-11-15T06:33:19.177 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: code, '\n'.join(err))) 2023-11-15T06:33:19.177 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi062 ... 2023-11-15T06:33:19.177 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: Verifying port 9100 ... 2023-11-15T06:33:19.178 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: Non-zero exit code 1 from systemctl start ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7@node-exporter.smithi062 2023-11-15T06:33:19.178 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: systemctl: stderr Job for ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7@node-exporter.smithi062.service failed because a timeout was exceeded. 2023-11-15T06:33:19.178 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: systemctl: stderr See "systemctl status ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7@node-exporter.smithi062.service" and "journalctl -xe" for details. 2023-11-15T06:33:19.179 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: Traceback (most recent call last): 2023-11-15T06:33:19.179 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: File "/var/lib/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-15T06:33:19.179 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: main() 2023-11-15T06:33:19.180 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: File "/var/lib/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-15T06:33:19.180 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: r = ctx.func(ctx) 2023-11-15T06:33:19.180 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: File "/var/lib/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-15T06:33:19.181 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: return func(ctx) 2023-11-15T06:33:19.181 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: File "/var/lib/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-11-15T06:33:19.182 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: ports=daemon_ports) 2023-11-15T06:33:19.182 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: File "/var/lib/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2023-11-15T06:33:19.182 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: c, osd_fsid=osd_fsid, ports=ports) 2023-11-15T06:33:19.183 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: File "/var/lib/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2023-11-15T06:33:19.183 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-11-15T06:33:19.183 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: File "/var/lib/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-11-15T06:33:19.184 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-11-15T06:33:19.184 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: RuntimeError: Failed command: systemctl start ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7@node-exporter.smithi062 2023-11-15T06:33:19.184 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cephadm 2023-11-15T06:33:17.962779+0000 mgr.smithi062.rwurtk (mgr.14164) 230 : cephadm [INF] Deploying daemon prometheus.smithi062 on smithi062 2023-11-15T06:33:19.639 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:19.636+0000 7fcbda5cd700 1 -- 172.21.15.81:0/2432933255 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbd40f4740 msgr2=0x7fcbd40f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:19.640 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:19.636+0000 7fcbda5cd700 1 --2- 172.21.15.81:0/2432933255 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbd40f4740 0x7fcbd40f4b60 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7fcbc8009fb0 tx=0x7fcbc8009b30).stop 2023-11-15T06:33:19.640 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:19.636+0000 7fcbda5cd700 1 -- 172.21.15.81:0/2432933255 shutdown_connections 2023-11-15T06:33:19.640 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:19.636+0000 7fcbda5cd700 1 --2- 172.21.15.81:0/2432933255 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbd40f4740 0x7fcbd40f4b60 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:19.640 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:19.636+0000 7fcbda5cd700 1 -- 172.21.15.81:0/2432933255 >> 172.21.15.81:0/2432933255 conn(0x7fcbd40f0260 msgr2=0x7fcbd40f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:33:19.641 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:19.636+0000 7fcbda5cd700 1 -- 172.21.15.81:0/2432933255 shutdown_connections 2023-11-15T06:33:19.641 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:19.636+0000 7fcbda5cd700 1 -- 172.21.15.81:0/2432933255 wait complete. 2023-11-15T06:33:19.641 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:19.636+0000 7fcbda5cd700 1 Processor -- start 2023-11-15T06:33:19.641 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:19.636+0000 7fcbda5cd700 1 -- start start 2023-11-15T06:33:19.641 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:19.637+0000 7fcbda5cd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbd40f4740 0x7fcbd41066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:33:19.642 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:19.637+0000 7fcbda5cd700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd4106bc0 con 0x7fcbd40f4740 2023-11-15T06:33:19.644 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:19.637+0000 7fcbd3fff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbd40f4740 0x7fcbd41066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:33:19.644 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:19.637+0000 7fcbd3fff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbd40f4740 0x7fcbd41066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:37916/0 (socket says 172.21.15.81:37916) 2023-11-15T06:33:19.644 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:19.637+0000 7fcbd3fff700 1 -- 172.21.15.81:0/3199007610 learned_addr learned my addr 172.21.15.81:0/3199007610 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:33:19.644 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:19.637+0000 7fcbd3fff700 1 -- 172.21.15.81:0/3199007610 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbc8005040 con 0x7fcbd40f4740 2023-11-15T06:33:19.644 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:19.637+0000 7fcbd3fff700 1 --2- 172.21.15.81:0/3199007610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbd40f4740 0x7fcbd41066d0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7fcbc8013510 tx=0x7fcbc800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:33:19.645 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:19.638+0000 7fcbd0ff9700 1 -- 172.21.15.81:0/3199007610 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcbc8009320 con 0x7fcbd40f4740 2023-11-15T06:33:19.645 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:19.638+0000 7fcbda5cd700 1 -- 172.21.15.81:0/3199007610 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbd4106dc0 con 0x7fcbd40f4740 2023-11-15T06:33:19.645 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:19.638+0000 7fcbd0ff9700 1 -- 172.21.15.81:0/3199007610 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcbc8013f00 con 0x7fcbd40f4740 2023-11-15T06:33:19.645 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:19.638+0000 7fcbd0ff9700 1 -- 172.21.15.81:0/3199007610 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcbc8014d90 con 0x7fcbd40f4740 2023-11-15T06:33:19.645 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:19.638+0000 7fcbda5cd700 1 -- 172.21.15.81:0/3199007610 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbd4107250 con 0x7fcbd40f4740 2023-11-15T06:33:19.646 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:19.639+0000 7fcbd0ff9700 1 -- 172.21.15.81:0/3199007610 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcbc80095f0 con 0x7fcbd40f4740 2023-11-15T06:33:19.646 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:19.639+0000 7fcbda5cd700 1 -- 172.21.15.81:0/3199007610 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbd4045ad0 con 0x7fcbd40f4740 2023-11-15T06:33:19.646 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:19.639+0000 7fcbd0ff9700 1 --2- 172.21.15.81:0/3199007610 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fcbbc02ff60 0x7fcbbc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:33:19.647 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:19.639+0000 7fcbd0ff9700 1 -- 172.21.15.81:0/3199007610 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcbc8041000 con 0x7fcbd40f4740 2023-11-15T06:33:19.647 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:19.639+0000 7fcbd37fe700 1 --2- 172.21.15.81:0/3199007610 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fcbbc02ff60 0x7fcbbc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:33:19.647 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:19.639+0000 7fcbd37fe700 1 --2- 172.21.15.81:0/3199007610 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fcbbc02ff60 0x7fcbbc032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fcbc40096a0 tx=0x7fcbc4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:33:19.648 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:19.644+0000 7fcbd0ff9700 1 -- 172.21.15.81:0/3199007610 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcbc8014420 con 0x7fcbd40f4740 2023-11-15T06:33:19.851 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:19.848+0000 7fcbda5cd700 1 -- 172.21.15.81:0/3199007610 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcbd4108100 con 0x7fcbd40f4740 2023-11-15T06:33:19.852 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:19.849+0000 7fcbd0ff9700 1 -- 172.21.15.81:0/3199007610 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcbc801d7f0 con 0x7fcbd40f4740 2023-11-15T06:33:19.852 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:33:19.852 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:33:19.853 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:19.850+0000 7fcbda5cd700 1 -- 172.21.15.81:0/3199007610 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fcbbc02ff60 msgr2=0x7fcbbc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:19.854 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:19.850+0000 7fcbda5cd700 1 --2- 172.21.15.81:0/3199007610 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fcbbc02ff60 0x7fcbbc032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fcbc40096a0 tx=0x7fcbc4006b40).stop 2023-11-15T06:33:19.854 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:19.850+0000 7fcbda5cd700 1 -- 172.21.15.81:0/3199007610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbd40f4740 msgr2=0x7fcbd41066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:19.854 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:19.850+0000 7fcbda5cd700 1 --2- 172.21.15.81:0/3199007610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbd40f4740 0x7fcbd41066d0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7fcbc8013510 tx=0x7fcbc800b040).stop 2023-11-15T06:33:19.855 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:19.850+0000 7fcbda5cd700 1 -- 172.21.15.81:0/3199007610 shutdown_connections 2023-11-15T06:33:19.855 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:19.850+0000 7fcbda5cd700 1 --2- 172.21.15.81:0/3199007610 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fcbbc02ff60 0x7fcbbc032420 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:19.855 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:19.850+0000 7fcbda5cd700 1 --2- 172.21.15.81:0/3199007610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbd40f4740 0x7fcbd41066d0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:19.855 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:19.850+0000 7fcbda5cd700 1 -- 172.21.15.81:0/3199007610 >> 172.21.15.81:0/3199007610 conn(0x7fcbd40f0260 msgr2=0x7fcbd40f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:33:19.855 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:19.850+0000 7fcbda5cd700 1 -- 172.21.15.81:0/3199007610 shutdown_connections 2023-11-15T06:33:19.856 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:19.850+0000 7fcbda5cd700 1 -- 172.21.15.81:0/3199007610 wait complete. 2023-11-15T06:33:19.856 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:33:20.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:19 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:33:19.850556+0000 mon.smithi062 (mon.0) 286 : audit [DBG] from='client.? 172.21.15.81:0/3199007610' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:33:21.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:20 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:33:19.849188+0000 mgr.smithi062.rwurtk (mgr.14164) 231 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:33:21.234 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:33:21.234 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:33:22.445 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:22.441+0000 7fcc9759e700 1 -- 172.21.15.81:0/3090944251 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc980f51a0 msgr2=0x7fcc980f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:22.445 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:22.441+0000 7fcc9759e700 1 --2- 172.21.15.81:0/3090944251 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc980f51a0 0x7fcc980f55c0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7fcc8c009fb0 tx=0x7fcc8c009b30).stop 2023-11-15T06:33:22.445 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:22.441+0000 7fcc9759e700 1 -- 172.21.15.81:0/3090944251 shutdown_connections 2023-11-15T06:33:22.446 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:22.441+0000 7fcc9759e700 1 --2- 172.21.15.81:0/3090944251 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc980f51a0 0x7fcc980f55c0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:22.446 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:22.441+0000 7fcc9759e700 1 -- 172.21.15.81:0/3090944251 >> 172.21.15.81:0/3090944251 conn(0x7fcc980f09a0 msgr2=0x7fcc980f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:33:22.446 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:22.441+0000 7fcc9759e700 1 -- 172.21.15.81:0/3090944251 shutdown_connections 2023-11-15T06:33:22.446 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:22.441+0000 7fcc9759e700 1 -- 172.21.15.81:0/3090944251 wait complete. 2023-11-15T06:33:22.446 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:22.441+0000 7fcc9759e700 1 Processor -- start 2023-11-15T06:33:22.447 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:22.441+0000 7fcc9759e700 1 -- start start 2023-11-15T06:33:22.447 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:22.441+0000 7fcc9759e700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc980f51a0 0x7fcc981091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:33:22.447 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:22.441+0000 7fcc9759e700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc98060220 con 0x7fcc980f51a0 2023-11-15T06:33:22.447 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:22.442+0000 7fcc9659c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc980f51a0 0x7fcc981091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:33:22.447 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:22.442+0000 7fcc9659c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc980f51a0 0x7fcc981091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:53762/0 (socket says 172.21.15.81:53762) 2023-11-15T06:33:22.448 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:22.442+0000 7fcc9659c700 1 -- 172.21.15.81:0/3524706504 learned_addr learned my addr 172.21.15.81:0/3524706504 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:33:22.448 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:22.442+0000 7fcc9659c700 1 -- 172.21.15.81:0/3524706504 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc8c005040 con 0x7fcc980f51a0 2023-11-15T06:33:22.448 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:22.442+0000 7fcc9659c700 1 --2- 172.21.15.81:0/3524706504 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc980f51a0 0x7fcc981091d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fcc8c009830 tx=0x7fcc8c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:33:22.449 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:22.442+0000 7fcc86ffd700 1 -- 172.21.15.81:0/3524706504 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc8c009320 con 0x7fcc980f51a0 2023-11-15T06:33:22.450 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:22.442+0000 7fcc9759e700 1 -- 172.21.15.81:0/3524706504 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc98060420 con 0x7fcc980f51a0 2023-11-15T06:33:22.450 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:22.442+0000 7fcc86ffd700 1 -- 172.21.15.81:0/3524706504 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcc8c0136c0 con 0x7fcc980f51a0 2023-11-15T06:33:22.450 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:22.442+0000 7fcc86ffd700 1 -- 172.21.15.81:0/3524706504 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc8c01ee90 con 0x7fcc980f51a0 2023-11-15T06:33:22.450 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:22.442+0000 7fcc9759e700 1 -- 172.21.15.81:0/3524706504 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc98109880 con 0x7fcc980f51a0 2023-11-15T06:33:22.450 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:22.443+0000 7fcc86ffd700 1 -- 172.21.15.81:0/3524706504 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcc8c01e420 con 0x7fcc980f51a0 2023-11-15T06:33:22.451 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:22.443+0000 7fcc9759e700 1 -- 172.21.15.81:0/3524706504 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc98059100 con 0x7fcc980f51a0 2023-11-15T06:33:22.451 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:22.443+0000 7fcc86ffd700 1 --2- 172.21.15.81:0/3524706504 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fcc8002ff10 0x7fcc800323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:33:22.451 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:22.443+0000 7fcc86ffd700 1 -- 172.21.15.81:0/3524706504 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcc8c040fa0 con 0x7fcc980f51a0 2023-11-15T06:33:22.451 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:22.444+0000 7fcc95d9b700 1 --2- 172.21.15.81:0/3524706504 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fcc8002ff10 0x7fcc800323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:33:22.451 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:22.444+0000 7fcc95d9b700 1 --2- 172.21.15.81:0/3524706504 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fcc8002ff10 0x7fcc800323d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fcc880096a0 tx=0x7fcc88006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:33:22.452 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:22.448+0000 7fcc86ffd700 1 -- 172.21.15.81:0/3524706504 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcc8c0095f0 con 0x7fcc980f51a0 2023-11-15T06:33:22.656 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:22.652+0000 7fcc9759e700 1 -- 172.21.15.81:0/3524706504 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcc980fabe0 con 0x7fcc980f51a0 2023-11-15T06:33:22.656 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:22.653+0000 7fcc86ffd700 1 -- 172.21.15.81:0/3524706504 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcc8c02c2d0 con 0x7fcc980f51a0 2023-11-15T06:33:22.657 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:33:22.657 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:33:22.658 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:22.654+0000 7fcc9759e700 1 -- 172.21.15.81:0/3524706504 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fcc8002ff10 msgr2=0x7fcc800323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:22.658 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:22.655+0000 7fcc9759e700 1 --2- 172.21.15.81:0/3524706504 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fcc8002ff10 0x7fcc800323d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fcc880096a0 tx=0x7fcc88006b40).stop 2023-11-15T06:33:22.658 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:22.655+0000 7fcc9759e700 1 -- 172.21.15.81:0/3524706504 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc980f51a0 msgr2=0x7fcc981091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:22.658 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:22.655+0000 7fcc9759e700 1 --2- 172.21.15.81:0/3524706504 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc980f51a0 0x7fcc981091d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fcc8c009830 tx=0x7fcc8c00b040).stop 2023-11-15T06:33:22.659 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:22.655+0000 7fcc9759e700 1 -- 172.21.15.81:0/3524706504 shutdown_connections 2023-11-15T06:33:22.659 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:22.655+0000 7fcc9759e700 1 --2- 172.21.15.81:0/3524706504 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fcc8002ff10 0x7fcc800323d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:22.659 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:22.655+0000 7fcc9759e700 1 --2- 172.21.15.81:0/3524706504 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc980f51a0 0x7fcc981091d0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:22.659 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:22.655+0000 7fcc9759e700 1 -- 172.21.15.81:0/3524706504 >> 172.21.15.81:0/3524706504 conn(0x7fcc980f09a0 msgr2=0x7fcc980f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:33:22.659 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:22.655+0000 7fcc9759e700 1 -- 172.21.15.81:0/3524706504 shutdown_connections 2023-11-15T06:33:22.660 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:22.655+0000 7fcc9759e700 1 -- 172.21.15.81:0/3524706504 wait complete. 2023-11-15T06:33:22.660 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:33:23.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:22 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:33:21.849495+0000 mgr.smithi062.rwurtk (mgr.14164) 232 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:33:23.169 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:22 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:33:22.655091+0000 mon.smithi062 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.81:0/3524706504' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:33:23.990 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:33:23.991 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:33:25.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:24 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:33:23.849735+0000 mgr.smithi062.rwurtk (mgr.14164) 233 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:33:25.187 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:25.183+0000 7f72a2f5a700 1 -- 172.21.15.81:0/2220115387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f729c0f51a0 msgr2=0x7f729c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:25.188 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:25.183+0000 7f72a2f5a700 1 --2- 172.21.15.81:0/2220115387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f729c0f51a0 0x7f729c0f55c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f7298009fb0 tx=0x7f7298009b30).stop 2023-11-15T06:33:25.188 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:25.183+0000 7f72a2f5a700 1 -- 172.21.15.81:0/2220115387 shutdown_connections 2023-11-15T06:33:25.188 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:25.183+0000 7f72a2f5a700 1 --2- 172.21.15.81:0/2220115387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f729c0f51a0 0x7f729c0f55c0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:25.188 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:25.183+0000 7f72a2f5a700 1 -- 172.21.15.81:0/2220115387 >> 172.21.15.81:0/2220115387 conn(0x7f729c0f09a0 msgr2=0x7f729c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:33:25.189 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:25.183+0000 7f72a2f5a700 1 -- 172.21.15.81:0/2220115387 shutdown_connections 2023-11-15T06:33:25.189 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:25.183+0000 7f72a2f5a700 1 -- 172.21.15.81:0/2220115387 wait complete. 2023-11-15T06:33:25.189 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:25.183+0000 7f72a2f5a700 1 Processor -- start 2023-11-15T06:33:25.189 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:25.183+0000 7f72a2f5a700 1 -- start start 2023-11-15T06:33:25.189 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:25.184+0000 7f72a2f5a700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f729c0f51a0 0x7f729c10b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:33:25.190 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:25.184+0000 7f72a2f5a700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f729c106fc0 con 0x7f729c0f51a0 2023-11-15T06:33:25.190 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:25.184+0000 7f72a1f58700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f729c0f51a0 0x7f729c10b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:33:25.190 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:25.184+0000 7f72a1f58700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f729c0f51a0 0x7f729c10b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:53792/0 (socket says 172.21.15.81:53792) 2023-11-15T06:33:25.190 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:25.184+0000 7f72a1f58700 1 -- 172.21.15.81:0/80466033 learned_addr learned my addr 172.21.15.81:0/80466033 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:33:25.190 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:25.184+0000 7f72a1f58700 1 -- 172.21.15.81:0/80466033 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7298005040 con 0x7f729c0f51a0 2023-11-15T06:33:25.191 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:25.184+0000 7f72a1f58700 1 --2- 172.21.15.81:0/80466033 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f729c0f51a0 0x7f729c10b5d0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f7298013510 tx=0x7f729800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:33:25.191 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:25.184+0000 7f72927fc700 1 -- 172.21.15.81:0/80466033 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7298009320 con 0x7f729c0f51a0 2023-11-15T06:33:25.191 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:25.184+0000 7f72927fc700 1 -- 172.21.15.81:0/80466033 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7298013f00 con 0x7f729c0f51a0 2023-11-15T06:33:25.192 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:25.184+0000 7f72927fc700 1 -- 172.21.15.81:0/80466033 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7298014dc0 con 0x7f729c0f51a0 2023-11-15T06:33:25.192 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:25.184+0000 7f72a2f5a700 1 -- 172.21.15.81:0/80466033 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f729c060220 con 0x7f729c0f51a0 2023-11-15T06:33:25.192 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:25.184+0000 7f72a2f5a700 1 -- 172.21.15.81:0/80466033 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f729c10bac0 con 0x7f729c0f51a0 2023-11-15T06:33:25.193 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:25.185+0000 7f72927fc700 1 -- 172.21.15.81:0/80466033 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f72980095f0 con 0x7f729c0f51a0 2023-11-15T06:33:25.193 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:25.185+0000 7f72a2f5a700 1 -- 172.21.15.81:0/80466033 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f729c045ad0 con 0x7f729c0f51a0 2023-11-15T06:33:25.194 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:25.185+0000 7f72927fc700 1 --2- 172.21.15.81:0/80466033 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f728802ff60 0x7f7288032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:33:25.194 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:25.185+0000 7f72927fc700 1 -- 172.21.15.81:0/80466033 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f72980410c0 con 0x7f729c0f51a0 2023-11-15T06:33:25.194 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:25.186+0000 7f72a1757700 1 --2- 172.21.15.81:0/80466033 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f728802ff60 0x7f7288032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:33:25.194 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:25.186+0000 7f72a1757700 1 --2- 172.21.15.81:0/80466033 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f728802ff60 0x7f7288032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f728c0096a0 tx=0x7f728c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:33:25.195 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:25.190+0000 7f72927fc700 1 -- 172.21.15.81:0/80466033 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7298014420 con 0x7f729c0f51a0 2023-11-15T06:33:25.399 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:25.395+0000 7f72a2f5a700 1 -- 172.21.15.81:0/80466033 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f729c10c280 con 0x7f729c0f51a0 2023-11-15T06:33:25.399 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:25.396+0000 7f72927fc700 1 -- 172.21.15.81:0/80466033 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7298010230 con 0x7f729c0f51a0 2023-11-15T06:33:25.400 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:33:25.400 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:33:25.402 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:25.397+0000 7f72a2f5a700 1 -- 172.21.15.81:0/80466033 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f728802ff60 msgr2=0x7f7288032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:25.403 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:25.397+0000 7f72a2f5a700 1 --2- 172.21.15.81:0/80466033 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f728802ff60 0x7f7288032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f728c0096a0 tx=0x7f728c006b40).stop 2023-11-15T06:33:25.403 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:25.397+0000 7f72a2f5a700 1 -- 172.21.15.81:0/80466033 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f729c0f51a0 msgr2=0x7f729c10b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:25.404 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:25.397+0000 7f72a2f5a700 1 --2- 172.21.15.81:0/80466033 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f729c0f51a0 0x7f729c10b5d0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f7298013510 tx=0x7f729800b040).stop 2023-11-15T06:33:25.404 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:25.397+0000 7f72a2f5a700 1 -- 172.21.15.81:0/80466033 shutdown_connections 2023-11-15T06:33:25.404 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:25.397+0000 7f72a2f5a700 1 --2- 172.21.15.81:0/80466033 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f728802ff60 0x7f7288032420 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:25.405 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:25.397+0000 7f72a2f5a700 1 --2- 172.21.15.81:0/80466033 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f729c0f51a0 0x7f729c10b5d0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:25.405 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:25.397+0000 7f72a2f5a700 1 -- 172.21.15.81:0/80466033 >> 172.21.15.81:0/80466033 conn(0x7f729c0f09a0 msgr2=0x7f729c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:33:25.405 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:25.397+0000 7f72a2f5a700 1 -- 172.21.15.81:0/80466033 shutdown_connections 2023-11-15T06:33:25.405 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:25.397+0000 7f72a2f5a700 1 -- 172.21.15.81:0/80466033 wait complete. 2023-11-15T06:33:25.406 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:33:26.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:25 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:33:25.397664+0000 mon.smithi062 (mon.0) 288 : audit [DBG] from='client.? 172.21.15.81:0/80466033' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:33:26.840 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:33:26.841 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:33:27.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:26 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:33:25.850089+0000 mgr.smithi062.rwurtk (mgr.14164) 234 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:33:28.152 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:28.149+0000 7fd01bd07700 1 -- 172.21.15.81:0/2565364427 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0140f6510 msgr2=0x7fd0140f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:28.153 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:28.149+0000 7fd01bd07700 1 --2- 172.21.15.81:0/2565364427 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0140f6510 0x7fd0140f6930 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7fd004009fb0 tx=0x7fd004009b30).stop 2023-11-15T06:33:28.153 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:28.149+0000 7fd01bd07700 1 -- 172.21.15.81:0/2565364427 shutdown_connections 2023-11-15T06:33:28.153 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:28.149+0000 7fd01bd07700 1 --2- 172.21.15.81:0/2565364427 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0140f6510 0x7fd0140f6930 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:28.153 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:28.149+0000 7fd01bd07700 1 -- 172.21.15.81:0/2565364427 >> 172.21.15.81:0/2565364427 conn(0x7fd0140f1c70 msgr2=0x7fd0140f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:33:28.153 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:28.149+0000 7fd01bd07700 1 -- 172.21.15.81:0/2565364427 shutdown_connections 2023-11-15T06:33:28.154 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:28.149+0000 7fd01bd07700 1 -- 172.21.15.81:0/2565364427 wait complete. 2023-11-15T06:33:28.154 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:28.150+0000 7fd01bd07700 1 Processor -- start 2023-11-15T06:33:28.154 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:28.150+0000 7fd01bd07700 1 -- start start 2023-11-15T06:33:28.154 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:28.150+0000 7fd01bd07700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0140f6510 0x7fd014106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:33:28.154 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:28.150+0000 7fd01bd07700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd014106c20 con 0x7fd0140f6510 2023-11-15T06:33:28.155 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:28.150+0000 7fd019aa3700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0140f6510 0x7fd014106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:33:28.155 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:28.150+0000 7fd019aa3700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0140f6510 0x7fd014106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:53808/0 (socket says 172.21.15.81:53808) 2023-11-15T06:33:28.155 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:28.150+0000 7fd019aa3700 1 -- 172.21.15.81:0/3513803838 learned_addr learned my addr 172.21.15.81:0/3513803838 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:33:28.155 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:28.150+0000 7fd019aa3700 1 -- 172.21.15.81:0/3513803838 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd004005040 con 0x7fd0140f6510 2023-11-15T06:33:28.155 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:28.150+0000 7fd019aa3700 1 --2- 172.21.15.81:0/3513803838 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0140f6510 0x7fd014106730 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7fd004014000 tx=0x7fd00400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:33:28.157 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:28.151+0000 7fd00a7fc700 1 -- 172.21.15.81:0/3513803838 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd0040095f0 con 0x7fd0140f6510 2023-11-15T06:33:28.158 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:28.151+0000 7fd01bd07700 1 -- 172.21.15.81:0/3513803838 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd014106e20 con 0x7fd0140f6510 2023-11-15T06:33:28.158 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:28.151+0000 7fd00a7fc700 1 -- 172.21.15.81:0/3513803838 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd004007e70 con 0x7fd0140f6510 2023-11-15T06:33:28.158 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:28.151+0000 7fd00a7fc700 1 -- 172.21.15.81:0/3513803838 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd004014ea0 con 0x7fd0140f6510 2023-11-15T06:33:28.159 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:28.151+0000 7fd01bd07700 1 -- 172.21.15.81:0/3513803838 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0141072b0 con 0x7fd0140f6510 2023-11-15T06:33:28.159 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:28.151+0000 7fd00a7fc700 1 -- 172.21.15.81:0/3513803838 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd004025070 con 0x7fd0140f6510 2023-11-15T06:33:28.159 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:28.151+0000 7fd01bd07700 1 -- 172.21.15.81:0/3513803838 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd014045ad0 con 0x7fd0140f6510 2023-11-15T06:33:28.159 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:28.152+0000 7fd00a7fc700 1 --2- 172.21.15.81:0/3513803838 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fd00002ff60 0x7fd000032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:33:28.159 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:28.152+0000 7fd00a7fc700 1 -- 172.21.15.81:0/3513803838 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd004040970 con 0x7fd0140f6510 2023-11-15T06:33:28.160 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:28.152+0000 7fd0192a2700 1 --2- 172.21.15.81:0/3513803838 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fd00002ff60 0x7fd000032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:33:28.160 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:28.152+0000 7fd0192a2700 1 --2- 172.21.15.81:0/3513803838 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fd00002ff60 0x7fd000032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fd0100096a0 tx=0x7fd010006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:33:28.160 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:28.156+0000 7fd00a7fc700 1 -- 172.21.15.81:0/3513803838 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd0040079c0 con 0x7fd0140f6510 2023-11-15T06:33:28.362 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:28.360+0000 7fd01bd07700 1 -- 172.21.15.81:0/3513803838 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd0140fbf60 con 0x7fd0140f6510 2023-11-15T06:33:28.363 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:28.361+0000 7fd00a7fc700 1 -- 172.21.15.81:0/3513803838 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd00400f300 con 0x7fd0140f6510 2023-11-15T06:33:28.363 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:33:28.364 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:33:28.365 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:28.362+0000 7fd01bd07700 1 -- 172.21.15.81:0/3513803838 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fd00002ff60 msgr2=0x7fd000032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:28.365 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:28.362+0000 7fd01bd07700 1 --2- 172.21.15.81:0/3513803838 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fd00002ff60 0x7fd000032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fd0100096a0 tx=0x7fd010006b40).stop 2023-11-15T06:33:28.365 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:28.362+0000 7fd01bd07700 1 -- 172.21.15.81:0/3513803838 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0140f6510 msgr2=0x7fd014106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:28.365 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:28.362+0000 7fd01bd07700 1 --2- 172.21.15.81:0/3513803838 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0140f6510 0x7fd014106730 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7fd004014000 tx=0x7fd00400b040).stop 2023-11-15T06:33:28.366 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:28.362+0000 7fd01bd07700 1 -- 172.21.15.81:0/3513803838 shutdown_connections 2023-11-15T06:33:28.366 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:28.362+0000 7fd01bd07700 1 --2- 172.21.15.81:0/3513803838 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fd00002ff60 0x7fd000032420 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:28.366 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:28.362+0000 7fd01bd07700 1 --2- 172.21.15.81:0/3513803838 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0140f6510 0x7fd014106730 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:28.366 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:28.362+0000 7fd01bd07700 1 -- 172.21.15.81:0/3513803838 >> 172.21.15.81:0/3513803838 conn(0x7fd0140f1c70 msgr2=0x7fd0140f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:33:28.366 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:28.362+0000 7fd01bd07700 1 -- 172.21.15.81:0/3513803838 shutdown_connections 2023-11-15T06:33:28.367 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:28.362+0000 7fd01bd07700 1 -- 172.21.15.81:0/3513803838 wait complete. 2023-11-15T06:33:28.367 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:33:28.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:28 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:33:27.850516+0000 mgr.smithi062.rwurtk (mgr.14164) 235 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:33:28.919 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:28 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:33:28.361869+0000 mon.smithi062 (mon.0) 289 : audit [DBG] from='client.? 172.21.15.81:0/3513803838' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:33:29.765 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:33:29.765 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:33:31.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:30 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:33:29.850878+0000 mgr.smithi062.rwurtk (mgr.14164) 236 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:33:31.172 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:31.167+0000 7f88a55da700 1 -- 172.21.15.81:0/1613728970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88a00f6510 msgr2=0x7f88a00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:31.172 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:31.167+0000 7f88a55da700 1 --2- 172.21.15.81:0/1613728970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88a00f6510 0x7f88a00f6930 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f8894009fb0 tx=0x7f8894009b30).stop 2023-11-15T06:33:31.172 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:31.167+0000 7f88a55da700 1 -- 172.21.15.81:0/1613728970 shutdown_connections 2023-11-15T06:33:31.172 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:31.167+0000 7f88a55da700 1 --2- 172.21.15.81:0/1613728970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88a00f6510 0x7f88a00f6930 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:31.173 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:31.167+0000 7f88a55da700 1 -- 172.21.15.81:0/1613728970 >> 172.21.15.81:0/1613728970 conn(0x7f88a00f1c70 msgr2=0x7f88a00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:33:31.173 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:31.167+0000 7f88a55da700 1 -- 172.21.15.81:0/1613728970 shutdown_connections 2023-11-15T06:33:31.173 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:31.167+0000 7f88a55da700 1 -- 172.21.15.81:0/1613728970 wait complete. 2023-11-15T06:33:31.173 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:31.168+0000 7f88a55da700 1 Processor -- start 2023-11-15T06:33:31.173 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:31.168+0000 7f88a55da700 1 -- start start 2023-11-15T06:33:31.174 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:31.168+0000 7f88a55da700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88a0106840 0x7f88a0106c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:33:31.174 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:31.168+0000 7f88a55da700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a0107150 con 0x7f88a0106840 2023-11-15T06:33:31.174 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:31.168+0000 7f889effd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88a0106840 0x7f88a0106c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:33:31.174 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:31.168+0000 7f889effd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88a0106840 0x7f88a0106c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:53814/0 (socket says 172.21.15.81:53814) 2023-11-15T06:33:31.174 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:31.168+0000 7f889effd700 1 -- 172.21.15.81:0/3771530812 learned_addr learned my addr 172.21.15.81:0/3771530812 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:33:31.175 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:31.168+0000 7f889effd700 1 -- 172.21.15.81:0/3771530812 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8894005040 con 0x7f88a0106840 2023-11-15T06:33:31.175 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:31.169+0000 7f889effd700 1 --2- 172.21.15.81:0/3771530812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88a0106840 0x7f88a0106c60 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f8894009f80 tx=0x7f8894009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:33:31.175 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:31.169+0000 7f88877fe700 1 -- 172.21.15.81:0/3771530812 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88940136c0 con 0x7f88a0106840 2023-11-15T06:33:31.175 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:31.169+0000 7f88877fe700 1 -- 172.21.15.81:0/3771530812 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8894013e70 con 0x7f88a0106840 2023-11-15T06:33:31.175 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:31.169+0000 7f88a55da700 1 -- 172.21.15.81:0/3771530812 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88a0107350 con 0x7f88a0106840 2023-11-15T06:33:31.176 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:31.169+0000 7f88877fe700 1 -- 172.21.15.81:0/3771530812 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8894014420 con 0x7f88a0106840 2023-11-15T06:33:31.176 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:31.169+0000 7f88a55da700 1 -- 172.21.15.81:0/3771530812 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88a0107ea0 con 0x7f88a0106840 2023-11-15T06:33:31.176 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:31.169+0000 7f88877fe700 1 -- 172.21.15.81:0/3771530812 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8894025060 con 0x7f88a0106840 2023-11-15T06:33:31.176 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:31.170+0000 7f88a55da700 1 -- 172.21.15.81:0/3771530812 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88a0059100 con 0x7f88a0106840 2023-11-15T06:33:31.176 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:31.170+0000 7f88877fe700 1 --2- 172.21.15.81:0/3771530812 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f888802ff60 0x7f8888032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:33:31.176 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:31.170+0000 7f88877fe700 1 -- 172.21.15.81:0/3771530812 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8894040ec0 con 0x7f88a0106840 2023-11-15T06:33:31.177 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:31.170+0000 7f889e7fc700 1 --2- 172.21.15.81:0/3771530812 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f888802ff60 0x7f8888032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:33:31.177 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:31.170+0000 7f889e7fc700 1 --2- 172.21.15.81:0/3771530812 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f888802ff60 0x7f8888032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f88900096a0 tx=0x7f8890006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:33:31.178 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:31.174+0000 7f88877fe700 1 -- 172.21.15.81:0/3771530812 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f889401cd10 con 0x7f88a0106840 2023-11-15T06:33:31.383 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:31.381+0000 7f88a55da700 1 -- 172.21.15.81:0/3771530812 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f88a0045ad0 con 0x7f88a0106840 2023-11-15T06:33:31.383 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:31.381+0000 7f88877fe700 1 -- 172.21.15.81:0/3771530812 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f889401cb30 con 0x7f88a0106840 2023-11-15T06:33:31.384 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:33:31.384 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:33:31.385 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:31.382+0000 7f88a55da700 1 -- 172.21.15.81:0/3771530812 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f888802ff60 msgr2=0x7f8888032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:31.385 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:31.382+0000 7f88a55da700 1 --2- 172.21.15.81:0/3771530812 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f888802ff60 0x7f8888032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f88900096a0 tx=0x7f8890006b40).stop 2023-11-15T06:33:31.386 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:31.383+0000 7f88a55da700 1 -- 172.21.15.81:0/3771530812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88a0106840 msgr2=0x7f88a0106c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:31.386 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:31.383+0000 7f88a55da700 1 --2- 172.21.15.81:0/3771530812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88a0106840 0x7f88a0106c60 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f8894009f80 tx=0x7f8894009b30).stop 2023-11-15T06:33:31.386 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:31.383+0000 7f88a55da700 1 -- 172.21.15.81:0/3771530812 shutdown_connections 2023-11-15T06:33:31.386 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:31.383+0000 7f88a55da700 1 --2- 172.21.15.81:0/3771530812 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f888802ff60 0x7f8888032420 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:31.387 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:31.383+0000 7f88a55da700 1 --2- 172.21.15.81:0/3771530812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88a0106840 0x7f88a0106c60 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:31.387 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:31.383+0000 7f88a55da700 1 -- 172.21.15.81:0/3771530812 >> 172.21.15.81:0/3771530812 conn(0x7f88a00f1c70 msgr2=0x7f88a00f27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:33:31.387 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:31.383+0000 7f88a55da700 1 -- 172.21.15.81:0/3771530812 shutdown_connections 2023-11-15T06:33:31.387 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:31.383+0000 7f88a55da700 1 -- 172.21.15.81:0/3771530812 wait complete. 2023-11-15T06:33:31.387 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:33:32.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:31 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:33:31.382079+0000 mon.smithi062 (mon.0) 290 : audit [DBG] from='client.? 172.21.15.81:0/3771530812' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:33:32.779 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:33:32.780 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:33:33.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:32 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:33:31.851179+0000 mgr.smithi062.rwurtk (mgr.14164) 237 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:33:34.020 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:34.016+0000 7fdbbeebe700 1 -- 172.21.15.81:0/1604686087 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbb8060200 msgr2=0x7fdbb80f9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:34.020 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:34.016+0000 7fdbbeebe700 1 --2- 172.21.15.81:0/1604686087 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbb8060200 0x7fdbb80f9170 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7fdbac009fb0 tx=0x7fdbac009b30).stop 2023-11-15T06:33:34.021 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:34.016+0000 7fdbbeebe700 1 -- 172.21.15.81:0/1604686087 shutdown_connections 2023-11-15T06:33:34.021 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:34.016+0000 7fdbbeebe700 1 --2- 172.21.15.81:0/1604686087 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbb8060200 0x7fdbb80f9170 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:34.021 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:34.016+0000 7fdbbeebe700 1 -- 172.21.15.81:0/1604686087 >> 172.21.15.81:0/1604686087 conn(0x7fdbb80f0b70 msgr2=0x7fdbb80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:33:34.021 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:34.016+0000 7fdbbeebe700 1 -- 172.21.15.81:0/1604686087 shutdown_connections 2023-11-15T06:33:34.021 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:34.016+0000 7fdbbeebe700 1 -- 172.21.15.81:0/1604686087 wait complete. 2023-11-15T06:33:34.022 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:34.017+0000 7fdbbeebe700 1 Processor -- start 2023-11-15T06:33:34.022 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:34.017+0000 7fdbbeebe700 1 -- start start 2023-11-15T06:33:34.022 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:34.017+0000 7fdbbeebe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbb8060200 0x7fdbb80f6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:33:34.022 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:34.017+0000 7fdbbeebe700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb80f5070 con 0x7fdbb8060200 2023-11-15T06:33:34.023 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:34.017+0000 7fdbbcc5a700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbb8060200 0x7fdbb80f6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:33:34.023 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:34.017+0000 7fdbbcc5a700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbb8060200 0x7fdbb80f6bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:39722/0 (socket says 172.21.15.81:39722) 2023-11-15T06:33:34.023 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:34.017+0000 7fdbbcc5a700 1 -- 172.21.15.81:0/453221303 learned_addr learned my addr 172.21.15.81:0/453221303 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:33:34.023 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:34.018+0000 7fdbbcc5a700 1 -- 172.21.15.81:0/453221303 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbac005040 con 0x7fdbb8060200 2023-11-15T06:33:34.023 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:34.018+0000 7fdbbcc5a700 1 --2- 172.21.15.81:0/453221303 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbb8060200 0x7fdbb80f6bf0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7fdbac013510 tx=0x7fdbac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:33:34.024 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:34.018+0000 7fdbb57fa700 1 -- 172.21.15.81:0/453221303 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbac009320 con 0x7fdbb8060200 2023-11-15T06:33:34.024 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:34.018+0000 7fdbbeebe700 1 -- 172.21.15.81:0/453221303 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbb80f5210 con 0x7fdbb8060200 2023-11-15T06:33:34.024 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:34.018+0000 7fdbb57fa700 1 -- 172.21.15.81:0/453221303 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdbac013f00 con 0x7fdbb8060200 2023-11-15T06:33:34.024 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:34.018+0000 7fdbb57fa700 1 -- 172.21.15.81:0/453221303 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbac014dc0 con 0x7fdbb8060200 2023-11-15T06:33:34.024 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:34.018+0000 7fdbbeebe700 1 -- 172.21.15.81:0/453221303 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbb80f56a0 con 0x7fdbb8060200 2023-11-15T06:33:34.025 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:34.019+0000 7fdbb57fa700 1 -- 172.21.15.81:0/453221303 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdbac01d3d0 con 0x7fdbb8060200 2023-11-15T06:33:34.025 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:34.019+0000 7fdbb57fa700 1 --2- 172.21.15.81:0/453221303 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fdba802ff60 0x7fdba8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:33:34.026 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:34.019+0000 7fdbb57fa700 1 -- 172.21.15.81:0/453221303 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdbac041030 con 0x7fdbb8060200 2023-11-15T06:33:34.026 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:34.019+0000 7fdbbeebe700 1 -- 172.21.15.81:0/453221303 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbb8059100 con 0x7fdbb8060200 2023-11-15T06:33:34.026 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:34.019+0000 7fdbb7fff700 1 --2- 172.21.15.81:0/453221303 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fdba802ff60 0x7fdba8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:33:34.026 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:34.020+0000 7fdbb7fff700 1 --2- 172.21.15.81:0/453221303 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fdba802ff60 0x7fdba8032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fdba40096a0 tx=0x7fdba4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:33:34.027 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:34.024+0000 7fdbb57fa700 1 -- 172.21.15.81:0/453221303 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdbac012920 con 0x7fdbb8060200 2023-11-15T06:33:34.232 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:34.230+0000 7fdbbeebe700 1 -- 172.21.15.81:0/453221303 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdbb80f5da0 con 0x7fdbb8060200 2023-11-15T06:33:34.232 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:34.230+0000 7fdbb57fa700 1 -- 172.21.15.81:0/453221303 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdbac02b020 con 0x7fdbb8060200 2023-11-15T06:33:34.233 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:33:34.233 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:33:34.239 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:34.232+0000 7fdbbeebe700 1 -- 172.21.15.81:0/453221303 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fdba802ff60 msgr2=0x7fdba8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:34.239 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:34.232+0000 7fdbbeebe700 1 --2- 172.21.15.81:0/453221303 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fdba802ff60 0x7fdba8032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fdba40096a0 tx=0x7fdba4006b40).stop 2023-11-15T06:33:34.239 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:34.232+0000 7fdbbeebe700 1 -- 172.21.15.81:0/453221303 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbb8060200 msgr2=0x7fdbb80f6bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:34.240 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:34.232+0000 7fdbbeebe700 1 --2- 172.21.15.81:0/453221303 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbb8060200 0x7fdbb80f6bf0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7fdbac013510 tx=0x7fdbac00b040).stop 2023-11-15T06:33:34.240 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:34.232+0000 7fdbbeebe700 1 -- 172.21.15.81:0/453221303 shutdown_connections 2023-11-15T06:33:34.240 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:34.232+0000 7fdbbeebe700 1 --2- 172.21.15.81:0/453221303 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fdba802ff60 0x7fdba8032420 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:34.240 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:34.232+0000 7fdbbeebe700 1 --2- 172.21.15.81:0/453221303 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbb8060200 0x7fdbb80f6bf0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:34.240 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:34.232+0000 7fdbbeebe700 1 -- 172.21.15.81:0/453221303 >> 172.21.15.81:0/453221303 conn(0x7fdbb80f0b70 msgr2=0x7fdbb80f16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:33:34.241 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:34.232+0000 7fdbbeebe700 1 -- 172.21.15.81:0/453221303 shutdown_connections 2023-11-15T06:33:34.241 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:34.232+0000 7fdbbeebe700 1 -- 172.21.15.81:0/453221303 wait complete. 2023-11-15T06:33:34.241 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:33:35.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:34 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:33:33.851571+0000 mgr.smithi062.rwurtk (mgr.14164) 238 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:33:35.169 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:34 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:33:34.231113+0000 mon.smithi062 (mon.0) 291 : audit [DBG] from='client.? 172.21.15.81:0/453221303' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:33:35.653 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:33:35.653 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:33:36.979 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:36.976+0000 7f55b68cd700 1 -- 172.21.15.81:0/1771556781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55b00f5410 msgr2=0x7f55b00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:36.979 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:36.976+0000 7f55b68cd700 1 --2- 172.21.15.81:0/1771556781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55b00f5410 0x7f55b00f5830 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f55a4009fb0 tx=0x7f55a4009b30).stop 2023-11-15T06:33:36.980 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:36.976+0000 7f55b68cd700 1 -- 172.21.15.81:0/1771556781 shutdown_connections 2023-11-15T06:33:36.980 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:36.976+0000 7f55b68cd700 1 --2- 172.21.15.81:0/1771556781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55b00f5410 0x7f55b00f5830 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:36.980 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:36.976+0000 7f55b68cd700 1 -- 172.21.15.81:0/1771556781 >> 172.21.15.81:0/1771556781 conn(0x7f55b00f0b70 msgr2=0x7f55b00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:33:36.980 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:36.976+0000 7f55b68cd700 1 -- 172.21.15.81:0/1771556781 shutdown_connections 2023-11-15T06:33:36.981 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:36.976+0000 7f55b68cd700 1 -- 172.21.15.81:0/1771556781 wait complete. 2023-11-15T06:33:36.981 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:36.976+0000 7f55b68cd700 1 Processor -- start 2023-11-15T06:33:36.981 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:36.976+0000 7f55b68cd700 1 -- start start 2023-11-15T06:33:36.981 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:36.977+0000 7f55b68cd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55b00f5410 0x7f55b00fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:33:36.981 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:36.977+0000 7f55b68cd700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b00fb820 con 0x7f55b00f5410 2023-11-15T06:33:36.982 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:36.977+0000 7f55affff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55b00f5410 0x7f55b00fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:33:36.982 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:36.977+0000 7f55affff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55b00f5410 0x7f55b00fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:39738/0 (socket says 172.21.15.81:39738) 2023-11-15T06:33:36.983 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:36.977+0000 7f55affff700 1 -- 172.21.15.81:0/4257692092 learned_addr learned my addr 172.21.15.81:0/4257692092 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:33:36.983 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:36.977+0000 7f55affff700 1 -- 172.21.15.81:0/4257692092 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55a4005040 con 0x7f55b00f5410 2023-11-15T06:33:36.983 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:36.977+0000 7f55affff700 1 --2- 172.21.15.81:0/4257692092 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55b00f5410 0x7f55b00fd3a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f55a4013510 tx=0x7f55a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:33:36.985 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:36.977+0000 7f55acff9700 1 -- 172.21.15.81:0/4257692092 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55a4009320 con 0x7f55b00f5410 2023-11-15T06:33:36.985 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:36.977+0000 7f55b68cd700 1 -- 172.21.15.81:0/4257692092 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55b00fb9c0 con 0x7f55b00f5410 2023-11-15T06:33:36.985 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:36.977+0000 7f55acff9700 1 -- 172.21.15.81:0/4257692092 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f55a4013f00 con 0x7f55b00f5410 2023-11-15T06:33:36.986 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:36.977+0000 7f55acff9700 1 -- 172.21.15.81:0/4257692092 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55a4014d90 con 0x7f55b00f5410 2023-11-15T06:33:36.986 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:36.978+0000 7f55b68cd700 1 -- 172.21.15.81:0/4257692092 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55b00fbe50 con 0x7f55b00f5410 2023-11-15T06:33:36.986 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:36.978+0000 7f55acff9700 1 -- 172.21.15.81:0/4257692092 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f55a4014420 con 0x7f55b00f5410 2023-11-15T06:33:36.986 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:36.978+0000 7f55b68cd700 1 -- 172.21.15.81:0/4257692092 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55b0059100 con 0x7f55b00f5410 2023-11-15T06:33:36.987 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:36.979+0000 7f55acff9700 1 --2- 172.21.15.81:0/4257692092 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f559802ff60 0x7f5598032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:33:36.987 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:36.979+0000 7f55acff9700 1 -- 172.21.15.81:0/4257692092 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f55a4041000 con 0x7f55b00f5410 2023-11-15T06:33:36.987 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:36.979+0000 7f55af7fe700 1 --2- 172.21.15.81:0/4257692092 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f559802ff60 0x7f5598032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:33:36.987 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:36.979+0000 7f55af7fe700 1 --2- 172.21.15.81:0/4257692092 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f559802ff60 0x7f5598032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f55a00096a0 tx=0x7f55a0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:33:36.988 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:36.984+0000 7f55acff9700 1 -- 172.21.15.81:0/4257692092 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f55a40095f0 con 0x7f55b00f5410 2023-11-15T06:33:37.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:36 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:33:35.851926+0000 mgr.smithi062.rwurtk (mgr.14164) 239 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:33:37.189 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:37.187+0000 7f55b68cd700 1 -- 172.21.15.81:0/4257692092 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f55b00fae60 con 0x7f55b00f5410 2023-11-15T06:33:37.190 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:37.187+0000 7f55acff9700 1 -- 172.21.15.81:0/4257692092 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f55a401db70 con 0x7f55b00f5410 2023-11-15T06:33:37.190 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:33:37.190 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:33:37.191 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:37.189+0000 7f55b68cd700 1 -- 172.21.15.81:0/4257692092 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f559802ff60 msgr2=0x7f5598032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:37.192 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:37.189+0000 7f55b68cd700 1 --2- 172.21.15.81:0/4257692092 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f559802ff60 0x7f5598032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f55a00096a0 tx=0x7f55a0006b40).stop 2023-11-15T06:33:37.192 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:37.189+0000 7f55b68cd700 1 -- 172.21.15.81:0/4257692092 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55b00f5410 msgr2=0x7f55b00fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:37.192 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:37.189+0000 7f55b68cd700 1 --2- 172.21.15.81:0/4257692092 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55b00f5410 0x7f55b00fd3a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f55a4013510 tx=0x7f55a400b040).stop 2023-11-15T06:33:37.192 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:37.189+0000 7f55b68cd700 1 -- 172.21.15.81:0/4257692092 shutdown_connections 2023-11-15T06:33:37.192 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:37.189+0000 7f55b68cd700 1 --2- 172.21.15.81:0/4257692092 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f559802ff60 0x7f5598032420 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:37.193 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:37.189+0000 7f55b68cd700 1 --2- 172.21.15.81:0/4257692092 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55b00f5410 0x7f55b00fd3a0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:37.193 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:37.189+0000 7f55b68cd700 1 -- 172.21.15.81:0/4257692092 >> 172.21.15.81:0/4257692092 conn(0x7f55b00f0b70 msgr2=0x7f55b005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:33:37.193 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:37.189+0000 7f55b68cd700 1 -- 172.21.15.81:0/4257692092 shutdown_connections 2023-11-15T06:33:37.193 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:37.189+0000 7f55b68cd700 1 -- 172.21.15.81:0/4257692092 wait complete. 2023-11-15T06:33:37.194 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:33:38.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:37 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:33:37.188315+0000 mon.smithi062 (mon.0) 292 : audit [DBG] from='client.? 172.21.15.81:0/4257692092' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:33:38.595 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:33:38.596 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:33:39.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:38 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:33:37.852285+0000 mgr.smithi062.rwurtk (mgr.14164) 240 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:33:39.808 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:39.806+0000 7ff971a3b700 1 -- 172.21.15.81:0/2906604172 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff96c0f51a0 msgr2=0x7ff96c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:39.809 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:39.806+0000 7ff971a3b700 1 --2- 172.21.15.81:0/2906604172 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff96c0f51a0 0x7ff96c0f55c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7ff960009fb0 tx=0x7ff960009b30).stop 2023-11-15T06:33:39.809 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:39.806+0000 7ff971a3b700 1 -- 172.21.15.81:0/2906604172 shutdown_connections 2023-11-15T06:33:39.809 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:39.806+0000 7ff971a3b700 1 --2- 172.21.15.81:0/2906604172 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff96c0f51a0 0x7ff96c0f55c0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:39.809 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:39.806+0000 7ff971a3b700 1 -- 172.21.15.81:0/2906604172 >> 172.21.15.81:0/2906604172 conn(0x7ff96c0f09a0 msgr2=0x7ff96c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:33:39.810 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:39.806+0000 7ff971a3b700 1 -- 172.21.15.81:0/2906604172 shutdown_connections 2023-11-15T06:33:39.810 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:39.806+0000 7ff971a3b700 1 -- 172.21.15.81:0/2906604172 wait complete. 2023-11-15T06:33:39.810 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:39.806+0000 7ff971a3b700 1 Processor -- start 2023-11-15T06:33:39.810 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:39.806+0000 7ff971a3b700 1 -- start start 2023-11-15T06:33:39.810 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:39.806+0000 7ff971a3b700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff96c0f51a0 0x7ff96c1091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:33:39.811 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:39.807+0000 7ff971a3b700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff96c060220 con 0x7ff96c0f51a0 2023-11-15T06:33:39.812 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:39.807+0000 7ff970a39700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff96c0f51a0 0x7ff96c1091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:33:39.813 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:39.807+0000 7ff970a39700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff96c0f51a0 0x7ff96c1091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:39746/0 (socket says 172.21.15.81:39746) 2023-11-15T06:33:39.813 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:39.807+0000 7ff970a39700 1 -- 172.21.15.81:0/3458891379 learned_addr learned my addr 172.21.15.81:0/3458891379 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:33:39.813 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:39.807+0000 7ff970a39700 1 -- 172.21.15.81:0/3458891379 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff960005040 con 0x7ff96c0f51a0 2023-11-15T06:33:39.813 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:39.808+0000 7ff970a39700 1 --2- 172.21.15.81:0/3458891379 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff96c0f51a0 0x7ff96c1091d0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7ff960013510 tx=0x7ff96000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:33:39.813 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:39.808+0000 7ff9697fa700 1 -- 172.21.15.81:0/3458891379 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff960009320 con 0x7ff96c0f51a0 2023-11-15T06:33:39.814 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:39.808+0000 7ff9697fa700 1 -- 172.21.15.81:0/3458891379 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff960013f00 con 0x7ff96c0f51a0 2023-11-15T06:33:39.814 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:39.808+0000 7ff971a3b700 1 -- 172.21.15.81:0/3458891379 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff96c060420 con 0x7ff96c0f51a0 2023-11-15T06:33:39.815 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:39.808+0000 7ff9697fa700 1 -- 172.21.15.81:0/3458891379 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff960014dc0 con 0x7ff96c0f51a0 2023-11-15T06:33:39.815 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:39.808+0000 7ff971a3b700 1 -- 172.21.15.81:0/3458891379 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff96c109880 con 0x7ff96c0f51a0 2023-11-15T06:33:39.815 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:39.809+0000 7ff9697fa700 1 -- 172.21.15.81:0/3458891379 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff9600095f0 con 0x7ff96c0f51a0 2023-11-15T06:33:39.815 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:39.809+0000 7ff971a3b700 1 -- 172.21.15.81:0/3458891379 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff96c059100 con 0x7ff96c0f51a0 2023-11-15T06:33:39.815 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:39.809+0000 7ff9697fa700 1 --2- 172.21.15.81:0/3458891379 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff95c02ff60 0x7ff95c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:33:39.816 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:39.809+0000 7ff9697fa700 1 -- 172.21.15.81:0/3458891379 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff960041000 con 0x7ff96c0f51a0 2023-11-15T06:33:39.816 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:39.809+0000 7ff96bfff700 1 --2- 172.21.15.81:0/3458891379 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff95c02ff60 0x7ff95c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:33:39.816 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:39.810+0000 7ff96bfff700 1 --2- 172.21.15.81:0/3458891379 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff95c02ff60 0x7ff95c032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7ff9580096a0 tx=0x7ff958006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:33:39.816 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:39.814+0000 7ff9697fa700 1 -- 172.21.15.81:0/3458891379 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff96001c790 con 0x7ff96c0f51a0 2023-11-15T06:33:40.024 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:40.022+0000 7ff971a3b700 1 -- 172.21.15.81:0/3458891379 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff96c045ad0 con 0x7ff96c0f51a0 2023-11-15T06:33:40.025 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:40.022+0000 7ff9697fa700 1 -- 172.21.15.81:0/3458891379 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff960010230 con 0x7ff96c0f51a0 2023-11-15T06:33:40.025 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:33:40.025 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:33:40.026 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:40.024+0000 7ff971a3b700 1 -- 172.21.15.81:0/3458891379 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff95c02ff60 msgr2=0x7ff95c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:40.027 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:40.024+0000 7ff971a3b700 1 --2- 172.21.15.81:0/3458891379 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff95c02ff60 0x7ff95c032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7ff9580096a0 tx=0x7ff958006b40).stop 2023-11-15T06:33:40.027 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:40.024+0000 7ff971a3b700 1 -- 172.21.15.81:0/3458891379 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff96c0f51a0 msgr2=0x7ff96c1091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:40.027 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:40.024+0000 7ff971a3b700 1 --2- 172.21.15.81:0/3458891379 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff96c0f51a0 0x7ff96c1091d0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7ff960013510 tx=0x7ff96000b040).stop 2023-11-15T06:33:40.027 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:40.024+0000 7ff971a3b700 1 -- 172.21.15.81:0/3458891379 shutdown_connections 2023-11-15T06:33:40.027 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:40.024+0000 7ff971a3b700 1 --2- 172.21.15.81:0/3458891379 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff95c02ff60 0x7ff95c032420 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:40.028 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:40.024+0000 7ff971a3b700 1 --2- 172.21.15.81:0/3458891379 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff96c0f51a0 0x7ff96c1091d0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:40.028 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:40.024+0000 7ff971a3b700 1 -- 172.21.15.81:0/3458891379 >> 172.21.15.81:0/3458891379 conn(0x7ff96c0f09a0 msgr2=0x7ff96c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:33:40.028 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:40.024+0000 7ff971a3b700 1 -- 172.21.15.81:0/3458891379 shutdown_connections 2023-11-15T06:33:40.028 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:40.024+0000 7ff971a3b700 1 -- 172.21.15.81:0/3458891379 wait complete. 2023-11-15T06:33:40.029 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:33:41.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:40 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:33:39.852600+0000 mgr.smithi062.rwurtk (mgr.14164) 241 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:33:41.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:40 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:33:40.023406+0000 mon.smithi062 (mon.0) 293 : audit [DBG] from='client.? 172.21.15.81:0/3458891379' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:33:41.475 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:33:41.476 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:33:42.717 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:42.714+0000 7f4e7153d700 1 -- 172.21.15.81:0/2158997188 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e6c0f6510 msgr2=0x7f4e6c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:42.717 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:42.714+0000 7f4e7153d700 1 --2- 172.21.15.81:0/2158997188 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e6c0f6510 0x7f4e6c0f6930 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f4e60009fb0 tx=0x7f4e60009b30).stop 2023-11-15T06:33:42.718 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:42.714+0000 7f4e7153d700 1 -- 172.21.15.81:0/2158997188 shutdown_connections 2023-11-15T06:33:42.718 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:42.714+0000 7f4e7153d700 1 --2- 172.21.15.81:0/2158997188 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e6c0f6510 0x7f4e6c0f6930 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:42.718 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:42.714+0000 7f4e7153d700 1 -- 172.21.15.81:0/2158997188 >> 172.21.15.81:0/2158997188 conn(0x7f4e6c0f1c70 msgr2=0x7f4e6c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:33:42.718 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:42.714+0000 7f4e7153d700 1 -- 172.21.15.81:0/2158997188 shutdown_connections 2023-11-15T06:33:42.718 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:42.714+0000 7f4e7153d700 1 -- 172.21.15.81:0/2158997188 wait complete. 2023-11-15T06:33:42.719 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:42.715+0000 7f4e7153d700 1 Processor -- start 2023-11-15T06:33:42.719 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:42.715+0000 7f4e7153d700 1 -- start start 2023-11-15T06:33:42.719 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:42.715+0000 7f4e7153d700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e6c0f6510 0x7f4e6c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:33:42.719 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:42.715+0000 7f4e7153d700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6c069e20 con 0x7f4e6c0f6510 2023-11-15T06:33:42.719 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:42.715+0000 7f4e6affd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e6c0f6510 0x7f4e6c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:33:42.719 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:42.715+0000 7f4e6affd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e6c0f6510 0x7f4e6c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:44012/0 (socket says 172.21.15.81:44012) 2023-11-15T06:33:42.720 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:42.715+0000 7f4e6affd700 1 -- 172.21.15.81:0/1122770801 learned_addr learned my addr 172.21.15.81:0/1122770801 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:33:42.721 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:42.715+0000 7f4e6affd700 1 -- 172.21.15.81:0/1122770801 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e60005040 con 0x7f4e6c0f6510 2023-11-15T06:33:42.722 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:42.715+0000 7f4e6affd700 1 --2- 172.21.15.81:0/1122770801 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e6c0f6510 0x7f4e6c06b9a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f4e60013510 tx=0x7f4e6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:33:42.722 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:42.715+0000 7f4e537fe700 1 -- 172.21.15.81:0/1122770801 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e60009320 con 0x7f4e6c0f6510 2023-11-15T06:33:42.722 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:42.716+0000 7f4e7153d700 1 -- 172.21.15.81:0/1122770801 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e6c069fc0 con 0x7f4e6c0f6510 2023-11-15T06:33:42.723 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:42.716+0000 7f4e537fe700 1 -- 172.21.15.81:0/1122770801 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4e60013f00 con 0x7f4e6c0f6510 2023-11-15T06:33:42.723 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:42.716+0000 7f4e537fe700 1 -- 172.21.15.81:0/1122770801 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e60014d90 con 0x7f4e6c0f6510 2023-11-15T06:33:42.724 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:42.716+0000 7f4e7153d700 1 -- 172.21.15.81:0/1122770801 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e6c06a450 con 0x7f4e6c0f6510 2023-11-15T06:33:42.724 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:42.716+0000 7f4e537fe700 1 -- 172.21.15.81:0/1122770801 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4e600263a0 con 0x7f4e6c0f6510 2023-11-15T06:33:42.724 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:42.716+0000 7f4e7153d700 1 -- 172.21.15.81:0/1122770801 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e6c059100 con 0x7f4e6c0f6510 2023-11-15T06:33:42.724 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:42.716+0000 7f4e537fe700 1 --2- 172.21.15.81:0/1122770801 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f4e5402ff60 0x7f4e54032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:33:42.724 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:42.717+0000 7f4e537fe700 1 -- 172.21.15.81:0/1122770801 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4e600413e0 con 0x7f4e6c0f6510 2023-11-15T06:33:42.725 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:42.717+0000 7f4e6a7fc700 1 --2- 172.21.15.81:0/1122770801 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f4e5402ff60 0x7f4e54032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:33:42.725 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:42.717+0000 7f4e6a7fc700 1 --2- 172.21.15.81:0/1122770801 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f4e5402ff60 0x7f4e54032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f4e5c0096a0 tx=0x7f4e5c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:33:42.725 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:42.721+0000 7f4e537fe700 1 -- 172.21.15.81:0/1122770801 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4e600126f0 con 0x7f4e6c0f6510 2023-11-15T06:33:42.926 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:42.924+0000 7f4e7153d700 1 -- 172.21.15.81:0/1122770801 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4e6c0fbf60 con 0x7f4e6c0f6510 2023-11-15T06:33:42.927 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:42.925+0000 7f4e537fe700 1 -- 172.21.15.81:0/1122770801 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4e60012330 con 0x7f4e6c0f6510 2023-11-15T06:33:42.927 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:33:42.928 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:33:42.929 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:42.926+0000 7f4e7153d700 1 -- 172.21.15.81:0/1122770801 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f4e5402ff60 msgr2=0x7f4e54032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:42.929 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:42.926+0000 7f4e7153d700 1 --2- 172.21.15.81:0/1122770801 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f4e5402ff60 0x7f4e54032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f4e5c0096a0 tx=0x7f4e5c006b40).stop 2023-11-15T06:33:42.929 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:42.926+0000 7f4e7153d700 1 -- 172.21.15.81:0/1122770801 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e6c0f6510 msgr2=0x7f4e6c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:42.929 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:42.926+0000 7f4e7153d700 1 --2- 172.21.15.81:0/1122770801 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e6c0f6510 0x7f4e6c06b9a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f4e60013510 tx=0x7f4e6000b040).stop 2023-11-15T06:33:42.929 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:42.926+0000 7f4e7153d700 1 -- 172.21.15.81:0/1122770801 shutdown_connections 2023-11-15T06:33:42.930 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:42.926+0000 7f4e7153d700 1 --2- 172.21.15.81:0/1122770801 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f4e5402ff60 0x7f4e54032420 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:42.930 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:42.926+0000 7f4e7153d700 1 --2- 172.21.15.81:0/1122770801 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e6c0f6510 0x7f4e6c06b9a0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:42.930 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:42.926+0000 7f4e7153d700 1 -- 172.21.15.81:0/1122770801 >> 172.21.15.81:0/1122770801 conn(0x7f4e6c0f1c70 msgr2=0x7f4e6c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:33:42.930 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:42.926+0000 7f4e7153d700 1 -- 172.21.15.81:0/1122770801 shutdown_connections 2023-11-15T06:33:42.931 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:42.926+0000 7f4e7153d700 1 -- 172.21.15.81:0/1122770801 wait complete. 2023-11-15T06:33:42.931 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:33:43.170 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:42 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:33:41.852952+0000 mgr.smithi062.rwurtk (mgr.14164) 242 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:33:44.331 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:33:44.332 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:33:44.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:43 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:33:42.925818+0000 mon.smithi062 (mon.0) 294 : audit [DBG] from='client.? 172.21.15.81:0/1122770801' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:33:45.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:44 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:33:43.853268+0000 mgr.smithi062.rwurtk (mgr.14164) 243 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:33:45.545 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:45.541+0000 7f10746ae700 1 -- 172.21.15.81:0/2743647897 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f106c060280 msgr2=0x7f106c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:45.546 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:45.542+0000 7f10746ae700 1 --2- 172.21.15.81:0/2743647897 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f106c060280 0x7f106c0f90f0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f105c009fb0 tx=0x7f105c009b30).stop 2023-11-15T06:33:45.546 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:45.542+0000 7f10746ae700 1 -- 172.21.15.81:0/2743647897 shutdown_connections 2023-11-15T06:33:45.546 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:45.542+0000 7f10746ae700 1 --2- 172.21.15.81:0/2743647897 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f106c060280 0x7f106c0f90f0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:45.546 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:45.542+0000 7f10746ae700 1 -- 172.21.15.81:0/2743647897 >> 172.21.15.81:0/2743647897 conn(0x7f106c0f0b70 msgr2=0x7f106c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:33:45.546 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:45.542+0000 7f10746ae700 1 -- 172.21.15.81:0/2743647897 shutdown_connections 2023-11-15T06:33:45.547 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:45.542+0000 7f10746ae700 1 -- 172.21.15.81:0/2743647897 wait complete. 2023-11-15T06:33:45.547 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:45.542+0000 7f10746ae700 1 Processor -- start 2023-11-15T06:33:45.547 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:45.542+0000 7f10746ae700 1 -- start start 2023-11-15T06:33:45.547 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:45.542+0000 7f10746ae700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f106c060280 0x7f106c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:33:45.547 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:45.542+0000 7f10746ae700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106c106bd0 con 0x7f106c060280 2023-11-15T06:33:45.548 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:45.542+0000 7f107244a700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f106c060280 0x7f106c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:33:45.548 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:45.543+0000 7f107244a700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f106c060280 0x7f106c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:44038/0 (socket says 172.21.15.81:44038) 2023-11-15T06:33:45.548 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:45.543+0000 7f107244a700 1 -- 172.21.15.81:0/264550567 learned_addr learned my addr 172.21.15.81:0/264550567 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:33:45.548 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:45.543+0000 7f107244a700 1 -- 172.21.15.81:0/264550567 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f105c005040 con 0x7f106c060280 2023-11-15T06:33:45.548 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:45.543+0000 7f107244a700 1 --2- 172.21.15.81:0/264550567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f106c060280 0x7f106c1066e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f105c013510 tx=0x7f105c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:33:45.549 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:45.543+0000 7f1062ffd700 1 -- 172.21.15.81:0/264550567 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f105c009320 con 0x7f106c060280 2023-11-15T06:33:45.549 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:45.543+0000 7f1062ffd700 1 -- 172.21.15.81:0/264550567 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f105c013f00 con 0x7f106c060280 2023-11-15T06:33:45.549 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:45.543+0000 7f10746ae700 1 -- 172.21.15.81:0/264550567 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f106c106dd0 con 0x7f106c060280 2023-11-15T06:33:45.549 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:45.543+0000 7f1062ffd700 1 -- 172.21.15.81:0/264550567 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f105c014dc0 con 0x7f106c060280 2023-11-15T06:33:45.549 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:45.543+0000 7f10746ae700 1 -- 172.21.15.81:0/264550567 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f106c107260 con 0x7f106c060280 2023-11-15T06:33:45.550 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:45.544+0000 7f1062ffd700 1 -- 172.21.15.81:0/264550567 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f105c0095f0 con 0x7f106c060280 2023-11-15T06:33:45.551 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:45.544+0000 7f10746ae700 1 -- 172.21.15.81:0/264550567 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f106c045ad0 con 0x7f106c060280 2023-11-15T06:33:45.551 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:45.544+0000 7f1062ffd700 1 --2- 172.21.15.81:0/264550567 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f105802ff60 0x7f1058032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:33:45.551 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:45.544+0000 7f1062ffd700 1 -- 172.21.15.81:0/264550567 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f105c0410c0 con 0x7f106c060280 2023-11-15T06:33:45.551 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:45.544+0000 7f1071c49700 1 --2- 172.21.15.81:0/264550567 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f105802ff60 0x7f1058032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:33:45.551 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:45.545+0000 7f1071c49700 1 --2- 172.21.15.81:0/264550567 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f105802ff60 0x7f1058032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f10680096a0 tx=0x7f1068006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:33:45.552 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:45.549+0000 7f1062ffd700 1 -- 172.21.15.81:0/264550567 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f105c06d2b0 con 0x7f106c060280 2023-11-15T06:33:45.755 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:45.753+0000 7f10746ae700 1 -- 172.21.15.81:0/264550567 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f106c108110 con 0x7f106c060280 2023-11-15T06:33:45.756 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:45.754+0000 7f1062ffd700 1 -- 172.21.15.81:0/264550567 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f105c010230 con 0x7f106c060280 2023-11-15T06:33:45.756 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:33:45.757 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:33:45.758 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:45.756+0000 7f10746ae700 1 -- 172.21.15.81:0/264550567 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f105802ff60 msgr2=0x7f1058032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:45.758 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:45.756+0000 7f10746ae700 1 --2- 172.21.15.81:0/264550567 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f105802ff60 0x7f1058032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f10680096a0 tx=0x7f1068006b40).stop 2023-11-15T06:33:45.758 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:45.756+0000 7f10746ae700 1 -- 172.21.15.81:0/264550567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f106c060280 msgr2=0x7f106c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:45.758 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:45.756+0000 7f10746ae700 1 --2- 172.21.15.81:0/264550567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f106c060280 0x7f106c1066e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f105c013510 tx=0x7f105c00b040).stop 2023-11-15T06:33:45.759 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:45.756+0000 7f10746ae700 1 -- 172.21.15.81:0/264550567 shutdown_connections 2023-11-15T06:33:45.759 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:45.756+0000 7f10746ae700 1 --2- 172.21.15.81:0/264550567 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f105802ff60 0x7f1058032420 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:45.759 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:45.756+0000 7f10746ae700 1 --2- 172.21.15.81:0/264550567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f106c060280 0x7f106c1066e0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:45.759 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:45.756+0000 7f10746ae700 1 -- 172.21.15.81:0/264550567 >> 172.21.15.81:0/264550567 conn(0x7f106c0f0b70 msgr2=0x7f106c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:33:45.760 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:45.756+0000 7f10746ae700 1 -- 172.21.15.81:0/264550567 shutdown_connections 2023-11-15T06:33:45.760 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:45.756+0000 7f10746ae700 1 -- 172.21.15.81:0/264550567 wait complete. 2023-11-15T06:33:45.760 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:33:46.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:45 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:33:45.754887+0000 mon.smithi062 (mon.0) 295 : audit [DBG] from='client.? 172.21.15.81:0/264550567' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:33:47.190 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:33:47.191 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:33:47.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:46 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:33:45.853551+0000 mgr.smithi062.rwurtk (mgr.14164) 244 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:33:48.466 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:48.462+0000 7f67250c8700 1 -- 172.21.15.81:0/2206647536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67200f5410 msgr2=0x7f67200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:48.466 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:48.462+0000 7f67250c8700 1 --2- 172.21.15.81:0/2206647536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67200f5410 0x7f67200f5830 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f6708009fb0 tx=0x7f6708009b30).stop 2023-11-15T06:33:48.466 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:48.462+0000 7f67250c8700 1 -- 172.21.15.81:0/2206647536 shutdown_connections 2023-11-15T06:33:48.467 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:48.462+0000 7f67250c8700 1 --2- 172.21.15.81:0/2206647536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67200f5410 0x7f67200f5830 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:48.467 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:48.462+0000 7f67250c8700 1 -- 172.21.15.81:0/2206647536 >> 172.21.15.81:0/2206647536 conn(0x7f67200f0b70 msgr2=0x7f67200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:33:48.467 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:48.462+0000 7f67250c8700 1 -- 172.21.15.81:0/2206647536 shutdown_connections 2023-11-15T06:33:48.467 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:48.462+0000 7f67250c8700 1 -- 172.21.15.81:0/2206647536 wait complete. 2023-11-15T06:33:48.467 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:48.463+0000 7f67250c8700 1 Processor -- start 2023-11-15T06:33:48.468 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:48.463+0000 7f67250c8700 1 -- start start 2023-11-15T06:33:48.468 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:48.463+0000 7f67250c8700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67200f5410 0x7f6720106760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:33:48.468 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:48.463+0000 7f67250c8700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6720106c50 con 0x7f67200f5410 2023-11-15T06:33:48.468 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:48.463+0000 7f671ed9d700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67200f5410 0x7f6720106760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:33:48.468 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:48.463+0000 7f671ed9d700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67200f5410 0x7f6720106760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:44056/0 (socket says 172.21.15.81:44056) 2023-11-15T06:33:48.469 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:48.463+0000 7f671ed9d700 1 -- 172.21.15.81:0/1834172154 learned_addr learned my addr 172.21.15.81:0/1834172154 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:33:48.469 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:48.463+0000 7f671ed9d700 1 -- 172.21.15.81:0/1834172154 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6708005040 con 0x7f67200f5410 2023-11-15T06:33:48.469 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:48.464+0000 7f671ed9d700 1 --2- 172.21.15.81:0/1834172154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67200f5410 0x7f6720106760 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f6708009f80 tx=0x7f670800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:33:48.469 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:48.464+0000 7f67177fe700 1 -- 172.21.15.81:0/1834172154 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6708009320 con 0x7f67200f5410 2023-11-15T06:33:48.469 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:48.464+0000 7f67250c8700 1 -- 172.21.15.81:0/1834172154 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6720106e50 con 0x7f67200f5410 2023-11-15T06:33:48.470 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:48.464+0000 7f67177fe700 1 -- 172.21.15.81:0/1834172154 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6708013ec0 con 0x7f67200f5410 2023-11-15T06:33:48.470 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:48.464+0000 7f67177fe700 1 -- 172.21.15.81:0/1834172154 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f670801ece0 con 0x7f67200f5410 2023-11-15T06:33:48.470 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:48.464+0000 7f67250c8700 1 -- 172.21.15.81:0/1834172154 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67201072e0 con 0x7f67200f5410 2023-11-15T06:33:48.471 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:48.465+0000 7f67177fe700 1 -- 172.21.15.81:0/1834172154 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f670801e420 con 0x7f67200f5410 2023-11-15T06:33:48.471 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:48.465+0000 7f67250c8700 1 -- 172.21.15.81:0/1834172154 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6720045ad0 con 0x7f67200f5410 2023-11-15T06:33:48.471 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:48.465+0000 7f67177fe700 1 --2- 172.21.15.81:0/1834172154 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f670c02ff60 0x7f670c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:33:48.472 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:48.465+0000 7f67177fe700 1 -- 172.21.15.81:0/1834172154 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6708040d20 con 0x7f67200f5410 2023-11-15T06:33:48.472 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:48.465+0000 7f671e59c700 1 --2- 172.21.15.81:0/1834172154 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f670c02ff60 0x7f670c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:33:48.472 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:48.465+0000 7f671e59c700 1 --2- 172.21.15.81:0/1834172154 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f670c02ff60 0x7f670c032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f67100096a0 tx=0x7f6710006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:33:48.472 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:48.470+0000 7f67177fe700 1 -- 172.21.15.81:0/1834172154 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f670801c790 con 0x7f67200f5410 2023-11-15T06:33:48.679 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:48.676+0000 7f67250c8700 1 -- 172.21.15.81:0/1834172154 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6720108190 con 0x7f67200f5410 2023-11-15T06:33:48.679 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:48.677+0000 7f67177fe700 1 -- 172.21.15.81:0/1834172154 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f670802c2d0 con 0x7f67200f5410 2023-11-15T06:33:48.680 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:33:48.680 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:33:48.681 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:48.678+0000 7f67250c8700 1 -- 172.21.15.81:0/1834172154 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f670c02ff60 msgr2=0x7f670c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:48.681 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:48.678+0000 7f67250c8700 1 --2- 172.21.15.81:0/1834172154 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f670c02ff60 0x7f670c032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f67100096a0 tx=0x7f6710006b40).stop 2023-11-15T06:33:48.682 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:48.678+0000 7f67250c8700 1 -- 172.21.15.81:0/1834172154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67200f5410 msgr2=0x7f6720106760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:48.682 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:48.678+0000 7f67250c8700 1 --2- 172.21.15.81:0/1834172154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67200f5410 0x7f6720106760 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f6708009f80 tx=0x7f670800b040).stop 2023-11-15T06:33:48.682 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:48.678+0000 7f67250c8700 1 -- 172.21.15.81:0/1834172154 shutdown_connections 2023-11-15T06:33:48.682 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:48.678+0000 7f67250c8700 1 --2- 172.21.15.81:0/1834172154 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f670c02ff60 0x7f670c032420 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:48.682 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:48.678+0000 7f67250c8700 1 --2- 172.21.15.81:0/1834172154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67200f5410 0x7f6720106760 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:48.683 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:48.678+0000 7f67250c8700 1 -- 172.21.15.81:0/1834172154 >> 172.21.15.81:0/1834172154 conn(0x7f67200f0b70 msgr2=0x7f672005fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:33:48.683 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:48.678+0000 7f67250c8700 1 -- 172.21.15.81:0/1834172154 shutdown_connections 2023-11-15T06:33:48.683 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:48.679+0000 7f67250c8700 1 -- 172.21.15.81:0/1834172154 wait complete. 2023-11-15T06:33:48.683 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:33:48.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:48 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:33:47.853869+0000 mgr.smithi062.rwurtk (mgr.14164) 245 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:33:49.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:49 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:33:48.678012+0000 mon.smithi062 (mon.0) 296 : audit [DBG] from='client.? 172.21.15.81:0/1834172154' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:33:50.077 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:33:50.078 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:33:50.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:50 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:33:49.854159+0000 mgr.smithi062.rwurtk (mgr.14164) 246 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:33:51.347 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:51.345+0000 7ff90ed74700 1 -- 172.21.15.81:0/3859664303 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff9080f5410 msgr2=0x7ff9080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:51.348 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:51.345+0000 7ff90ed74700 1 --2- 172.21.15.81:0/3859664303 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff9080f5410 0x7ff9080f5830 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7ff8fc009fb0 tx=0x7ff8fc009b30).stop 2023-11-15T06:33:51.348 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:51.345+0000 7ff90ed74700 1 -- 172.21.15.81:0/3859664303 shutdown_connections 2023-11-15T06:33:51.348 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:51.345+0000 7ff90ed74700 1 --2- 172.21.15.81:0/3859664303 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff9080f5410 0x7ff9080f5830 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:51.348 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:51.345+0000 7ff90ed74700 1 -- 172.21.15.81:0/3859664303 >> 172.21.15.81:0/3859664303 conn(0x7ff9080f0b70 msgr2=0x7ff9080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:33:51.349 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:51.345+0000 7ff90ed74700 1 -- 172.21.15.81:0/3859664303 shutdown_connections 2023-11-15T06:33:51.349 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:51.345+0000 7ff90ed74700 1 -- 172.21.15.81:0/3859664303 wait complete. 2023-11-15T06:33:51.349 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:51.345+0000 7ff90ed74700 1 Processor -- start 2023-11-15T06:33:51.349 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:51.345+0000 7ff90ed74700 1 -- start start 2023-11-15T06:33:51.349 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:51.345+0000 7ff90ed74700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff9080f5410 0x7ff90810b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:33:51.350 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:51.345+0000 7ff90ed74700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff908107240 con 0x7ff9080f5410 2023-11-15T06:33:51.351 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:51.346+0000 7ff90cb10700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff9080f5410 0x7ff90810b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:33:51.351 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:51.346+0000 7ff90cb10700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff9080f5410 0x7ff90810b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:44076/0 (socket says 172.21.15.81:44076) 2023-11-15T06:33:51.352 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:51.346+0000 7ff90cb10700 1 -- 172.21.15.81:0/2503363544 learned_addr learned my addr 172.21.15.81:0/2503363544 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:33:51.352 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:51.346+0000 7ff90cb10700 1 -- 172.21.15.81:0/2503363544 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8fc005040 con 0x7ff9080f5410 2023-11-15T06:33:51.352 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:51.346+0000 7ff90cb10700 1 --2- 172.21.15.81:0/2503363544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff9080f5410 0x7ff90810b850 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7ff8fc013510 tx=0x7ff8fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:33:51.352 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:51.346+0000 7ff9057fa700 1 -- 172.21.15.81:0/2503363544 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8fc009320 con 0x7ff9080f5410 2023-11-15T06:33:51.352 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:51.346+0000 7ff90ed74700 1 -- 172.21.15.81:0/2503363544 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff908060220 con 0x7ff9080f5410 2023-11-15T06:33:51.353 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:51.346+0000 7ff9057fa700 1 -- 172.21.15.81:0/2503363544 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff8fc013f00 con 0x7ff9080f5410 2023-11-15T06:33:51.353 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:51.346+0000 7ff9057fa700 1 -- 172.21.15.81:0/2503363544 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8fc014d90 con 0x7ff9080f5410 2023-11-15T06:33:51.353 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:51.347+0000 7ff90ed74700 1 -- 172.21.15.81:0/2503363544 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff90810bd40 con 0x7ff9080f5410 2023-11-15T06:33:51.353 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:51.347+0000 7ff9057fa700 1 -- 172.21.15.81:0/2503363544 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff8fc014420 con 0x7ff9080f5410 2023-11-15T06:33:51.353 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:51.347+0000 7ff90ed74700 1 -- 172.21.15.81:0/2503363544 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff908045ad0 con 0x7ff9080f5410 2023-11-15T06:33:51.354 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:51.347+0000 7ff9057fa700 1 --2- 172.21.15.81:0/2503363544 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff8f802ff60 0x7ff8f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:33:51.354 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:51.347+0000 7ff9057fa700 1 -- 172.21.15.81:0/2503363544 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff8fc041000 con 0x7ff9080f5410 2023-11-15T06:33:51.354 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:51.348+0000 7ff907fff700 1 --2- 172.21.15.81:0/2503363544 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff8f802ff60 0x7ff8f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:33:51.354 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:51.348+0000 7ff907fff700 1 --2- 172.21.15.81:0/2503363544 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff8f802ff60 0x7ff8f8032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7ff8f40096a0 tx=0x7ff8f4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:33:51.355 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:51.352+0000 7ff9057fa700 1 -- 172.21.15.81:0/2503363544 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff8fc06d2b0 con 0x7ff9080f5410 2023-11-15T06:33:51.556 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:51.554+0000 7ff90ed74700 1 -- 172.21.15.81:0/2503363544 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff9080faf50 con 0x7ff9080f5410 2023-11-15T06:33:51.557 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:51.555+0000 7ff9057fa700 1 -- 172.21.15.81:0/2503363544 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff8fc01db70 con 0x7ff9080f5410 2023-11-15T06:33:51.557 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:33:51.558 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:33:51.558 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:51.556+0000 7ff90ed74700 1 -- 172.21.15.81:0/2503363544 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff8f802ff60 msgr2=0x7ff8f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:51.559 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:51.556+0000 7ff90ed74700 1 --2- 172.21.15.81:0/2503363544 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff8f802ff60 0x7ff8f8032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7ff8f40096a0 tx=0x7ff8f4006b40).stop 2023-11-15T06:33:51.559 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:51.556+0000 7ff90ed74700 1 -- 172.21.15.81:0/2503363544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff9080f5410 msgr2=0x7ff90810b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:51.559 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:51.556+0000 7ff90ed74700 1 --2- 172.21.15.81:0/2503363544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff9080f5410 0x7ff90810b850 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7ff8fc013510 tx=0x7ff8fc00b040).stop 2023-11-15T06:33:51.559 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:51.556+0000 7ff90ed74700 1 -- 172.21.15.81:0/2503363544 shutdown_connections 2023-11-15T06:33:51.560 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:51.556+0000 7ff90ed74700 1 --2- 172.21.15.81:0/2503363544 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7ff8f802ff60 0x7ff8f8032420 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:51.560 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:51.556+0000 7ff90ed74700 1 --2- 172.21.15.81:0/2503363544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff9080f5410 0x7ff90810b850 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:51.560 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:51.556+0000 7ff90ed74700 1 -- 172.21.15.81:0/2503363544 >> 172.21.15.81:0/2503363544 conn(0x7ff9080f0b70 msgr2=0x7ff9080f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:33:51.560 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:51.556+0000 7ff90ed74700 1 -- 172.21.15.81:0/2503363544 shutdown_connections 2023-11-15T06:33:51.560 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:51.556+0000 7ff90ed74700 1 -- 172.21.15.81:0/2503363544 wait complete. 2023-11-15T06:33:51.561 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:33:51.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:51 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:33:51.555628+0000 mon.smithi062 (mon.0) 297 : audit [DBG] from='client.? 172.21.15.81:0/2503363544' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:33:52.965 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:33:52.965 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:33:53.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:52 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:33:51.854527+0000 mgr.smithi062.rwurtk (mgr.14164) 247 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:33:54.190 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:54.186+0000 7f57f8da5700 1 -- 172.21.15.81:0/2004618960 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57f40f6510 msgr2=0x7f57f40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:54.191 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:54.186+0000 7f57f8da5700 1 --2- 172.21.15.81:0/2004618960 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57f40f6510 0x7f57f40f6930 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f57e8009fb0 tx=0x7f57e8009b30).stop 2023-11-15T06:33:54.191 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:54.186+0000 7f57f8da5700 1 -- 172.21.15.81:0/2004618960 shutdown_connections 2023-11-15T06:33:54.191 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:54.186+0000 7f57f8da5700 1 --2- 172.21.15.81:0/2004618960 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57f40f6510 0x7f57f40f6930 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:54.192 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:54.186+0000 7f57f8da5700 1 -- 172.21.15.81:0/2004618960 >> 172.21.15.81:0/2004618960 conn(0x7f57f40f1c70 msgr2=0x7f57f40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:33:54.192 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:54.186+0000 7f57f8da5700 1 -- 172.21.15.81:0/2004618960 shutdown_connections 2023-11-15T06:33:54.192 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:54.186+0000 7f57f8da5700 1 -- 172.21.15.81:0/2004618960 wait complete. 2023-11-15T06:33:54.192 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:54.187+0000 7f57f8da5700 1 Processor -- start 2023-11-15T06:33:54.193 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:54.187+0000 7f57f8da5700 1 -- start start 2023-11-15T06:33:54.193 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:54.187+0000 7f57f8da5700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57f40f6510 0x7f57f406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:33:54.194 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:54.187+0000 7f57f8da5700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57f4069e20 con 0x7f57f40f6510 2023-11-15T06:33:54.194 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:54.187+0000 7f57f259c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57f40f6510 0x7f57f406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:33:54.194 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:54.187+0000 7f57f259c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57f40f6510 0x7f57f406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:54554/0 (socket says 172.21.15.81:54554) 2023-11-15T06:33:54.194 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:54.187+0000 7f57f259c700 1 -- 172.21.15.81:0/4132863283 learned_addr learned my addr 172.21.15.81:0/4132863283 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:33:54.195 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:54.188+0000 7f57f259c700 1 -- 172.21.15.81:0/4132863283 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e8005040 con 0x7f57f40f6510 2023-11-15T06:33:54.195 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:54.188+0000 7f57f259c700 1 --2- 172.21.15.81:0/4132863283 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57f40f6510 0x7f57f406b9a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f57e8014000 tx=0x7f57e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:33:54.195 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:54.188+0000 7f57e2ffd700 1 -- 172.21.15.81:0/4132863283 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57e80095f0 con 0x7f57f40f6510 2023-11-15T06:33:54.195 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:54.188+0000 7f57f8da5700 1 -- 172.21.15.81:0/4132863283 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57f4069fc0 con 0x7f57f40f6510 2023-11-15T06:33:54.196 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:54.188+0000 7f57e2ffd700 1 -- 172.21.15.81:0/4132863283 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f57e8007e70 con 0x7f57f40f6510 2023-11-15T06:33:54.196 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:54.188+0000 7f57e2ffd700 1 -- 172.21.15.81:0/4132863283 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57e8014ea0 con 0x7f57f40f6510 2023-11-15T06:33:54.196 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:54.188+0000 7f57f8da5700 1 -- 172.21.15.81:0/4132863283 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57f406a450 con 0x7f57f40f6510 2023-11-15T06:33:54.197 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:54.189+0000 7f57e2ffd700 1 -- 172.21.15.81:0/4132863283 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f57e801d4e0 con 0x7f57f40f6510 2023-11-15T06:33:54.197 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:54.189+0000 7f57e2ffd700 1 --2- 172.21.15.81:0/4132863283 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f57dc02ff60 0x7f57dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:33:54.198 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:54.189+0000 7f57e2ffd700 1 -- 172.21.15.81:0/4132863283 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f57e8040890 con 0x7f57f40f6510 2023-11-15T06:33:54.198 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:54.189+0000 7f57f8da5700 1 -- 172.21.15.81:0/4132863283 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57f4059100 con 0x7f57f40f6510 2023-11-15T06:33:54.198 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:54.189+0000 7f57f1d9b700 1 --2- 172.21.15.81:0/4132863283 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f57dc02ff60 0x7f57dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:33:54.199 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:54.190+0000 7f57f1d9b700 1 --2- 172.21.15.81:0/4132863283 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f57dc02ff60 0x7f57dc032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f57e40096a0 tx=0x7f57e4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:33:54.199 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:54.194+0000 7f57e2ffd700 1 -- 172.21.15.81:0/4132863283 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f57e800f8f0 con 0x7f57f40f6510 2023-11-15T06:33:54.399 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:54.397+0000 7f57f8da5700 1 -- 172.21.15.81:0/4132863283 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f57f40fbf60 con 0x7f57f40f6510 2023-11-15T06:33:54.400 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:54.398+0000 7f57e2ffd700 1 -- 172.21.15.81:0/4132863283 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f57e802c090 con 0x7f57f40f6510 2023-11-15T06:33:54.400 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:33:54.401 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:33:54.402 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:54.399+0000 7f57f8da5700 1 -- 172.21.15.81:0/4132863283 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f57dc02ff60 msgr2=0x7f57dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:54.402 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:54.399+0000 7f57f8da5700 1 --2- 172.21.15.81:0/4132863283 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f57dc02ff60 0x7f57dc032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f57e40096a0 tx=0x7f57e4006b40).stop 2023-11-15T06:33:54.402 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:54.399+0000 7f57f8da5700 1 -- 172.21.15.81:0/4132863283 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57f40f6510 msgr2=0x7f57f406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:54.403 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:54.399+0000 7f57f8da5700 1 --2- 172.21.15.81:0/4132863283 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57f40f6510 0x7f57f406b9a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f57e8014000 tx=0x7f57e800b040).stop 2023-11-15T06:33:54.403 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:54.399+0000 7f57f8da5700 1 -- 172.21.15.81:0/4132863283 shutdown_connections 2023-11-15T06:33:54.403 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:54.399+0000 7f57f8da5700 1 --2- 172.21.15.81:0/4132863283 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f57dc02ff60 0x7f57dc032420 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:54.403 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:54.399+0000 7f57f8da5700 1 --2- 172.21.15.81:0/4132863283 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f57f40f6510 0x7f57f406b9a0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:54.403 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:54.399+0000 7f57f8da5700 1 -- 172.21.15.81:0/4132863283 >> 172.21.15.81:0/4132863283 conn(0x7f57f40f1c70 msgr2=0x7f57f40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:33:54.404 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:54.399+0000 7f57f8da5700 1 -- 172.21.15.81:0/4132863283 shutdown_connections 2023-11-15T06:33:54.404 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:54.399+0000 7f57f8da5700 1 -- 172.21.15.81:0/4132863283 wait complete. 2023-11-15T06:33:54.404 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:33:55.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:54 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:33:53.854918+0000 mgr.smithi062.rwurtk (mgr.14164) 248 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:33:55.169 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:54 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:33:53.924689+0000 mon.smithi062 (mon.0) 298 : audit [INF] from='mgr.14164 172.21.15.62:0/147479783' entity='mgr.smithi062.rwurtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi062.rwurtk/mirror_snapshot_schedule"}]: dispatch 2023-11-15T06:33:55.169 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:54 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:33:53.924904+0000 mon.smithi062 (mon.0) 299 : audit [INF] from='mgr.14164 172.21.15.62:0/147479783' entity='mgr.smithi062.rwurtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi062.rwurtk/trash_purge_schedule"}]: dispatch 2023-11-15T06:33:55.169 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:54 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:33:54.398651+0000 mon.smithi062 (mon.0) 300 : audit [DBG] from='client.? 172.21.15.81:0/4132863283' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:33:55.801 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:33:55.801 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:33:57.018 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:57.015+0000 7fa14c485700 1 -- 172.21.15.81:0/1525944866 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1440f5410 msgr2=0x7fa1440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:57.019 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:57.015+0000 7fa14c485700 1 --2- 172.21.15.81:0/1525944866 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1440f5410 0x7fa1440f5830 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7fa138009fb0 tx=0x7fa138009b30).stop 2023-11-15T06:33:57.019 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:57.015+0000 7fa14c485700 1 -- 172.21.15.81:0/1525944866 shutdown_connections 2023-11-15T06:33:57.019 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:57.015+0000 7fa14c485700 1 --2- 172.21.15.81:0/1525944866 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1440f5410 0x7fa1440f5830 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:57.019 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:57.015+0000 7fa14c485700 1 -- 172.21.15.81:0/1525944866 >> 172.21.15.81:0/1525944866 conn(0x7fa1440f0b70 msgr2=0x7fa1440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:33:57.019 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:57.015+0000 7fa14c485700 1 -- 172.21.15.81:0/1525944866 shutdown_connections 2023-11-15T06:33:57.020 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:57.016+0000 7fa14c485700 1 -- 172.21.15.81:0/1525944866 wait complete. 2023-11-15T06:33:57.020 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:57.016+0000 7fa14c485700 1 Processor -- start 2023-11-15T06:33:57.020 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:57.016+0000 7fa14c485700 1 -- start start 2023-11-15T06:33:57.020 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:57.016+0000 7fa14c485700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1440f5410 0x7fa144105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:33:57.020 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:57.016+0000 7fa14c485700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa144105b20 con 0x7fa1440f5410 2023-11-15T06:33:57.023 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:57.017+0000 7fa14a221700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1440f5410 0x7fa144105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:33:57.024 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:57.017+0000 7fa14a221700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1440f5410 0x7fa144105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:54560/0 (socket says 172.21.15.81:54560) 2023-11-15T06:33:57.024 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:57.017+0000 7fa14a221700 1 -- 172.21.15.81:0/1223493560 learned_addr learned my addr 172.21.15.81:0/1223493560 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:33:57.024 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:57.017+0000 7fa14a221700 1 -- 172.21.15.81:0/1223493560 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa138005040 con 0x7fa1440f5410 2023-11-15T06:33:57.024 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:57.017+0000 7fa14a221700 1 --2- 172.21.15.81:0/1223493560 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1440f5410 0x7fa144105630 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fa138013510 tx=0x7fa13800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:33:57.025 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:57.017+0000 7fa136ffd700 1 -- 172.21.15.81:0/1223493560 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa138009320 con 0x7fa1440f5410 2023-11-15T06:33:57.025 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:57.017+0000 7fa14c485700 1 -- 172.21.15.81:0/1223493560 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa144105d20 con 0x7fa1440f5410 2023-11-15T06:33:57.025 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:57.017+0000 7fa136ffd700 1 -- 172.21.15.81:0/1223493560 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa138013f00 con 0x7fa1440f5410 2023-11-15T06:33:57.025 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:57.017+0000 7fa136ffd700 1 -- 172.21.15.81:0/1223493560 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa138014d90 con 0x7fa1440f5410 2023-11-15T06:33:57.025 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:57.017+0000 7fa14c485700 1 -- 172.21.15.81:0/1223493560 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1441061b0 con 0x7fa1440f5410 2023-11-15T06:33:57.026 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:57.018+0000 7fa136ffd700 1 -- 172.21.15.81:0/1223493560 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa1380095f0 con 0x7fa1440f5410 2023-11-15T06:33:57.026 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:57.018+0000 7fa14c485700 1 -- 172.21.15.81:0/1223493560 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa144045ad0 con 0x7fa1440f5410 2023-11-15T06:33:57.026 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:57.018+0000 7fa136ffd700 1 --2- 172.21.15.81:0/1223493560 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fa13002ff60 0x7fa130032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:33:57.026 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:57.018+0000 7fa136ffd700 1 -- 172.21.15.81:0/1223493560 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa138040fe0 con 0x7fa1440f5410 2023-11-15T06:33:57.026 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:57.018+0000 7fa149a20700 1 --2- 172.21.15.81:0/1223493560 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fa13002ff60 0x7fa130032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:33:57.027 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:57.019+0000 7fa149a20700 1 --2- 172.21.15.81:0/1223493560 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fa13002ff60 0x7fa130032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fa1400096a0 tx=0x7fa140006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:33:57.027 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:57.023+0000 7fa136ffd700 1 -- 172.21.15.81:0/1223493560 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa13801d7f0 con 0x7fa1440f5410 2023-11-15T06:33:57.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:56 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:33:55.855255+0000 mgr.smithi062.rwurtk (mgr.14164) 249 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:33:57.229 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:57.227+0000 7fa14c485700 1 -- 172.21.15.81:0/1223493560 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa1440fae60 con 0x7fa1440f5410 2023-11-15T06:33:57.230 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:57.228+0000 7fa136ffd700 1 -- 172.21.15.81:0/1223493560 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa138010230 con 0x7fa1440f5410 2023-11-15T06:33:57.230 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:33:57.231 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:33:57.232 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:57.229+0000 7fa14c485700 1 -- 172.21.15.81:0/1223493560 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fa13002ff60 msgr2=0x7fa130032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:57.232 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:57.229+0000 7fa14c485700 1 --2- 172.21.15.81:0/1223493560 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fa13002ff60 0x7fa130032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fa1400096a0 tx=0x7fa140006b40).stop 2023-11-15T06:33:57.232 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:57.229+0000 7fa14c485700 1 -- 172.21.15.81:0/1223493560 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1440f5410 msgr2=0x7fa144105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:57.232 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:57.229+0000 7fa14c485700 1 --2- 172.21.15.81:0/1223493560 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1440f5410 0x7fa144105630 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fa138013510 tx=0x7fa13800b040).stop 2023-11-15T06:33:57.232 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:57.229+0000 7fa14c485700 1 -- 172.21.15.81:0/1223493560 shutdown_connections 2023-11-15T06:33:57.233 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:57.230+0000 7fa14c485700 1 --2- 172.21.15.81:0/1223493560 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fa13002ff60 0x7fa130032420 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:57.233 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:57.230+0000 7fa14c485700 1 --2- 172.21.15.81:0/1223493560 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1440f5410 0x7fa144105630 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:57.233 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:57.230+0000 7fa14c485700 1 -- 172.21.15.81:0/1223493560 >> 172.21.15.81:0/1223493560 conn(0x7fa1440f0b70 msgr2=0x7fa1440f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:33:57.233 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:57.230+0000 7fa14c485700 1 -- 172.21.15.81:0/1223493560 shutdown_connections 2023-11-15T06:33:57.233 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:57.230+0000 7fa14c485700 1 -- 172.21.15.81:0/1223493560 wait complete. 2023-11-15T06:33:57.234 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:33:58.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:57 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:33:57.228675+0000 mon.smithi062 (mon.0) 301 : audit [DBG] from='client.? 172.21.15.81:0/1223493560' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:33:58.665 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:33:58.666 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:33:59.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:33:58 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:33:57.855582+0000 mgr.smithi062.rwurtk (mgr.14164) 250 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:33:59.975 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:59.971+0000 7fe343677700 1 -- 172.21.15.81:0/2110988081 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe33c0f6510 msgr2=0x7fe33c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:33:59.975 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:59.971+0000 7fe343677700 1 --2- 172.21.15.81:0/2110988081 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe33c0f6510 0x7fe33c0f6930 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7fe338009fb0 tx=0x7fe338009b30).stop 2023-11-15T06:33:59.975 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:59.971+0000 7fe343677700 1 -- 172.21.15.81:0/2110988081 shutdown_connections 2023-11-15T06:33:59.975 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:59.971+0000 7fe343677700 1 --2- 172.21.15.81:0/2110988081 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe33c0f6510 0x7fe33c0f6930 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:33:59.976 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:59.971+0000 7fe343677700 1 -- 172.21.15.81:0/2110988081 >> 172.21.15.81:0/2110988081 conn(0x7fe33c0f1c70 msgr2=0x7fe33c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:33:59.976 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:59.971+0000 7fe343677700 1 -- 172.21.15.81:0/2110988081 shutdown_connections 2023-11-15T06:33:59.976 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:59.971+0000 7fe343677700 1 -- 172.21.15.81:0/2110988081 wait complete. 2023-11-15T06:33:59.976 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:59.971+0000 7fe343677700 1 Processor -- start 2023-11-15T06:33:59.976 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:59.972+0000 7fe343677700 1 -- start start 2023-11-15T06:33:59.977 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:59.972+0000 7fe343677700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe33c0f6510 0x7fe33c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:33:59.977 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:59.972+0000 7fe343677700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe33c069e20 con 0x7fe33c0f6510 2023-11-15T06:33:59.977 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:59.972+0000 7fe341413700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe33c0f6510 0x7fe33c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:33:59.977 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:59.972+0000 7fe341413700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe33c0f6510 0x7fe33c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:54576/0 (socket says 172.21.15.81:54576) 2023-11-15T06:33:59.977 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:59.972+0000 7fe341413700 1 -- 172.21.15.81:0/3923589923 learned_addr learned my addr 172.21.15.81:0/3923589923 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:33:59.978 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:59.972+0000 7fe341413700 1 -- 172.21.15.81:0/3923589923 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe338005040 con 0x7fe33c0f6510 2023-11-15T06:33:59.978 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:59.972+0000 7fe341413700 1 --2- 172.21.15.81:0/3923589923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe33c0f6510 0x7fe33c06b9a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fe338014000 tx=0x7fe33800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:33:59.978 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:59.972+0000 7fe331ffb700 1 -- 172.21.15.81:0/3923589923 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3380095f0 con 0x7fe33c0f6510 2023-11-15T06:33:59.978 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:59.972+0000 7fe343677700 1 -- 172.21.15.81:0/3923589923 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe33c069fc0 con 0x7fe33c0f6510 2023-11-15T06:33:59.978 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:59.972+0000 7fe331ffb700 1 -- 172.21.15.81:0/3923589923 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe338007e70 con 0x7fe33c0f6510 2023-11-15T06:33:59.979 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:59.972+0000 7fe331ffb700 1 -- 172.21.15.81:0/3923589923 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe338014ea0 con 0x7fe33c0f6510 2023-11-15T06:33:59.979 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:59.973+0000 7fe343677700 1 -- 172.21.15.81:0/3923589923 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe33c06a450 con 0x7fe33c0f6510 2023-11-15T06:33:59.980 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:59.973+0000 7fe331ffb700 1 -- 172.21.15.81:0/3923589923 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe338025070 con 0x7fe33c0f6510 2023-11-15T06:33:59.980 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:59.973+0000 7fe343677700 1 -- 172.21.15.81:0/3923589923 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe33c059100 con 0x7fe33c0f6510 2023-11-15T06:33:59.980 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:59.974+0000 7fe331ffb700 1 --2- 172.21.15.81:0/3923589923 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fe32802ff60 0x7fe328032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:33:59.980 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:59.974+0000 7fe331ffb700 1 -- 172.21.15.81:0/3923589923 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe338040970 con 0x7fe33c0f6510 2023-11-15T06:33:59.981 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:59.974+0000 7fe340c12700 1 --2- 172.21.15.81:0/3923589923 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fe32802ff60 0x7fe328032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:33:59.981 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:59.974+0000 7fe340c12700 1 --2- 172.21.15.81:0/3923589923 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fe32802ff60 0x7fe328032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fe32c0096a0 tx=0x7fe32c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:33:59.981 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:33:59.978+0000 7fe331ffb700 1 -- 172.21.15.81:0/3923589923 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe3380079c0 con 0x7fe33c0f6510 2023-11-15T06:34:00.185 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:00.183+0000 7fe343677700 1 -- 172.21.15.81:0/3923589923 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe33c0fbf60 con 0x7fe33c0f6510 2023-11-15T06:34:00.186 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:00.184+0000 7fe331ffb700 1 -- 172.21.15.81:0/3923589923 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe33800f300 con 0x7fe33c0f6510 2023-11-15T06:34:00.186 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:34:00.187 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:34:00.187 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:00.185+0000 7fe343677700 1 -- 172.21.15.81:0/3923589923 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fe32802ff60 msgr2=0x7fe328032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:00.188 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:00.185+0000 7fe343677700 1 --2- 172.21.15.81:0/3923589923 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fe32802ff60 0x7fe328032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fe32c0096a0 tx=0x7fe32c006b40).stop 2023-11-15T06:34:00.188 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:00.185+0000 7fe343677700 1 -- 172.21.15.81:0/3923589923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe33c0f6510 msgr2=0x7fe33c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:00.188 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:00.185+0000 7fe343677700 1 --2- 172.21.15.81:0/3923589923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe33c0f6510 0x7fe33c06b9a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fe338014000 tx=0x7fe33800b040).stop 2023-11-15T06:34:00.188 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:00.185+0000 7fe343677700 1 -- 172.21.15.81:0/3923589923 shutdown_connections 2023-11-15T06:34:00.188 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:00.185+0000 7fe343677700 1 --2- 172.21.15.81:0/3923589923 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fe32802ff60 0x7fe328032420 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:00.189 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:00.185+0000 7fe343677700 1 --2- 172.21.15.81:0/3923589923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe33c0f6510 0x7fe33c06b9a0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:00.189 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:00.185+0000 7fe343677700 1 -- 172.21.15.81:0/3923589923 >> 172.21.15.81:0/3923589923 conn(0x7fe33c0f1c70 msgr2=0x7fe33c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:34:00.189 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:00.185+0000 7fe343677700 1 -- 172.21.15.81:0/3923589923 shutdown_connections 2023-11-15T06:34:00.190 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:00.185+0000 7fe343677700 1 -- 172.21.15.81:0/3923589923 wait complete. 2023-11-15T06:34:00.191 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:34:01.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:00 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:33:59.855932+0000 mgr.smithi062.rwurtk (mgr.14164) 251 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:34:01.169 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:00 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:34:00.184730+0000 mon.smithi062 (mon.0) 302 : audit [DBG] from='client.? 172.21.15.81:0/3923589923' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:34:01.580 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:34:01.581 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:34:02.775 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:02.770+0000 7f3c69b45700 1 -- 172.21.15.81:0/1092058002 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c64060280 msgr2=0x7f3c640f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:02.775 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:02.770+0000 7f3c69b45700 1 --2- 172.21.15.81:0/1092058002 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c64060280 0x7f3c640f90f0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f3c54009fb0 tx=0x7f3c54009b30).stop 2023-11-15T06:34:02.776 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:02.770+0000 7f3c69b45700 1 -- 172.21.15.81:0/1092058002 shutdown_connections 2023-11-15T06:34:02.776 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:02.770+0000 7f3c69b45700 1 --2- 172.21.15.81:0/1092058002 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c64060280 0x7f3c640f90f0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:02.776 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:02.770+0000 7f3c69b45700 1 -- 172.21.15.81:0/1092058002 >> 172.21.15.81:0/1092058002 conn(0x7f3c640f0b70 msgr2=0x7f3c640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:34:02.776 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:02.770+0000 7f3c69b45700 1 -- 172.21.15.81:0/1092058002 shutdown_connections 2023-11-15T06:34:02.777 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:02.770+0000 7f3c69b45700 1 -- 172.21.15.81:0/1092058002 wait complete. 2023-11-15T06:34:02.777 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:02.771+0000 7f3c69b45700 1 Processor -- start 2023-11-15T06:34:02.777 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:02.771+0000 7f3c69b45700 1 -- start start 2023-11-15T06:34:02.777 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:02.771+0000 7f3c69b45700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c64060280 0x7f3c641066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:34:02.777 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:02.771+0000 7f3c69b45700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c64106bd0 con 0x7f3c64060280 2023-11-15T06:34:02.778 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:02.771+0000 7f3c637fe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c64060280 0x7f3c641066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:34:02.778 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:02.771+0000 7f3c637fe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c64060280 0x7f3c641066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:54826/0 (socket says 172.21.15.81:54826) 2023-11-15T06:34:02.778 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:02.771+0000 7f3c637fe700 1 -- 172.21.15.81:0/1982647764 learned_addr learned my addr 172.21.15.81:0/1982647764 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:34:02.778 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:02.772+0000 7f3c637fe700 1 -- 172.21.15.81:0/1982647764 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c54005040 con 0x7f3c64060280 2023-11-15T06:34:02.778 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:02.772+0000 7f3c637fe700 1 --2- 172.21.15.81:0/1982647764 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c64060280 0x7f3c641066e0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f3c54013510 tx=0x7f3c5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:34:02.779 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:02.772+0000 7f3c4bfff700 1 -- 172.21.15.81:0/1982647764 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c54009320 con 0x7f3c64060280 2023-11-15T06:34:02.779 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:02.772+0000 7f3c69b45700 1 -- 172.21.15.81:0/1982647764 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c64106dd0 con 0x7f3c64060280 2023-11-15T06:34:02.779 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:02.772+0000 7f3c4bfff700 1 -- 172.21.15.81:0/1982647764 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3c54013f00 con 0x7f3c64060280 2023-11-15T06:34:02.779 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:02.772+0000 7f3c4bfff700 1 -- 172.21.15.81:0/1982647764 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c54014dc0 con 0x7f3c64060280 2023-11-15T06:34:02.780 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:02.772+0000 7f3c69b45700 1 -- 172.21.15.81:0/1982647764 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c64107260 con 0x7f3c64060280 2023-11-15T06:34:02.781 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:02.773+0000 7f3c4bfff700 1 -- 172.21.15.81:0/1982647764 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3c540095f0 con 0x7f3c64060280 2023-11-15T06:34:02.781 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:02.773+0000 7f3c69b45700 1 -- 172.21.15.81:0/1982647764 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c64045ad0 con 0x7f3c64060280 2023-11-15T06:34:02.781 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:02.773+0000 7f3c4bfff700 1 --2- 172.21.15.81:0/1982647764 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f3c4c02ff60 0x7f3c4c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:34:02.781 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:02.773+0000 7f3c4bfff700 1 -- 172.21.15.81:0/1982647764 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3c54041000 con 0x7f3c64060280 2023-11-15T06:34:02.782 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:02.773+0000 7f3c62ffd700 1 --2- 172.21.15.81:0/1982647764 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f3c4c02ff60 0x7f3c4c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:34:02.782 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:02.774+0000 7f3c62ffd700 1 --2- 172.21.15.81:0/1982647764 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f3c4c02ff60 0x7f3c4c032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f3c580096a0 tx=0x7f3c58006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:34:02.782 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:02.778+0000 7f3c4bfff700 1 -- 172.21.15.81:0/1982647764 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3c5401c790 con 0x7f3c64060280 2023-11-15T06:34:02.986 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:02.982+0000 7f3c69b45700 1 -- 172.21.15.81:0/1982647764 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3c64108110 con 0x7f3c64060280 2023-11-15T06:34:02.986 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:02.983+0000 7f3c4bfff700 1 -- 172.21.15.81:0/1982647764 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3c54010230 con 0x7f3c64060280 2023-11-15T06:34:02.987 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:34:02.987 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:34:02.988 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:02.985+0000 7f3c69b45700 1 -- 172.21.15.81:0/1982647764 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f3c4c02ff60 msgr2=0x7f3c4c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:02.988 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:02.985+0000 7f3c69b45700 1 --2- 172.21.15.81:0/1982647764 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f3c4c02ff60 0x7f3c4c032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f3c580096a0 tx=0x7f3c58006b40).stop 2023-11-15T06:34:02.988 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:02.985+0000 7f3c69b45700 1 -- 172.21.15.81:0/1982647764 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c64060280 msgr2=0x7f3c641066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:02.989 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:02.985+0000 7f3c69b45700 1 --2- 172.21.15.81:0/1982647764 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c64060280 0x7f3c641066e0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f3c54013510 tx=0x7f3c5400b040).stop 2023-11-15T06:34:02.989 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:02.985+0000 7f3c69b45700 1 -- 172.21.15.81:0/1982647764 shutdown_connections 2023-11-15T06:34:02.989 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:02.985+0000 7f3c69b45700 1 --2- 172.21.15.81:0/1982647764 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f3c4c02ff60 0x7f3c4c032420 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:02.989 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:02.985+0000 7f3c69b45700 1 --2- 172.21.15.81:0/1982647764 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c64060280 0x7f3c641066e0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:02.989 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:02.985+0000 7f3c69b45700 1 -- 172.21.15.81:0/1982647764 >> 172.21.15.81:0/1982647764 conn(0x7f3c640f0b70 msgr2=0x7f3c640f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:34:02.990 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:02.985+0000 7f3c69b45700 1 -- 172.21.15.81:0/1982647764 shutdown_connections 2023-11-15T06:34:02.990 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:02.985+0000 7f3c69b45700 1 -- 172.21.15.81:0/1982647764 wait complete. 2023-11-15T06:34:02.990 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:34:03.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:34:01.856326+0000 mgr.smithi062.rwurtk (mgr.14164) 252 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:34:04.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:03 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:34:02.985067+0000 mon.smithi062 (mon.0) 303 : audit [DBG] from='client.? 172.21.15.81:0/1982647764' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:34:04.425 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:34:04.425 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:34:05.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:04 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:34:03.856714+0000 mgr.smithi062.rwurtk (mgr.14164) 253 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:34:05.714 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:05.710+0000 7fc2aa159700 1 -- 172.21.15.81:0/3269273265 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2a40f51a0 msgr2=0x7fc2a40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:05.715 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:05.710+0000 7fc2aa159700 1 --2- 172.21.15.81:0/3269273265 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2a40f51a0 0x7fc2a40f55c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7fc2a0009fb0 tx=0x7fc2a0009b30).stop 2023-11-15T06:34:05.715 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:05.710+0000 7fc2aa159700 1 -- 172.21.15.81:0/3269273265 shutdown_connections 2023-11-15T06:34:05.715 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:05.710+0000 7fc2aa159700 1 --2- 172.21.15.81:0/3269273265 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2a40f51a0 0x7fc2a40f55c0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:05.715 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:05.710+0000 7fc2aa159700 1 -- 172.21.15.81:0/3269273265 >> 172.21.15.81:0/3269273265 conn(0x7fc2a40f09a0 msgr2=0x7fc2a40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:34:05.716 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:05.710+0000 7fc2aa159700 1 -- 172.21.15.81:0/3269273265 shutdown_connections 2023-11-15T06:34:05.716 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:05.710+0000 7fc2aa159700 1 -- 172.21.15.81:0/3269273265 wait complete. 2023-11-15T06:34:05.716 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:05.710+0000 7fc2aa159700 1 Processor -- start 2023-11-15T06:34:05.716 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:05.710+0000 7fc2aa159700 1 -- start start 2023-11-15T06:34:05.716 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:05.710+0000 7fc2aa159700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2a40f51a0 0x7fc2a41053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:34:05.717 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:05.710+0000 7fc2aa159700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a41058a0 con 0x7fc2a40f51a0 2023-11-15T06:34:05.717 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:05.711+0000 7fc2a9157700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2a40f51a0 0x7fc2a41053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:34:05.717 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:05.711+0000 7fc2a9157700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2a40f51a0 0x7fc2a41053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:54842/0 (socket says 172.21.15.81:54842) 2023-11-15T06:34:05.717 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:05.711+0000 7fc2a9157700 1 -- 172.21.15.81:0/1404987086 learned_addr learned my addr 172.21.15.81:0/1404987086 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:34:05.717 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:05.711+0000 7fc2a9157700 1 -- 172.21.15.81:0/1404987086 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a0005040 con 0x7fc2a40f51a0 2023-11-15T06:34:05.717 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:05.711+0000 7fc2a9157700 1 --2- 172.21.15.81:0/1404987086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2a40f51a0 0x7fc2a41053b0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7fc2a0013510 tx=0x7fc2a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:34:05.718 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:05.711+0000 7fc299ffb700 1 -- 172.21.15.81:0/1404987086 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2a0009320 con 0x7fc2a40f51a0 2023-11-15T06:34:05.718 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:05.711+0000 7fc2aa159700 1 -- 172.21.15.81:0/1404987086 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2a4105aa0 con 0x7fc2a40f51a0 2023-11-15T06:34:05.718 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:05.711+0000 7fc299ffb700 1 -- 172.21.15.81:0/1404987086 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc2a0013f00 con 0x7fc2a40f51a0 2023-11-15T06:34:05.718 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:05.711+0000 7fc299ffb700 1 -- 172.21.15.81:0/1404987086 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2a0014dc0 con 0x7fc2a40f51a0 2023-11-15T06:34:05.718 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:05.712+0000 7fc2aa159700 1 -- 172.21.15.81:0/1404987086 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2a4105f30 con 0x7fc2a40f51a0 2023-11-15T06:34:05.719 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:05.712+0000 7fc299ffb700 1 -- 172.21.15.81:0/1404987086 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc2a00095f0 con 0x7fc2a40f51a0 2023-11-15T06:34:05.720 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:05.712+0000 7fc2aa159700 1 -- 172.21.15.81:0/1404987086 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2a4045ad0 con 0x7fc2a40f51a0 2023-11-15T06:34:05.720 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:05.712+0000 7fc299ffb700 1 --2- 172.21.15.81:0/1404987086 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fc29002ff60 0x7fc290032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:34:05.720 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:05.713+0000 7fc299ffb700 1 -- 172.21.15.81:0/1404987086 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc2a0041040 con 0x7fc2a40f51a0 2023-11-15T06:34:05.720 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:05.713+0000 7fc2a8956700 1 --2- 172.21.15.81:0/1404987086 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fc29002ff60 0x7fc290032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:34:05.720 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:05.713+0000 7fc2a8956700 1 --2- 172.21.15.81:0/1404987086 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fc29002ff60 0x7fc290032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fc2940096a0 tx=0x7fc294006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:34:05.721 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:05.717+0000 7fc299ffb700 1 -- 172.21.15.81:0/1404987086 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc2a0014420 con 0x7fc2a40f51a0 2023-11-15T06:34:05.926 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:05.923+0000 7fc2aa159700 1 -- 172.21.15.81:0/1404987086 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc2a4106de0 con 0x7fc2a40f51a0 2023-11-15T06:34:05.926 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:05.923+0000 7fc299ffb700 1 -- 172.21.15.81:0/1404987086 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc2a0010230 con 0x7fc2a40f51a0 2023-11-15T06:34:05.927 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:34:05.927 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:34:05.928 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:05.925+0000 7fc2aa159700 1 -- 172.21.15.81:0/1404987086 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fc29002ff60 msgr2=0x7fc290032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:05.928 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:05.925+0000 7fc2aa159700 1 --2- 172.21.15.81:0/1404987086 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fc29002ff60 0x7fc290032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fc2940096a0 tx=0x7fc294006b40).stop 2023-11-15T06:34:05.928 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:05.925+0000 7fc2aa159700 1 -- 172.21.15.81:0/1404987086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2a40f51a0 msgr2=0x7fc2a41053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:05.929 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:05.925+0000 7fc2aa159700 1 --2- 172.21.15.81:0/1404987086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2a40f51a0 0x7fc2a41053b0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7fc2a0013510 tx=0x7fc2a000b040).stop 2023-11-15T06:34:05.929 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:05.925+0000 7fc2aa159700 1 -- 172.21.15.81:0/1404987086 shutdown_connections 2023-11-15T06:34:05.929 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:05.925+0000 7fc2aa159700 1 --2- 172.21.15.81:0/1404987086 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fc29002ff60 0x7fc290032420 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:05.929 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:05.925+0000 7fc2aa159700 1 --2- 172.21.15.81:0/1404987086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2a40f51a0 0x7fc2a41053b0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:05.929 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:05.925+0000 7fc2aa159700 1 -- 172.21.15.81:0/1404987086 >> 172.21.15.81:0/1404987086 conn(0x7fc2a40f09a0 msgr2=0x7fc2a40f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:34:05.930 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:05.925+0000 7fc2aa159700 1 -- 172.21.15.81:0/1404987086 shutdown_connections 2023-11-15T06:34:05.930 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:05.925+0000 7fc2aa159700 1 -- 172.21.15.81:0/1404987086 wait complete. 2023-11-15T06:34:05.930 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:34:07.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:06 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:34:05.857026+0000 mgr.smithi062.rwurtk (mgr.14164) 254 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:34:07.419 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:06 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:34:05.925194+0000 mon.smithi062 (mon.0) 304 : audit [DBG] from='client.? 172.21.15.81:0/1404987086' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:34:07.466 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:34:07.467 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:34:08.739 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:08.735+0000 7f342bb1d700 1 -- 172.21.15.81:0/2319918716 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34240f5410 msgr2=0x7f34240f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:08.739 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:08.735+0000 7f342bb1d700 1 --2- 172.21.15.81:0/2319918716 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34240f5410 0x7f34240f5830 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f3414009fb0 tx=0x7f3414009b30).stop 2023-11-15T06:34:08.740 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:08.735+0000 7f342bb1d700 1 -- 172.21.15.81:0/2319918716 shutdown_connections 2023-11-15T06:34:08.740 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:08.735+0000 7f342bb1d700 1 --2- 172.21.15.81:0/2319918716 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34240f5410 0x7f34240f5830 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:08.740 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:08.735+0000 7f342bb1d700 1 -- 172.21.15.81:0/2319918716 >> 172.21.15.81:0/2319918716 conn(0x7f34240f0b70 msgr2=0x7f34240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:34:08.740 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:08.735+0000 7f342bb1d700 1 -- 172.21.15.81:0/2319918716 shutdown_connections 2023-11-15T06:34:08.741 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:08.735+0000 7f342bb1d700 1 -- 172.21.15.81:0/2319918716 wait complete. 2023-11-15T06:34:08.741 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:08.736+0000 7f342bb1d700 1 Processor -- start 2023-11-15T06:34:08.741 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:08.736+0000 7f342bb1d700 1 -- start start 2023-11-15T06:34:08.741 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:08.736+0000 7f342bb1d700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34240f5410 0x7f3424109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:34:08.741 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:08.736+0000 7f342bb1d700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3424060220 con 0x7f34240f5410 2023-11-15T06:34:08.742 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:08.736+0000 7f34298b9700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34240f5410 0x7f3424109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:34:08.742 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:08.736+0000 7f34298b9700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34240f5410 0x7f3424109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:54862/0 (socket says 172.21.15.81:54862) 2023-11-15T06:34:08.742 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:08.736+0000 7f34298b9700 1 -- 172.21.15.81:0/3393519230 learned_addr learned my addr 172.21.15.81:0/3393519230 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:34:08.742 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:08.736+0000 7f34298b9700 1 -- 172.21.15.81:0/3393519230 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3414005040 con 0x7f34240f5410 2023-11-15T06:34:08.744 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:08.736+0000 7f34298b9700 1 --2- 172.21.15.81:0/3393519230 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34240f5410 0x7f3424109450 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f3414013510 tx=0x7f341400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:34:08.744 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:08.737+0000 7f341a7fc700 1 -- 172.21.15.81:0/3393519230 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3414009320 con 0x7f34240f5410 2023-11-15T06:34:08.744 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:08.737+0000 7f342bb1d700 1 -- 172.21.15.81:0/3393519230 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3424060420 con 0x7f34240f5410 2023-11-15T06:34:08.744 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:08.737+0000 7f341a7fc700 1 -- 172.21.15.81:0/3393519230 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3414013f00 con 0x7f34240f5410 2023-11-15T06:34:08.744 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:08.737+0000 7f341a7fc700 1 -- 172.21.15.81:0/3393519230 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3414014d90 con 0x7f34240f5410 2023-11-15T06:34:08.745 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:08.737+0000 7f342bb1d700 1 -- 172.21.15.81:0/3393519230 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3424109b00 con 0x7f34240f5410 2023-11-15T06:34:08.745 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:08.737+0000 7f341a7fc700 1 -- 172.21.15.81:0/3393519230 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f34140263a0 con 0x7f34240f5410 2023-11-15T06:34:08.745 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:08.738+0000 7f341a7fc700 1 --2- 172.21.15.81:0/3393519230 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f341002ff60 0x7f3410032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:34:08.745 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:08.738+0000 7f341a7fc700 1 -- 172.21.15.81:0/3393519230 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3414041020 con 0x7f34240f5410 2023-11-15T06:34:08.745 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:08.738+0000 7f342bb1d700 1 -- 172.21.15.81:0/3393519230 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3424059100 con 0x7f34240f5410 2023-11-15T06:34:08.746 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:08.738+0000 7f34290b8700 1 --2- 172.21.15.81:0/3393519230 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f341002ff60 0x7f3410032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:34:08.746 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:08.738+0000 7f34290b8700 1 --2- 172.21.15.81:0/3393519230 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f341002ff60 0x7f3410032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f34200096a0 tx=0x7f3420006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:34:08.746 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:08.742+0000 7f341a7fc700 1 -- 172.21.15.81:0/3393519230 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3414012920 con 0x7f34240f5410 2023-11-15T06:34:08.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:08 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:34:07.857389+0000 mgr.smithi062.rwurtk (mgr.14164) 255 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:34:08.952 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:08.949+0000 7f342bb1d700 1 -- 172.21.15.81:0/3393519230 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f342410a1e0 con 0x7f34240f5410 2023-11-15T06:34:08.952 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:08.949+0000 7f341a7fc700 1 -- 172.21.15.81:0/3393519230 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f341402b020 con 0x7f34240f5410 2023-11-15T06:34:08.953 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:34:08.953 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:34:08.954 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:08.951+0000 7f342bb1d700 1 -- 172.21.15.81:0/3393519230 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f341002ff60 msgr2=0x7f3410032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:08.954 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:08.951+0000 7f342bb1d700 1 --2- 172.21.15.81:0/3393519230 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f341002ff60 0x7f3410032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f34200096a0 tx=0x7f3420006b40).stop 2023-11-15T06:34:08.955 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:08.951+0000 7f342bb1d700 1 -- 172.21.15.81:0/3393519230 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34240f5410 msgr2=0x7f3424109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:08.955 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:08.951+0000 7f342bb1d700 1 --2- 172.21.15.81:0/3393519230 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34240f5410 0x7f3424109450 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f3414013510 tx=0x7f341400b040).stop 2023-11-15T06:34:08.955 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:08.951+0000 7f342bb1d700 1 -- 172.21.15.81:0/3393519230 shutdown_connections 2023-11-15T06:34:08.955 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:08.951+0000 7f342bb1d700 1 --2- 172.21.15.81:0/3393519230 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f341002ff60 0x7f3410032420 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:08.955 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:08.951+0000 7f342bb1d700 1 --2- 172.21.15.81:0/3393519230 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f34240f5410 0x7f3424109450 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:08.956 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:08.951+0000 7f342bb1d700 1 -- 172.21.15.81:0/3393519230 >> 172.21.15.81:0/3393519230 conn(0x7f34240f0b70 msgr2=0x7f34240f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:34:08.956 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:08.951+0000 7f342bb1d700 1 -- 172.21.15.81:0/3393519230 shutdown_connections 2023-11-15T06:34:08.956 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:08.951+0000 7f342bb1d700 1 -- 172.21.15.81:0/3393519230 wait complete. 2023-11-15T06:34:08.957 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:34:09.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:09 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:34:08.951333+0000 mon.smithi062 (mon.0) 305 : audit [DBG] from='client.? 172.21.15.81:0/3393519230' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:34:10.339 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:34:10.340 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:34:10.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:10 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:34:09.857720+0000 mgr.smithi062.rwurtk (mgr.14164) 256 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:34:11.591 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:11.588+0000 7f45edf97700 1 -- 172.21.15.81:0/1112775168 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45e80f51a0 msgr2=0x7f45e80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:11.592 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:11.588+0000 7f45edf97700 1 --2- 172.21.15.81:0/1112775168 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45e80f51a0 0x7f45e80f55c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f45dc009fb0 tx=0x7f45dc009b30).stop 2023-11-15T06:34:11.592 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:11.588+0000 7f45edf97700 1 -- 172.21.15.81:0/1112775168 shutdown_connections 2023-11-15T06:34:11.592 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:11.588+0000 7f45edf97700 1 --2- 172.21.15.81:0/1112775168 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45e80f51a0 0x7f45e80f55c0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:11.592 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:11.588+0000 7f45edf97700 1 -- 172.21.15.81:0/1112775168 >> 172.21.15.81:0/1112775168 conn(0x7f45e80f09a0 msgr2=0x7f45e80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:34:11.593 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:11.588+0000 7f45edf97700 1 -- 172.21.15.81:0/1112775168 shutdown_connections 2023-11-15T06:34:11.593 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:11.588+0000 7f45edf97700 1 -- 172.21.15.81:0/1112775168 wait complete. 2023-11-15T06:34:11.593 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:11.588+0000 7f45edf97700 1 Processor -- start 2023-11-15T06:34:11.593 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:11.588+0000 7f45edf97700 1 -- start start 2023-11-15T06:34:11.593 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:11.588+0000 7f45edf97700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45e80f51a0 0x7f45e81064c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:34:11.594 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:11.588+0000 7f45edf97700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45e81069b0 con 0x7f45e80f51a0 2023-11-15T06:34:11.595 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:11.589+0000 7f45ecf95700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45e80f51a0 0x7f45e81064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:34:11.595 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:11.589+0000 7f45ecf95700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45e80f51a0 0x7f45e81064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:54880/0 (socket says 172.21.15.81:54880) 2023-11-15T06:34:11.596 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:11.589+0000 7f45ecf95700 1 -- 172.21.15.81:0/76479814 learned_addr learned my addr 172.21.15.81:0/76479814 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:34:11.598 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:11.589+0000 7f45ecf95700 1 -- 172.21.15.81:0/76479814 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45dc005040 con 0x7f45e80f51a0 2023-11-15T06:34:11.598 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:11.589+0000 7f45ecf95700 1 --2- 172.21.15.81:0/76479814 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45e80f51a0 0x7f45e81064c0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f45dc013510 tx=0x7f45dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:34:11.599 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:11.589+0000 7f45e57fa700 1 -- 172.21.15.81:0/76479814 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45dc009320 con 0x7f45e80f51a0 2023-11-15T06:34:11.599 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:11.589+0000 7f45edf97700 1 -- 172.21.15.81:0/76479814 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45e8106bb0 con 0x7f45e80f51a0 2023-11-15T06:34:11.599 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:11.589+0000 7f45e57fa700 1 -- 172.21.15.81:0/76479814 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f45dc013f00 con 0x7f45e80f51a0 2023-11-15T06:34:11.599 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:11.589+0000 7f45e57fa700 1 -- 172.21.15.81:0/76479814 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45dc014dc0 con 0x7f45e80f51a0 2023-11-15T06:34:11.600 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:11.589+0000 7f45edf97700 1 -- 172.21.15.81:0/76479814 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45e8107040 con 0x7f45e80f51a0 2023-11-15T06:34:11.600 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:11.590+0000 7f45e57fa700 1 -- 172.21.15.81:0/76479814 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f45dc0095f0 con 0x7f45e80f51a0 2023-11-15T06:34:11.600 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:11.590+0000 7f45edf97700 1 -- 172.21.15.81:0/76479814 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45e8045ad0 con 0x7f45e80f51a0 2023-11-15T06:34:11.600 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:11.591+0000 7f45e57fa700 1 --2- 172.21.15.81:0/76479814 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f45d802ff60 0x7f45d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:34:11.600 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:11.591+0000 7f45e57fa700 1 -- 172.21.15.81:0/76479814 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f45dc041040 con 0x7f45e80f51a0 2023-11-15T06:34:11.601 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:11.591+0000 7f45e7fff700 1 --2- 172.21.15.81:0/76479814 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f45d802ff60 0x7f45d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:34:11.601 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:11.591+0000 7f45e7fff700 1 --2- 172.21.15.81:0/76479814 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f45d802ff60 0x7f45d8032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f45d40096a0 tx=0x7f45d4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:34:11.601 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:11.595+0000 7f45e57fa700 1 -- 172.21.15.81:0/76479814 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f45dc014420 con 0x7f45e80f51a0 2023-11-15T06:34:11.804 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:11.801+0000 7f45edf97700 1 -- 172.21.15.81:0/76479814 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f45e8107ef0 con 0x7f45e80f51a0 2023-11-15T06:34:11.806 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:11.801+0000 7f45e57fa700 1 -- 172.21.15.81:0/76479814 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f45dc014420 con 0x7f45e80f51a0 2023-11-15T06:34:11.806 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:34:11.806 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:34:11.807 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:11.803+0000 7f45edf97700 1 -- 172.21.15.81:0/76479814 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f45d802ff60 msgr2=0x7f45d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:11.807 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:11.803+0000 7f45edf97700 1 --2- 172.21.15.81:0/76479814 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f45d802ff60 0x7f45d8032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f45d40096a0 tx=0x7f45d4006b40).stop 2023-11-15T06:34:11.808 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:11.803+0000 7f45edf97700 1 -- 172.21.15.81:0/76479814 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45e80f51a0 msgr2=0x7f45e81064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:11.808 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:11.803+0000 7f45edf97700 1 --2- 172.21.15.81:0/76479814 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45e80f51a0 0x7f45e81064c0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f45dc013510 tx=0x7f45dc00b040).stop 2023-11-15T06:34:11.808 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:11.803+0000 7f45edf97700 1 -- 172.21.15.81:0/76479814 shutdown_connections 2023-11-15T06:34:11.808 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:11.803+0000 7f45edf97700 1 --2- 172.21.15.81:0/76479814 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f45d802ff60 0x7f45d8032420 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:11.808 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:11.803+0000 7f45edf97700 1 --2- 172.21.15.81:0/76479814 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45e80f51a0 0x7f45e81064c0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:11.809 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:11.803+0000 7f45edf97700 1 -- 172.21.15.81:0/76479814 >> 172.21.15.81:0/76479814 conn(0x7f45e80f09a0 msgr2=0x7f45e80f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:34:11.809 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:11.803+0000 7f45edf97700 1 -- 172.21.15.81:0/76479814 shutdown_connections 2023-11-15T06:34:11.809 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:11.803+0000 7f45edf97700 1 -- 172.21.15.81:0/76479814 wait complete. 2023-11-15T06:34:11.809 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:34:12.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:11 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:34:11.803248+0000 mon.smithi062 (mon.0) 306 : audit [DBG] from='client.? 172.21.15.81:0/76479814' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:34:13.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:12 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:34:11.857927+0000 mgr.smithi062.rwurtk (mgr.14164) 257 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:34:13.233 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:34:13.233 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:34:14.483 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:14.479+0000 7f258691a700 1 -- 172.21.15.81:0/1734226553 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f25800601e0 msgr2=0x7f25800f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:14.484 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:14.479+0000 7f258691a700 1 --2- 172.21.15.81:0/1734226553 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f25800601e0 0x7f25800f9190 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f2570009fb0 tx=0x7f2570009b30).stop 2023-11-15T06:34:14.484 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:14.479+0000 7f258691a700 1 -- 172.21.15.81:0/1734226553 shutdown_connections 2023-11-15T06:34:14.484 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:14.479+0000 7f258691a700 1 --2- 172.21.15.81:0/1734226553 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f25800601e0 0x7f25800f9190 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:14.484 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:14.479+0000 7f258691a700 1 -- 172.21.15.81:0/1734226553 >> 172.21.15.81:0/1734226553 conn(0x7f25800f0b70 msgr2=0x7f25800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:34:14.484 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:14.479+0000 7f258691a700 1 -- 172.21.15.81:0/1734226553 shutdown_connections 2023-11-15T06:34:14.485 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:14.479+0000 7f258691a700 1 -- 172.21.15.81:0/1734226553 wait complete. 2023-11-15T06:34:14.485 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:14.480+0000 7f258691a700 1 Processor -- start 2023-11-15T06:34:14.485 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:14.480+0000 7f258691a700 1 -- start start 2023-11-15T06:34:14.485 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:14.480+0000 7f258691a700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f25800601e0 0x7f25800f6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:34:14.485 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:14.480+0000 7f258691a700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25800f5090 con 0x7f25800601e0 2023-11-15T06:34:14.487 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:14.481+0000 7f257ffff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f25800601e0 0x7f25800f6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:34:14.487 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:14.481+0000 7f257ffff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f25800601e0 0x7f25800f6c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:58914/0 (socket says 172.21.15.81:58914) 2023-11-15T06:34:14.487 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:14.481+0000 7f257ffff700 1 -- 172.21.15.81:0/2157506041 learned_addr learned my addr 172.21.15.81:0/2157506041 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:34:14.488 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:14.481+0000 7f257ffff700 1 -- 172.21.15.81:0/2157506041 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2570005040 con 0x7f25800601e0 2023-11-15T06:34:14.488 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:14.481+0000 7f257ffff700 1 --2- 172.21.15.81:0/2157506041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f25800601e0 0x7f25800f6c10 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f2570004dd0 tx=0x7f2570009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:34:14.488 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:14.481+0000 7f257cff9700 1 -- 172.21.15.81:0/2157506041 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2570018070 con 0x7f25800601e0 2023-11-15T06:34:14.488 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:14.481+0000 7f258691a700 1 -- 172.21.15.81:0/2157506041 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25800f5230 con 0x7f25800601e0 2023-11-15T06:34:14.488 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:14.481+0000 7f257cff9700 1 -- 172.21.15.81:0/2157506041 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2570013410 con 0x7f25800601e0 2023-11-15T06:34:14.489 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:14.481+0000 7f257cff9700 1 -- 172.21.15.81:0/2157506041 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f257001c3c0 con 0x7f25800601e0 2023-11-15T06:34:14.489 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:14.481+0000 7f258691a700 1 -- 172.21.15.81:0/2157506041 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25800f56c0 con 0x7f25800601e0 2023-11-15T06:34:14.489 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:14.482+0000 7f257cff9700 1 -- 172.21.15.81:0/2157506041 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f257001d420 con 0x7f25800601e0 2023-11-15T06:34:14.489 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:14.482+0000 7f258691a700 1 -- 172.21.15.81:0/2157506041 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2580059100 con 0x7f25800601e0 2023-11-15T06:34:14.489 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:14.482+0000 7f257cff9700 1 --2- 172.21.15.81:0/2157506041 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f256802ff60 0x7f2568032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:34:14.490 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:14.482+0000 7f257cff9700 1 -- 172.21.15.81:0/2157506041 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2570041310 con 0x7f25800601e0 2023-11-15T06:34:14.490 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:14.482+0000 7f257f7fe700 1 --2- 172.21.15.81:0/2157506041 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f256802ff60 0x7f2568032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:34:14.490 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:14.483+0000 7f257f7fe700 1 --2- 172.21.15.81:0/2157506041 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f256802ff60 0x7f2568032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f25740096a0 tx=0x7f2574006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:34:14.490 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:14.487+0000 7f257cff9700 1 -- 172.21.15.81:0/2157506041 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f257001d820 con 0x7f25800601e0 2023-11-15T06:34:14.694 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:14.691+0000 7f258691a700 1 -- 172.21.15.81:0/2157506041 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2580045ad0 con 0x7f25800601e0 2023-11-15T06:34:14.695 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:14.692+0000 7f257cff9700 1 -- 172.21.15.81:0/2157506041 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f257001b370 con 0x7f25800601e0 2023-11-15T06:34:14.696 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:34:14.696 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:34:14.697 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:14.693+0000 7f258691a700 1 -- 172.21.15.81:0/2157506041 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f256802ff60 msgr2=0x7f2568032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:14.697 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:14.693+0000 7f258691a700 1 --2- 172.21.15.81:0/2157506041 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f256802ff60 0x7f2568032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f25740096a0 tx=0x7f2574006b40).stop 2023-11-15T06:34:14.697 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:14.693+0000 7f258691a700 1 -- 172.21.15.81:0/2157506041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f25800601e0 msgr2=0x7f25800f6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:14.698 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:14.693+0000 7f258691a700 1 --2- 172.21.15.81:0/2157506041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f25800601e0 0x7f25800f6c10 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f2570004dd0 tx=0x7f2570009b30).stop 2023-11-15T06:34:14.698 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:14.694+0000 7f258691a700 1 -- 172.21.15.81:0/2157506041 shutdown_connections 2023-11-15T06:34:14.698 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:14.694+0000 7f258691a700 1 --2- 172.21.15.81:0/2157506041 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f256802ff60 0x7f2568032420 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:14.698 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:14.694+0000 7f258691a700 1 --2- 172.21.15.81:0/2157506041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f25800601e0 0x7f25800f6c10 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:14.698 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:14.694+0000 7f258691a700 1 -- 172.21.15.81:0/2157506041 >> 172.21.15.81:0/2157506041 conn(0x7f25800f0b70 msgr2=0x7f25800f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:34:14.699 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:14.694+0000 7f258691a700 1 -- 172.21.15.81:0/2157506041 shutdown_connections 2023-11-15T06:34:14.699 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:14.694+0000 7f258691a700 1 -- 172.21.15.81:0/2157506041 wait complete. 2023-11-15T06:34:14.699 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:34:15.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:14 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:34:13.858271+0000 mgr.smithi062.rwurtk (mgr.14164) 258 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:34:15.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:14 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:34:14.693959+0000 mon.smithi062 (mon.0) 307 : audit [DBG] from='client.? 172.21.15.81:0/2157506041' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:34:16.079 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:34:16.079 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:34:17.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:16 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:34:15.858654+0000 mgr.smithi062.rwurtk (mgr.14164) 259 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:34:17.272 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:17.269+0000 7fbf16d21700 1 -- 172.21.15.81:0/4027954337 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf100601e0 msgr2=0x7fbf100f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:17.272 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:17.269+0000 7fbf16d21700 1 --2- 172.21.15.81:0/4027954337 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf100601e0 0x7fbf100f8f20 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7fbf0c009fb0 tx=0x7fbf0c009b30).stop 2023-11-15T06:34:17.273 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:17.269+0000 7fbf16d21700 1 -- 172.21.15.81:0/4027954337 shutdown_connections 2023-11-15T06:34:17.273 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:17.269+0000 7fbf16d21700 1 --2- 172.21.15.81:0/4027954337 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf100601e0 0x7fbf100f8f20 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:17.273 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:17.269+0000 7fbf16d21700 1 -- 172.21.15.81:0/4027954337 >> 172.21.15.81:0/4027954337 conn(0x7fbf100f09a0 msgr2=0x7fbf100f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:34:17.273 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:17.269+0000 7fbf16d21700 1 -- 172.21.15.81:0/4027954337 shutdown_connections 2023-11-15T06:34:17.273 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:17.269+0000 7fbf16d21700 1 -- 172.21.15.81:0/4027954337 wait complete. 2023-11-15T06:34:17.275 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:17.269+0000 7fbf16d21700 1 Processor -- start 2023-11-15T06:34:17.276 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:17.270+0000 7fbf16d21700 1 -- start start 2023-11-15T06:34:17.276 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:17.270+0000 7fbf16d21700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf100601e0 0x7fbf100f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:34:17.276 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:17.270+0000 7fbf16d21700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf100f4e20 con 0x7fbf100601e0 2023-11-15T06:34:17.276 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:17.270+0000 7fbf15d1f700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf100601e0 0x7fbf100f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:34:17.277 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:17.270+0000 7fbf15d1f700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf100601e0 0x7fbf100f69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:58920/0 (socket says 172.21.15.81:58920) 2023-11-15T06:34:17.277 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:17.270+0000 7fbf15d1f700 1 -- 172.21.15.81:0/1086801313 learned_addr learned my addr 172.21.15.81:0/1086801313 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:34:17.277 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:17.270+0000 7fbf15d1f700 1 -- 172.21.15.81:0/1086801313 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf0c005040 con 0x7fbf100601e0 2023-11-15T06:34:17.277 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:17.270+0000 7fbf15d1f700 1 --2- 172.21.15.81:0/1086801313 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf100601e0 0x7fbf100f69a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7fbf0c014000 tx=0x7fbf0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:34:17.277 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:17.270+0000 7fbf067fc700 1 -- 172.21.15.81:0/1086801313 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf0c0095f0 con 0x7fbf100601e0 2023-11-15T06:34:17.278 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:17.270+0000 7fbf16d21700 1 -- 172.21.15.81:0/1086801313 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf100f4fc0 con 0x7fbf100601e0 2023-11-15T06:34:17.278 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:17.270+0000 7fbf067fc700 1 -- 172.21.15.81:0/1086801313 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbf0c007e70 con 0x7fbf100601e0 2023-11-15T06:34:17.278 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:17.270+0000 7fbf067fc700 1 -- 172.21.15.81:0/1086801313 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf0c014ea0 con 0x7fbf100601e0 2023-11-15T06:34:17.278 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:17.271+0000 7fbf16d21700 1 -- 172.21.15.81:0/1086801313 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf100f5450 con 0x7fbf100601e0 2023-11-15T06:34:17.278 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:17.271+0000 7fbf067fc700 1 -- 172.21.15.81:0/1086801313 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbf0c014690 con 0x7fbf100601e0 2023-11-15T06:34:17.279 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:17.271+0000 7fbf16d21700 1 -- 172.21.15.81:0/1086801313 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf10059100 con 0x7fbf100601e0 2023-11-15T06:34:17.279 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:17.271+0000 7fbf067fc700 1 --2- 172.21.15.81:0/1086801313 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fbefc02ff60 0x7fbefc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:34:17.279 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:17.271+0000 7fbf067fc700 1 -- 172.21.15.81:0/1086801313 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbf0c0408f0 con 0x7fbf100601e0 2023-11-15T06:34:17.279 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:17.272+0000 7fbf1551e700 1 --2- 172.21.15.81:0/1086801313 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fbefc02ff60 0x7fbefc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:34:17.279 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:17.272+0000 7fbf1551e700 1 --2- 172.21.15.81:0/1086801313 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fbefc02ff60 0x7fbefc032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fbf000096a0 tx=0x7fbf00006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:34:17.280 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:17.276+0000 7fbf067fc700 1 -- 172.21.15.81:0/1086801313 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbf0c00f300 con 0x7fbf100601e0 2023-11-15T06:34:17.486 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:17.482+0000 7fbf16d21700 1 -- 172.21.15.81:0/1086801313 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbf100fbcd0 con 0x7fbf100601e0 2023-11-15T06:34:17.486 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:17.483+0000 7fbf067fc700 1 -- 172.21.15.81:0/1086801313 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbf0c00f300 con 0x7fbf100601e0 2023-11-15T06:34:17.487 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:34:17.487 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:34:17.488 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:17.484+0000 7fbf16d21700 1 -- 172.21.15.81:0/1086801313 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fbefc02ff60 msgr2=0x7fbefc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:17.488 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:17.484+0000 7fbf16d21700 1 --2- 172.21.15.81:0/1086801313 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fbefc02ff60 0x7fbefc032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fbf000096a0 tx=0x7fbf00006b40).stop 2023-11-15T06:34:17.488 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:17.484+0000 7fbf16d21700 1 -- 172.21.15.81:0/1086801313 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf100601e0 msgr2=0x7fbf100f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:17.488 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:17.484+0000 7fbf16d21700 1 --2- 172.21.15.81:0/1086801313 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf100601e0 0x7fbf100f69a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7fbf0c014000 tx=0x7fbf0c00b040).stop 2023-11-15T06:34:17.489 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:17.484+0000 7fbf16d21700 1 -- 172.21.15.81:0/1086801313 shutdown_connections 2023-11-15T06:34:17.489 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:17.484+0000 7fbf16d21700 1 --2- 172.21.15.81:0/1086801313 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fbefc02ff60 0x7fbefc032420 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:17.489 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:17.484+0000 7fbf16d21700 1 --2- 172.21.15.81:0/1086801313 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf100601e0 0x7fbf100f69a0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:17.489 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:17.484+0000 7fbf16d21700 1 -- 172.21.15.81:0/1086801313 >> 172.21.15.81:0/1086801313 conn(0x7fbf100f09a0 msgr2=0x7fbf100f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:34:17.489 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:17.484+0000 7fbf16d21700 1 -- 172.21.15.81:0/1086801313 shutdown_connections 2023-11-15T06:34:17.490 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:17.484+0000 7fbf16d21700 1 -- 172.21.15.81:0/1086801313 wait complete. 2023-11-15T06:34:17.490 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:34:18.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:17 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:34:17.484694+0000 mon.smithi062 (mon.0) 308 : audit [DBG] from='client.? 172.21.15.81:0/1086801313' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:34:18.878 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:34:18.879 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:34:19.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:18 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:34:17.859040+0000 mgr.smithi062.rwurtk (mgr.14164) 260 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:34:20.114 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:20.107+0000 7fbd19f81700 1 -- 172.21.15.81:0/2268762617 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd140f5410 msgr2=0x7fbd140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:20.115 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:20.107+0000 7fbd19f81700 1 --2- 172.21.15.81:0/2268762617 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd140f5410 0x7fbd140f5830 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7fbd08009fb0 tx=0x7fbd08009b30).stop 2023-11-15T06:34:20.115 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:20.107+0000 7fbd19f81700 1 -- 172.21.15.81:0/2268762617 shutdown_connections 2023-11-15T06:34:20.115 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:20.107+0000 7fbd19f81700 1 --2- 172.21.15.81:0/2268762617 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd140f5410 0x7fbd140f5830 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:20.116 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:20.107+0000 7fbd19f81700 1 -- 172.21.15.81:0/2268762617 >> 172.21.15.81:0/2268762617 conn(0x7fbd140f0b70 msgr2=0x7fbd140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:34:20.117 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:20.107+0000 7fbd19f81700 1 -- 172.21.15.81:0/2268762617 shutdown_connections 2023-11-15T06:34:20.117 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:20.107+0000 7fbd19f81700 1 -- 172.21.15.81:0/2268762617 wait complete. 2023-11-15T06:34:20.117 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:20.107+0000 7fbd19f81700 1 Processor -- start 2023-11-15T06:34:20.117 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:20.108+0000 7fbd19f81700 1 -- start start 2023-11-15T06:34:20.118 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:20.108+0000 7fbd19f81700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd140f5410 0x7fbd14106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:34:20.118 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:20.108+0000 7fbd19f81700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd14106c00 con 0x7fbd140f5410 2023-11-15T06:34:20.118 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:20.108+0000 7fbd137fe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd140f5410 0x7fbd14106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:34:20.118 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:20.108+0000 7fbd137fe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd140f5410 0x7fbd14106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:58950/0 (socket says 172.21.15.81:58950) 2023-11-15T06:34:20.118 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:20.108+0000 7fbd137fe700 1 -- 172.21.15.81:0/1864556845 learned_addr learned my addr 172.21.15.81:0/1864556845 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:34:20.119 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:20.108+0000 7fbd137fe700 1 -- 172.21.15.81:0/1864556845 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd08005040 con 0x7fbd140f5410 2023-11-15T06:34:20.119 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:20.109+0000 7fbd137fe700 1 --2- 172.21.15.81:0/1864556845 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd140f5410 0x7fbd14106710 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7fbd08014000 tx=0x7fbd0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:34:20.119 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:20.109+0000 7fbcfbfff700 1 -- 172.21.15.81:0/1864556845 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd080095f0 con 0x7fbd140f5410 2023-11-15T06:34:20.119 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:20.109+0000 7fbcfbfff700 1 -- 172.21.15.81:0/1864556845 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbd08007e70 con 0x7fbd140f5410 2023-11-15T06:34:20.119 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:20.109+0000 7fbd19f81700 1 -- 172.21.15.81:0/1864556845 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd14106e00 con 0x7fbd140f5410 2023-11-15T06:34:20.120 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:20.109+0000 7fbcfbfff700 1 -- 172.21.15.81:0/1864556845 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd08014ea0 con 0x7fbd140f5410 2023-11-15T06:34:20.120 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:20.109+0000 7fbd19f81700 1 -- 172.21.15.81:0/1864556845 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd14107290 con 0x7fbd140f5410 2023-11-15T06:34:20.120 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:20.110+0000 7fbcfbfff700 1 -- 172.21.15.81:0/1864556845 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbd08025070 con 0x7fbd140f5410 2023-11-15T06:34:20.120 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:20.110+0000 7fbd19f81700 1 -- 172.21.15.81:0/1864556845 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd14045ad0 con 0x7fbd140f5410 2023-11-15T06:34:20.120 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:20.110+0000 7fbcfbfff700 1 --2- 172.21.15.81:0/1864556845 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fbcfc02ff60 0x7fbcfc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:34:20.121 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:20.110+0000 7fbcfbfff700 1 -- 172.21.15.81:0/1864556845 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbd08040c40 con 0x7fbd140f5410 2023-11-15T06:34:20.121 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:20.110+0000 7fbd12ffd700 1 --2- 172.21.15.81:0/1864556845 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fbcfc02ff60 0x7fbcfc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:34:20.121 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:20.110+0000 7fbd12ffd700 1 --2- 172.21.15.81:0/1864556845 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fbcfc02ff60 0x7fbcfc032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fbd040096a0 tx=0x7fbd04006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:34:20.122 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:20.115+0000 7fbcfbfff700 1 -- 172.21.15.81:0/1864556845 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbd08025950 con 0x7fbd140f5410 2023-11-15T06:34:20.322 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:20.318+0000 7fbd19f81700 1 -- 172.21.15.81:0/1864556845 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbd140fae60 con 0x7fbd140f5410 2023-11-15T06:34:20.322 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:20.319+0000 7fbcfbfff700 1 -- 172.21.15.81:0/1864556845 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbd08025950 con 0x7fbd140f5410 2023-11-15T06:34:20.323 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:34:20.323 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:34:20.324 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:20.321+0000 7fbd19f81700 1 -- 172.21.15.81:0/1864556845 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fbcfc02ff60 msgr2=0x7fbcfc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:20.324 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:20.321+0000 7fbd19f81700 1 --2- 172.21.15.81:0/1864556845 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fbcfc02ff60 0x7fbcfc032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fbd040096a0 tx=0x7fbd04006b40).stop 2023-11-15T06:34:20.325 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:20.321+0000 7fbd19f81700 1 -- 172.21.15.81:0/1864556845 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd140f5410 msgr2=0x7fbd14106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:20.325 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:20.321+0000 7fbd19f81700 1 --2- 172.21.15.81:0/1864556845 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd140f5410 0x7fbd14106710 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7fbd08014000 tx=0x7fbd0800b040).stop 2023-11-15T06:34:20.325 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:20.321+0000 7fbd19f81700 1 -- 172.21.15.81:0/1864556845 shutdown_connections 2023-11-15T06:34:20.325 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:20.321+0000 7fbd19f81700 1 --2- 172.21.15.81:0/1864556845 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fbcfc02ff60 0x7fbcfc032420 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:20.325 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:20.321+0000 7fbd19f81700 1 --2- 172.21.15.81:0/1864556845 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd140f5410 0x7fbd14106710 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:20.326 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:20.321+0000 7fbd19f81700 1 -- 172.21.15.81:0/1864556845 >> 172.21.15.81:0/1864556845 conn(0x7fbd140f0b70 msgr2=0x7fbd1405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:34:20.326 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:20.321+0000 7fbd19f81700 1 -- 172.21.15.81:0/1864556845 shutdown_connections 2023-11-15T06:34:20.326 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:20.321+0000 7fbd19f81700 1 -- 172.21.15.81:0/1864556845 wait complete. 2023-11-15T06:34:20.327 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:34:21.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:20 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:34:19.859427+0000 mgr.smithi062.rwurtk (mgr.14164) 261 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:34:21.419 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:20 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:34:20.321076+0000 mon.smithi062 (mon.0) 309 : audit [DBG] from='client.? 172.21.15.81:0/1864556845' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:34:21.676 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:34:21.677 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:34:22.852 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:22.849+0000 7f4f3bc36700 1 -- 172.21.15.81:0/3030302582 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f340f5410 msgr2=0x7f4f340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:22.852 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:22.849+0000 7f4f3bc36700 1 --2- 172.21.15.81:0/3030302582 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f340f5410 0x7f4f340f5830 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f4f24009fb0 tx=0x7f4f24009b30).stop 2023-11-15T06:34:22.852 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:22.849+0000 7f4f3bc36700 1 -- 172.21.15.81:0/3030302582 shutdown_connections 2023-11-15T06:34:22.852 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:22.849+0000 7f4f3bc36700 1 --2- 172.21.15.81:0/3030302582 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f340f5410 0x7f4f340f5830 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:22.853 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:22.849+0000 7f4f3bc36700 1 -- 172.21.15.81:0/3030302582 >> 172.21.15.81:0/3030302582 conn(0x7f4f340f0b70 msgr2=0x7f4f340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:34:22.853 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:22.849+0000 7f4f3bc36700 1 -- 172.21.15.81:0/3030302582 shutdown_connections 2023-11-15T06:34:22.853 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:22.849+0000 7f4f3bc36700 1 -- 172.21.15.81:0/3030302582 wait complete. 2023-11-15T06:34:22.853 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:22.850+0000 7f4f3bc36700 1 Processor -- start 2023-11-15T06:34:22.855 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:22.850+0000 7f4f3bc36700 1 -- start start 2023-11-15T06:34:22.855 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:22.850+0000 7f4f3bc36700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f340f5410 0x7f4f34105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:34:22.855 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:22.850+0000 7f4f3bc36700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f34105b20 con 0x7f4f340f5410 2023-11-15T06:34:22.856 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:22.850+0000 7f4f399d2700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f340f5410 0x7f4f34105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:34:22.856 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:22.850+0000 7f4f399d2700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f340f5410 0x7f4f34105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:51744/0 (socket says 172.21.15.81:51744) 2023-11-15T06:34:22.856 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:22.850+0000 7f4f399d2700 1 -- 172.21.15.81:0/3192172741 learned_addr learned my addr 172.21.15.81:0/3192172741 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:34:22.857 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:22.851+0000 7f4f399d2700 1 -- 172.21.15.81:0/3192172741 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f24005040 con 0x7f4f340f5410 2023-11-15T06:34:22.857 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:22.851+0000 7f4f399d2700 1 --2- 172.21.15.81:0/3192172741 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f340f5410 0x7f4f34105630 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f4f24013510 tx=0x7f4f2400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:34:22.857 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:22.851+0000 7f4f2a7fc700 1 -- 172.21.15.81:0/3192172741 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f24009320 con 0x7f4f340f5410 2023-11-15T06:34:22.857 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:22.851+0000 7f4f3bc36700 1 -- 172.21.15.81:0/3192172741 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f34105d20 con 0x7f4f340f5410 2023-11-15T06:34:22.857 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:22.851+0000 7f4f2a7fc700 1 -- 172.21.15.81:0/3192172741 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4f24013f00 con 0x7f4f340f5410 2023-11-15T06:34:22.857 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:22.851+0000 7f4f2a7fc700 1 -- 172.21.15.81:0/3192172741 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f24014d90 con 0x7f4f340f5410 2023-11-15T06:34:22.858 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:22.851+0000 7f4f3bc36700 1 -- 172.21.15.81:0/3192172741 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f341061b0 con 0x7f4f340f5410 2023-11-15T06:34:22.858 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:22.852+0000 7f4f2a7fc700 1 -- 172.21.15.81:0/3192172741 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4f240263a0 con 0x7f4f340f5410 2023-11-15T06:34:22.858 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:22.852+0000 7f4f2a7fc700 1 --2- 172.21.15.81:0/3192172741 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f4f2002ff60 0x7f4f20032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:34:22.858 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:22.852+0000 7f4f2a7fc700 1 -- 172.21.15.81:0/3192172741 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4f24041010 con 0x7f4f340f5410 2023-11-15T06:34:22.858 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:22.852+0000 7f4f3bc36700 1 -- 172.21.15.81:0/3192172741 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f34045ad0 con 0x7f4f340f5410 2023-11-15T06:34:22.859 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:22.852+0000 7f4f391d1700 1 --2- 172.21.15.81:0/3192172741 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f4f2002ff60 0x7f4f20032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:34:22.859 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:22.852+0000 7f4f391d1700 1 --2- 172.21.15.81:0/3192172741 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f4f2002ff60 0x7f4f20032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f4f300096a0 tx=0x7f4f30006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:34:22.859 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:22.857+0000 7f4f2a7fc700 1 -- 172.21.15.81:0/3192172741 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4f24012920 con 0x7f4f340f5410 2023-11-15T06:34:23.066 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:23.064+0000 7f4f3bc36700 1 -- 172.21.15.81:0/3192172741 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4f34106ed0 con 0x7f4f340f5410 2023-11-15T06:34:23.067 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:23.065+0000 7f4f2a7fc700 1 -- 172.21.15.81:0/3192172741 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4f24012330 con 0x7f4f340f5410 2023-11-15T06:34:23.067 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:34:23.067 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:34:23.068 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:23.066+0000 7f4f3bc36700 1 -- 172.21.15.81:0/3192172741 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f4f2002ff60 msgr2=0x7f4f20032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:23.069 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:23.066+0000 7f4f3bc36700 1 --2- 172.21.15.81:0/3192172741 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f4f2002ff60 0x7f4f20032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f4f300096a0 tx=0x7f4f30006b40).stop 2023-11-15T06:34:23.069 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:23.066+0000 7f4f3bc36700 1 -- 172.21.15.81:0/3192172741 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f340f5410 msgr2=0x7f4f34105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:23.069 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:23.066+0000 7f4f3bc36700 1 --2- 172.21.15.81:0/3192172741 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f340f5410 0x7f4f34105630 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f4f24013510 tx=0x7f4f2400b040).stop 2023-11-15T06:34:23.069 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:23.066+0000 7f4f3bc36700 1 -- 172.21.15.81:0/3192172741 shutdown_connections 2023-11-15T06:34:23.070 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:23.066+0000 7f4f3bc36700 1 --2- 172.21.15.81:0/3192172741 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f4f2002ff60 0x7f4f20032420 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:23.070 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:23.066+0000 7f4f3bc36700 1 --2- 172.21.15.81:0/3192172741 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f340f5410 0x7f4f34105630 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:23.070 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:23.066+0000 7f4f3bc36700 1 -- 172.21.15.81:0/3192172741 >> 172.21.15.81:0/3192172741 conn(0x7f4f340f0b70 msgr2=0x7f4f340f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:34:23.070 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:23.066+0000 7f4f3bc36700 1 -- 172.21.15.81:0/3192172741 shutdown_connections 2023-11-15T06:34:23.071 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:23.066+0000 7f4f3bc36700 1 -- 172.21.15.81:0/3192172741 wait complete. 2023-11-15T06:34:23.071 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:34:23.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:22 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:34:21.859747+0000 mgr.smithi062.rwurtk (mgr.14164) 262 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:34:24.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:23 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:34:23.065500+0000 mon.smithi062 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.81:0/3192172741' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:34:24.486 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:34:24.487 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:34:25.422 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:24 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:34:23.860101+0000 mgr.smithi062.rwurtk (mgr.14164) 263 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:34:25.754 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:25.751+0000 7f3913269700 1 -- 172.21.15.81:0/770110119 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f390c0f6510 msgr2=0x7f390c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:25.754 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:25.751+0000 7f3913269700 1 --2- 172.21.15.81:0/770110119 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f390c0f6510 0x7f390c0f6930 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f38fc009fb0 tx=0x7f38fc009b30).stop 2023-11-15T06:34:25.755 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:25.751+0000 7f3913269700 1 -- 172.21.15.81:0/770110119 shutdown_connections 2023-11-15T06:34:25.755 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:25.751+0000 7f3913269700 1 --2- 172.21.15.81:0/770110119 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f390c0f6510 0x7f390c0f6930 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:25.755 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:25.751+0000 7f3913269700 1 -- 172.21.15.81:0/770110119 >> 172.21.15.81:0/770110119 conn(0x7f390c0f1c70 msgr2=0x7f390c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:34:25.755 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:25.751+0000 7f3913269700 1 -- 172.21.15.81:0/770110119 shutdown_connections 2023-11-15T06:34:25.756 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:25.751+0000 7f3913269700 1 -- 172.21.15.81:0/770110119 wait complete. 2023-11-15T06:34:25.756 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:25.752+0000 7f3913269700 1 Processor -- start 2023-11-15T06:34:25.756 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:25.752+0000 7f3913269700 1 -- start start 2023-11-15T06:34:25.756 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:25.752+0000 7f3913269700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f390c0f6510 0x7f390c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:34:25.756 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:25.752+0000 7f3913269700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f390c106c20 con 0x7f390c0f6510 2023-11-15T06:34:25.756 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:25.752+0000 7f3911005700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f390c0f6510 0x7f390c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:34:25.757 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:25.752+0000 7f3911005700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f390c0f6510 0x7f390c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:51762/0 (socket says 172.21.15.81:51762) 2023-11-15T06:34:25.757 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:25.752+0000 7f3911005700 1 -- 172.21.15.81:0/2799058384 learned_addr learned my addr 172.21.15.81:0/2799058384 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:34:25.758 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:25.752+0000 7f3911005700 1 -- 172.21.15.81:0/2799058384 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38fc005040 con 0x7f390c0f6510 2023-11-15T06:34:25.758 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:25.753+0000 7f3911005700 1 --2- 172.21.15.81:0/2799058384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f390c0f6510 0x7f390c106730 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f38fc013510 tx=0x7f38fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:34:25.759 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:25.753+0000 7f3901ffb700 1 -- 172.21.15.81:0/2799058384 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38fc009320 con 0x7f390c0f6510 2023-11-15T06:34:25.759 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:25.753+0000 7f3901ffb700 1 -- 172.21.15.81:0/2799058384 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f38fc013f00 con 0x7f390c0f6510 2023-11-15T06:34:25.759 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:25.753+0000 7f3913269700 1 -- 172.21.15.81:0/2799058384 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f390c106e20 con 0x7f390c0f6510 2023-11-15T06:34:25.759 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:25.753+0000 7f3901ffb700 1 -- 172.21.15.81:0/2799058384 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38fc014d90 con 0x7f390c0f6510 2023-11-15T06:34:25.759 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:25.753+0000 7f3913269700 1 -- 172.21.15.81:0/2799058384 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f390c1072b0 con 0x7f390c0f6510 2023-11-15T06:34:25.760 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:25.754+0000 7f3901ffb700 1 -- 172.21.15.81:0/2799058384 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f38fc0095f0 con 0x7f390c0f6510 2023-11-15T06:34:25.760 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:25.754+0000 7f3913269700 1 -- 172.21.15.81:0/2799058384 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f390c045ad0 con 0x7f390c0f6510 2023-11-15T06:34:25.760 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:25.754+0000 7f3901ffb700 1 --2- 172.21.15.81:0/2799058384 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f38f802ff60 0x7f38f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:34:25.760 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:25.754+0000 7f3901ffb700 1 -- 172.21.15.81:0/2799058384 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f38fc040fc0 con 0x7f390c0f6510 2023-11-15T06:34:25.760 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:25.754+0000 7f3910804700 1 --2- 172.21.15.81:0/2799058384 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f38f802ff60 0x7f38f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:34:25.761 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:25.755+0000 7f3910804700 1 --2- 172.21.15.81:0/2799058384 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f38f802ff60 0x7f38f8032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f39080096a0 tx=0x7f3908006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:34:25.761 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:25.759+0000 7f3901ffb700 1 -- 172.21.15.81:0/2799058384 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f38fc014420 con 0x7f390c0f6510 2023-11-15T06:34:25.968 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:25.964+0000 7f3913269700 1 -- 172.21.15.81:0/2799058384 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f390c0fbf60 con 0x7f390c0f6510 2023-11-15T06:34:25.969 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:25.965+0000 7f3901ffb700 1 -- 172.21.15.81:0/2799058384 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f38fc010230 con 0x7f390c0f6510 2023-11-15T06:34:25.969 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:34:25.970 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:34:25.971 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:25.967+0000 7f3913269700 1 -- 172.21.15.81:0/2799058384 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f38f802ff60 msgr2=0x7f38f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:25.972 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:25.967+0000 7f3913269700 1 --2- 172.21.15.81:0/2799058384 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f38f802ff60 0x7f38f8032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f39080096a0 tx=0x7f3908006b40).stop 2023-11-15T06:34:25.972 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:25.967+0000 7f3913269700 1 -- 172.21.15.81:0/2799058384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f390c0f6510 msgr2=0x7f390c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:25.972 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:25.967+0000 7f3913269700 1 --2- 172.21.15.81:0/2799058384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f390c0f6510 0x7f390c106730 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f38fc013510 tx=0x7f38fc00b040).stop 2023-11-15T06:34:25.973 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:25.967+0000 7f3913269700 1 -- 172.21.15.81:0/2799058384 shutdown_connections 2023-11-15T06:34:25.973 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:25.967+0000 7f3913269700 1 --2- 172.21.15.81:0/2799058384 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f38f802ff60 0x7f38f8032420 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:25.973 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:25.967+0000 7f3913269700 1 --2- 172.21.15.81:0/2799058384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f390c0f6510 0x7f390c106730 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:25.974 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:25.967+0000 7f3913269700 1 -- 172.21.15.81:0/2799058384 >> 172.21.15.81:0/2799058384 conn(0x7f390c0f1c70 msgr2=0x7f390c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:34:25.974 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:25.967+0000 7f3913269700 1 -- 172.21.15.81:0/2799058384 shutdown_connections 2023-11-15T06:34:25.975 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:25.967+0000 7f3913269700 1 -- 172.21.15.81:0/2799058384 wait complete. 2023-11-15T06:34:25.975 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:34:27.379 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:34:27.380 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:34:27.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:26 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:34:25.860499+0000 mgr.smithi062.rwurtk (mgr.14164) 264 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:34:27.419 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:26 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:34:25.966178+0000 mon.smithi062 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.81:0/2799058384' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:34:28.567 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:28.565+0000 7f987358f700 1 -- 172.21.15.81:0/205486641 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f986c060220 msgr2=0x7f986c0f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:28.568 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:28.565+0000 7f987358f700 1 --2- 172.21.15.81:0/205486641 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f986c060220 0x7f986c0f9150 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f985c009960 tx=0x7f985c004d10).stop 2023-11-15T06:34:28.568 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:28.565+0000 7f987358f700 1 -- 172.21.15.81:0/205486641 shutdown_connections 2023-11-15T06:34:28.568 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:28.565+0000 7f987358f700 1 --2- 172.21.15.81:0/205486641 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f986c060220 0x7f986c0f9150 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:28.568 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:28.565+0000 7f987358f700 1 -- 172.21.15.81:0/205486641 >> 172.21.15.81:0/205486641 conn(0x7f986c0f0b70 msgr2=0x7f986c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:34:28.569 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:28.565+0000 7f987358f700 1 -- 172.21.15.81:0/205486641 shutdown_connections 2023-11-15T06:34:28.569 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:28.565+0000 7f987358f700 1 -- 172.21.15.81:0/205486641 wait complete. 2023-11-15T06:34:28.571 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:28.566+0000 7f987358f700 1 Processor -- start 2023-11-15T06:34:28.571 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:28.566+0000 7f987358f700 1 -- start start 2023-11-15T06:34:28.571 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:28.566+0000 7f987358f700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f986c060220 0x7f986c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:34:28.571 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:28.566+0000 7f987358f700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f986c106bd0 con 0x7f986c060220 2023-11-15T06:34:28.572 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:28.566+0000 7f987132b700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f986c060220 0x7f986c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:34:28.572 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:28.566+0000 7f987132b700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f986c060220 0x7f986c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:51784/0 (socket says 172.21.15.81:51784) 2023-11-15T06:34:28.572 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:28.566+0000 7f987132b700 1 -- 172.21.15.81:0/2294305995 learned_addr learned my addr 172.21.15.81:0/2294305995 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:34:28.572 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:28.566+0000 7f987132b700 1 -- 172.21.15.81:0/2294305995 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f985c009640 con 0x7f986c060220 2023-11-15T06:34:28.572 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:28.567+0000 7f987132b700 1 --2- 172.21.15.81:0/2294305995 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f986c060220 0x7f986c1066e0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f985c007bd0 tx=0x7f985c004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:34:28.573 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:28.567+0000 7f9869ffb700 1 -- 172.21.15.81:0/2294305995 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f985c018070 con 0x7f986c060220 2023-11-15T06:34:28.573 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:28.567+0000 7f987358f700 1 -- 172.21.15.81:0/2294305995 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f986c106dd0 con 0x7f986c060220 2023-11-15T06:34:28.573 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:28.567+0000 7f9869ffb700 1 -- 172.21.15.81:0/2294305995 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f985c00b870 con 0x7f986c060220 2023-11-15T06:34:28.574 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:28.567+0000 7f9869ffb700 1 -- 172.21.15.81:0/2294305995 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f985c01d420 con 0x7f986c060220 2023-11-15T06:34:28.574 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:28.567+0000 7f987358f700 1 -- 172.21.15.81:0/2294305995 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f986c107260 con 0x7f986c060220 2023-11-15T06:34:28.574 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:28.568+0000 7f9869ffb700 1 -- 172.21.15.81:0/2294305995 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f985c024060 con 0x7f986c060220 2023-11-15T06:34:28.574 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:28.568+0000 7f987358f700 1 -- 172.21.15.81:0/2294305995 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f986c045ad0 con 0x7f986c060220 2023-11-15T06:34:28.574 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:28.568+0000 7f9869ffb700 1 --2- 172.21.15.81:0/2294305995 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f985402ff60 0x7f9854032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:34:28.575 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:28.568+0000 7f9869ffb700 1 -- 172.21.15.81:0/2294305995 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f985c041150 con 0x7f986c060220 2023-11-15T06:34:28.575 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:28.568+0000 7f9870b2a700 1 --2- 172.21.15.81:0/2294305995 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f985402ff60 0x7f9854032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:34:28.575 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:28.568+0000 7f9870b2a700 1 --2- 172.21.15.81:0/2294305995 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f985402ff60 0x7f9854032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f98600096a0 tx=0x7f9860006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:34:28.575 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:28.573+0000 7f9869ffb700 1 -- 172.21.15.81:0/2294305995 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f985c025520 con 0x7f986c060220 2023-11-15T06:34:28.781 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:28.779+0000 7f987358f700 1 -- 172.21.15.81:0/2294305995 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f986c108060 con 0x7f986c060220 2023-11-15T06:34:28.782 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:28.780+0000 7f9869ffb700 1 -- 172.21.15.81:0/2294305995 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f985c01a2d0 con 0x7f986c060220 2023-11-15T06:34:28.783 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:34:28.783 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:34:28.784 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:28.782+0000 7f987358f700 1 -- 172.21.15.81:0/2294305995 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f985402ff60 msgr2=0x7f9854032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:28.784 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:28.782+0000 7f987358f700 1 --2- 172.21.15.81:0/2294305995 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f985402ff60 0x7f9854032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f98600096a0 tx=0x7f9860006b40).stop 2023-11-15T06:34:28.784 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:28.782+0000 7f987358f700 1 -- 172.21.15.81:0/2294305995 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f986c060220 msgr2=0x7f986c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:28.785 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:28.782+0000 7f987358f700 1 --2- 172.21.15.81:0/2294305995 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f986c060220 0x7f986c1066e0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f985c007bd0 tx=0x7f985c004d10).stop 2023-11-15T06:34:28.785 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:28.782+0000 7f987358f700 1 -- 172.21.15.81:0/2294305995 shutdown_connections 2023-11-15T06:34:28.785 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:28.782+0000 7f987358f700 1 --2- 172.21.15.81:0/2294305995 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f985402ff60 0x7f9854032420 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:28.785 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:28.782+0000 7f987358f700 1 --2- 172.21.15.81:0/2294305995 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f986c060220 0x7f986c1066e0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:28.785 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:28.782+0000 7f987358f700 1 -- 172.21.15.81:0/2294305995 >> 172.21.15.81:0/2294305995 conn(0x7f986c0f0b70 msgr2=0x7f986c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:34:28.785 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:28.782+0000 7f987358f700 1 -- 172.21.15.81:0/2294305995 shutdown_connections 2023-11-15T06:34:28.786 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:28.782+0000 7f987358f700 1 -- 172.21.15.81:0/2294305995 wait complete. 2023-11-15T06:34:28.786 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:34:28.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:28 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:34:27.860885+0000 mgr.smithi062.rwurtk (mgr.14164) 265 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:34:29.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:29 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:34:28.780966+0000 mon.smithi062 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.81:0/2294305995' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:34:30.225 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:34:30.225 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:34:31.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:30 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:34:29.861249+0000 mgr.smithi062.rwurtk (mgr.14164) 266 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:34:31.444 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:31.441+0000 7fba4337b700 1 -- 172.21.15.81:0/3880540430 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba3c0f6510 msgr2=0x7fba3c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:31.445 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:31.441+0000 7fba4337b700 1 --2- 172.21.15.81:0/3880540430 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba3c0f6510 0x7fba3c0f6930 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7fba2c009fb0 tx=0x7fba2c009b30).stop 2023-11-15T06:34:31.445 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:31.441+0000 7fba4337b700 1 -- 172.21.15.81:0/3880540430 shutdown_connections 2023-11-15T06:34:31.445 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:31.441+0000 7fba4337b700 1 --2- 172.21.15.81:0/3880540430 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba3c0f6510 0x7fba3c0f6930 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:31.445 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:31.441+0000 7fba4337b700 1 -- 172.21.15.81:0/3880540430 >> 172.21.15.81:0/3880540430 conn(0x7fba3c0f1c70 msgr2=0x7fba3c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:34:31.446 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:31.441+0000 7fba4337b700 1 -- 172.21.15.81:0/3880540430 shutdown_connections 2023-11-15T06:34:31.446 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:31.441+0000 7fba4337b700 1 -- 172.21.15.81:0/3880540430 wait complete. 2023-11-15T06:34:31.446 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:31.442+0000 7fba4337b700 1 Processor -- start 2023-11-15T06:34:31.446 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:31.442+0000 7fba4337b700 1 -- start start 2023-11-15T06:34:31.446 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:31.442+0000 7fba4337b700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba3c0f6510 0x7fba3c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:34:31.447 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:31.442+0000 7fba4337b700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3c106c20 con 0x7fba3c0f6510 2023-11-15T06:34:31.447 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:31.442+0000 7fba41117700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba3c0f6510 0x7fba3c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:34:31.447 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:31.442+0000 7fba41117700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba3c0f6510 0x7fba3c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:51806/0 (socket says 172.21.15.81:51806) 2023-11-15T06:34:31.447 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:31.442+0000 7fba41117700 1 -- 172.21.15.81:0/3509389468 learned_addr learned my addr 172.21.15.81:0/3509389468 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:34:31.447 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:31.442+0000 7fba41117700 1 -- 172.21.15.81:0/3509389468 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba2c005040 con 0x7fba3c0f6510 2023-11-15T06:34:31.448 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:31.443+0000 7fba41117700 1 --2- 172.21.15.81:0/3509389468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba3c0f6510 0x7fba3c106730 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7fba2c013510 tx=0x7fba2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:34:31.449 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:31.443+0000 7fba31ffb700 1 -- 172.21.15.81:0/3509389468 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba2c009320 con 0x7fba3c0f6510 2023-11-15T06:34:31.449 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:31.443+0000 7fba4337b700 1 -- 172.21.15.81:0/3509389468 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba3c106e20 con 0x7fba3c0f6510 2023-11-15T06:34:31.449 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:31.443+0000 7fba31ffb700 1 -- 172.21.15.81:0/3509389468 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fba2c013f00 con 0x7fba3c0f6510 2023-11-15T06:34:31.450 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:31.443+0000 7fba31ffb700 1 -- 172.21.15.81:0/3509389468 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba2c014d90 con 0x7fba3c0f6510 2023-11-15T06:34:31.450 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:31.443+0000 7fba4337b700 1 -- 172.21.15.81:0/3509389468 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba3c1072b0 con 0x7fba3c0f6510 2023-11-15T06:34:31.450 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:31.444+0000 7fba31ffb700 1 -- 172.21.15.81:0/3509389468 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fba2c0095f0 con 0x7fba3c0f6510 2023-11-15T06:34:31.450 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:31.444+0000 7fba4337b700 1 -- 172.21.15.81:0/3509389468 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba3c045ad0 con 0x7fba3c0f6510 2023-11-15T06:34:31.450 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:31.444+0000 7fba31ffb700 1 --2- 172.21.15.81:0/3509389468 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fba2802ff60 0x7fba28032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:34:31.451 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:31.444+0000 7fba31ffb700 1 -- 172.21.15.81:0/3509389468 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fba2c041000 con 0x7fba3c0f6510 2023-11-15T06:34:31.451 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:31.444+0000 7fba40916700 1 --2- 172.21.15.81:0/3509389468 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fba2802ff60 0x7fba28032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:34:31.451 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:31.445+0000 7fba40916700 1 --2- 172.21.15.81:0/3509389468 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fba2802ff60 0x7fba28032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fba380096a0 tx=0x7fba38006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:34:31.452 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:31.448+0000 7fba31ffb700 1 -- 172.21.15.81:0/3509389468 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fba2c014420 con 0x7fba3c0f6510 2023-11-15T06:34:31.654 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:31.651+0000 7fba4337b700 1 -- 172.21.15.81:0/3509389468 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fba3c0fbf60 con 0x7fba3c0f6510 2023-11-15T06:34:31.655 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:31.652+0000 7fba31ffb700 1 -- 172.21.15.81:0/3509389468 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fba2c01d7f0 con 0x7fba3c0f6510 2023-11-15T06:34:31.655 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:34:31.656 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:34:31.657 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:31.654+0000 7fba4337b700 1 -- 172.21.15.81:0/3509389468 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fba2802ff60 msgr2=0x7fba28032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:31.658 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:31.654+0000 7fba4337b700 1 --2- 172.21.15.81:0/3509389468 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fba2802ff60 0x7fba28032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fba380096a0 tx=0x7fba38006b40).stop 2023-11-15T06:34:31.658 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:31.654+0000 7fba4337b700 1 -- 172.21.15.81:0/3509389468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba3c0f6510 msgr2=0x7fba3c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:31.658 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:31.654+0000 7fba4337b700 1 --2- 172.21.15.81:0/3509389468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba3c0f6510 0x7fba3c106730 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7fba2c013510 tx=0x7fba2c00b040).stop 2023-11-15T06:34:31.659 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:31.654+0000 7fba4337b700 1 -- 172.21.15.81:0/3509389468 shutdown_connections 2023-11-15T06:34:31.659 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:31.654+0000 7fba4337b700 1 --2- 172.21.15.81:0/3509389468 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fba2802ff60 0x7fba28032420 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:31.660 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:31.654+0000 7fba4337b700 1 --2- 172.21.15.81:0/3509389468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba3c0f6510 0x7fba3c106730 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:31.660 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:31.654+0000 7fba4337b700 1 -- 172.21.15.81:0/3509389468 >> 172.21.15.81:0/3509389468 conn(0x7fba3c0f1c70 msgr2=0x7fba3c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:34:31.660 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:31.654+0000 7fba4337b700 1 -- 172.21.15.81:0/3509389468 shutdown_connections 2023-11-15T06:34:31.661 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:31.654+0000 7fba4337b700 1 -- 172.21.15.81:0/3509389468 wait complete. 2023-11-15T06:34:31.661 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:34:32.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:31 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:34:31.653007+0000 mon.smithi062 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.81:0/3509389468' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:34:33.056 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:34:33.057 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:34:33.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:32 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:34:31.861499+0000 mgr.smithi062.rwurtk (mgr.14164) 267 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:34:34.380 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:34.377+0000 7f6f5f160700 1 -- 172.21.15.81:0/1422151935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f580f5410 msgr2=0x7f6f580f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:34.381 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:34.377+0000 7f6f5f160700 1 --2- 172.21.15.81:0/1422151935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f580f5410 0x7f6f580f5830 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f6f4c009fb0 tx=0x7f6f4c009b30).stop 2023-11-15T06:34:34.381 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:34.377+0000 7f6f5f160700 1 -- 172.21.15.81:0/1422151935 shutdown_connections 2023-11-15T06:34:34.381 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:34.377+0000 7f6f5f160700 1 --2- 172.21.15.81:0/1422151935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f580f5410 0x7f6f580f5830 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:34.381 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:34.377+0000 7f6f5f160700 1 -- 172.21.15.81:0/1422151935 >> 172.21.15.81:0/1422151935 conn(0x7f6f580f0b70 msgr2=0x7f6f580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:34:34.382 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:34.377+0000 7f6f5f160700 1 -- 172.21.15.81:0/1422151935 shutdown_connections 2023-11-15T06:34:34.382 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:34.377+0000 7f6f5f160700 1 -- 172.21.15.81:0/1422151935 wait complete. 2023-11-15T06:34:34.382 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:34.377+0000 7f6f5f160700 1 Processor -- start 2023-11-15T06:34:34.382 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:34.377+0000 7f6f5f160700 1 -- start start 2023-11-15T06:34:34.382 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:34.378+0000 7f6f5f160700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f580f5410 0x7f6f58109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:34:34.383 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:34.378+0000 7f6f5f160700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f58060220 con 0x7f6f580f5410 2023-11-15T06:34:34.383 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:34.378+0000 7f6f5cefc700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f580f5410 0x7f6f58109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:34:34.383 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:34.378+0000 7f6f5cefc700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f580f5410 0x7f6f58109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:59928/0 (socket says 172.21.15.81:59928) 2023-11-15T06:34:34.383 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:34.378+0000 7f6f5cefc700 1 -- 172.21.15.81:0/1025402927 learned_addr learned my addr 172.21.15.81:0/1025402927 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:34:34.384 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:34.378+0000 7f6f5cefc700 1 -- 172.21.15.81:0/1025402927 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f4c005040 con 0x7f6f580f5410 2023-11-15T06:34:34.384 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:34.378+0000 7f6f5cefc700 1 --2- 172.21.15.81:0/1025402927 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f580f5410 0x7f6f58109450 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f6f4c014000 tx=0x7f6f4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:34:34.384 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:34.378+0000 7f6f557fa700 1 -- 172.21.15.81:0/1025402927 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f4c0095f0 con 0x7f6f580f5410 2023-11-15T06:34:34.384 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:34.378+0000 7f6f5f160700 1 -- 172.21.15.81:0/1025402927 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f58060420 con 0x7f6f580f5410 2023-11-15T06:34:34.384 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:34.378+0000 7f6f557fa700 1 -- 172.21.15.81:0/1025402927 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6f4c007e70 con 0x7f6f580f5410 2023-11-15T06:34:34.384 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:34.378+0000 7f6f557fa700 1 -- 172.21.15.81:0/1025402927 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f4c014ea0 con 0x7f6f580f5410 2023-11-15T06:34:34.385 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:34.378+0000 7f6f5f160700 1 -- 172.21.15.81:0/1025402927 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f58109b00 con 0x7f6f580f5410 2023-11-15T06:34:34.386 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:34.379+0000 7f6f557fa700 1 -- 172.21.15.81:0/1025402927 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6f4c025070 con 0x7f6f580f5410 2023-11-15T06:34:34.386 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:34.379+0000 7f6f5f160700 1 -- 172.21.15.81:0/1025402927 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f58059100 con 0x7f6f580f5410 2023-11-15T06:34:34.386 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:34.379+0000 7f6f557fa700 1 --2- 172.21.15.81:0/1025402927 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f6f4802ff60 0x7f6f48032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:34:34.386 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:34.379+0000 7f6f557fa700 1 -- 172.21.15.81:0/1025402927 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6f4c040970 con 0x7f6f580f5410 2023-11-15T06:34:34.387 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:34.380+0000 7f6f57fff700 1 --2- 172.21.15.81:0/1025402927 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f6f4802ff60 0x7f6f48032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:34:34.387 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:34.380+0000 7f6f57fff700 1 --2- 172.21.15.81:0/1025402927 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f6f4802ff60 0x7f6f48032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f6f440096a0 tx=0x7f6f44006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:34:34.387 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:34.384+0000 7f6f557fa700 1 -- 172.21.15.81:0/1025402927 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6f4c0079c0 con 0x7f6f580f5410 2023-11-15T06:34:34.591 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:34.589+0000 7f6f5f160700 1 -- 172.21.15.81:0/1025402927 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6f58045ad0 con 0x7f6f580f5410 2023-11-15T06:34:34.592 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:34.590+0000 7f6f557fa700 1 -- 172.21.15.81:0/1025402927 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6f4c00f300 con 0x7f6f580f5410 2023-11-15T06:34:34.593 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:34:34.593 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:34:34.594 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:34.592+0000 7f6f5f160700 1 -- 172.21.15.81:0/1025402927 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f6f4802ff60 msgr2=0x7f6f48032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:34.594 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:34.592+0000 7f6f5f160700 1 --2- 172.21.15.81:0/1025402927 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f6f4802ff60 0x7f6f48032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f6f440096a0 tx=0x7f6f44006b40).stop 2023-11-15T06:34:34.594 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:34.592+0000 7f6f5f160700 1 -- 172.21.15.81:0/1025402927 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f580f5410 msgr2=0x7f6f58109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:34.594 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:34.592+0000 7f6f5f160700 1 --2- 172.21.15.81:0/1025402927 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f580f5410 0x7f6f58109450 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f6f4c014000 tx=0x7f6f4c00b040).stop 2023-11-15T06:34:34.595 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:34.592+0000 7f6f5f160700 1 -- 172.21.15.81:0/1025402927 shutdown_connections 2023-11-15T06:34:34.595 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:34.592+0000 7f6f5f160700 1 --2- 172.21.15.81:0/1025402927 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f6f4802ff60 0x7f6f48032420 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:34.596 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:34.592+0000 7f6f5f160700 1 --2- 172.21.15.81:0/1025402927 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f580f5410 0x7f6f58109450 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:34.596 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:34.592+0000 7f6f5f160700 1 -- 172.21.15.81:0/1025402927 >> 172.21.15.81:0/1025402927 conn(0x7f6f580f0b70 msgr2=0x7f6f580f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:34:34.596 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:34.592+0000 7f6f5f160700 1 -- 172.21.15.81:0/1025402927 shutdown_connections 2023-11-15T06:34:34.596 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:34.592+0000 7f6f5f160700 1 -- 172.21.15.81:0/1025402927 wait complete. 2023-11-15T06:34:34.597 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:34:35.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:34 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:34:33.861732+0000 mgr.smithi062.rwurtk (mgr.14164) 268 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:34:35.169 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:34 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:34:34.591088+0000 mon.smithi062 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.81:0/1025402927' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:34:36.022 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:34:36.023 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:34:37.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:36 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:34:35.862056+0000 mgr.smithi062.rwurtk (mgr.14164) 269 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:34:37.320 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:37.316+0000 7fee0523c700 1 -- 172.21.15.81:0/398350605 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee000f6510 msgr2=0x7fee000f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:37.321 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:37.316+0000 7fee0523c700 1 --2- 172.21.15.81:0/398350605 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee000f6510 0x7fee000f6930 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7fedf4009fb0 tx=0x7fedf4009b30).stop 2023-11-15T06:34:37.321 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:37.316+0000 7fee0523c700 1 -- 172.21.15.81:0/398350605 shutdown_connections 2023-11-15T06:34:37.322 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:37.316+0000 7fee0523c700 1 --2- 172.21.15.81:0/398350605 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee000f6510 0x7fee000f6930 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:37.322 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:37.316+0000 7fee0523c700 1 -- 172.21.15.81:0/398350605 >> 172.21.15.81:0/398350605 conn(0x7fee000f1c70 msgr2=0x7fee000f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:34:37.322 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:37.316+0000 7fee0523c700 1 -- 172.21.15.81:0/398350605 shutdown_connections 2023-11-15T06:34:37.322 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:37.316+0000 7fee0523c700 1 -- 172.21.15.81:0/398350605 wait complete. 2023-11-15T06:34:37.323 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:37.316+0000 7fee0523c700 1 Processor -- start 2023-11-15T06:34:37.323 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:37.316+0000 7fee0523c700 1 -- start start 2023-11-15T06:34:37.323 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:37.317+0000 7fee0523c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee000f6510 0x7fee001066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:34:37.323 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:37.317+0000 7fee0523c700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee00106bc0 con 0x7fee000f6510 2023-11-15T06:34:37.323 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:37.317+0000 7fedfed9d700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee000f6510 0x7fee001066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:34:37.324 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:37.317+0000 7fedfed9d700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee000f6510 0x7fee001066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:59952/0 (socket says 172.21.15.81:59952) 2023-11-15T06:34:37.325 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:37.317+0000 7fedfed9d700 1 -- 172.21.15.81:0/938801500 learned_addr learned my addr 172.21.15.81:0/938801500 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:34:37.325 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:37.317+0000 7fedfed9d700 1 -- 172.21.15.81:0/938801500 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf4005040 con 0x7fee000f6510 2023-11-15T06:34:37.325 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:37.317+0000 7fedfed9d700 1 --2- 172.21.15.81:0/938801500 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee000f6510 0x7fee001066d0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7fedf4013510 tx=0x7fedf400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:34:37.326 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:37.318+0000 7fede77fe700 1 -- 172.21.15.81:0/938801500 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fedf4009320 con 0x7fee000f6510 2023-11-15T06:34:37.326 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:37.318+0000 7fee0523c700 1 -- 172.21.15.81:0/938801500 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee00106dc0 con 0x7fee000f6510 2023-11-15T06:34:37.327 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:37.318+0000 7fede77fe700 1 -- 172.21.15.81:0/938801500 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fedf4013f00 con 0x7fee000f6510 2023-11-15T06:34:37.327 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:37.318+0000 7fede77fe700 1 -- 172.21.15.81:0/938801500 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fedf4014d90 con 0x7fee000f6510 2023-11-15T06:34:37.327 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:37.318+0000 7fee0523c700 1 -- 172.21.15.81:0/938801500 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee00107250 con 0x7fee000f6510 2023-11-15T06:34:37.329 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:37.319+0000 7fee0523c700 1 -- 172.21.15.81:0/938801500 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee00045ad0 con 0x7fee000f6510 2023-11-15T06:34:37.330 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:37.319+0000 7fede77fe700 1 -- 172.21.15.81:0/938801500 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fedf4014420 con 0x7fee000f6510 2023-11-15T06:34:37.330 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:37.319+0000 7fede77fe700 1 --2- 172.21.15.81:0/938801500 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fede802ff60 0x7fede8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:34:37.330 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:37.319+0000 7fede77fe700 1 -- 172.21.15.81:0/938801500 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fedf4041000 con 0x7fee000f6510 2023-11-15T06:34:37.331 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:37.319+0000 7fedfe59c700 1 --2- 172.21.15.81:0/938801500 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fede802ff60 0x7fede8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:34:37.331 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:37.319+0000 7fedfe59c700 1 --2- 172.21.15.81:0/938801500 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fede802ff60 0x7fede8032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fedf00096a0 tx=0x7fedf0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:34:37.331 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:37.324+0000 7fede77fe700 1 -- 172.21.15.81:0/938801500 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fedf40095f0 con 0x7fee000f6510 2023-11-15T06:34:37.532 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:37.530+0000 7fee0523c700 1 -- 172.21.15.81:0/938801500 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fee00108100 con 0x7fee000f6510 2023-11-15T06:34:37.532 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:37.530+0000 7fede77fe700 1 -- 172.21.15.81:0/938801500 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fedf401db70 con 0x7fee000f6510 2023-11-15T06:34:37.533 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:34:37.533 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:34:37.534 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:37.532+0000 7fee0523c700 1 -- 172.21.15.81:0/938801500 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fede802ff60 msgr2=0x7fede8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:37.535 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:37.532+0000 7fee0523c700 1 --2- 172.21.15.81:0/938801500 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fede802ff60 0x7fede8032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fedf00096a0 tx=0x7fedf0006b40).stop 2023-11-15T06:34:37.535 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:37.532+0000 7fee0523c700 1 -- 172.21.15.81:0/938801500 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee000f6510 msgr2=0x7fee001066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:37.535 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:37.532+0000 7fee0523c700 1 --2- 172.21.15.81:0/938801500 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee000f6510 0x7fee001066d0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7fedf4013510 tx=0x7fedf400b040).stop 2023-11-15T06:34:37.535 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:37.532+0000 7fee0523c700 1 -- 172.21.15.81:0/938801500 shutdown_connections 2023-11-15T06:34:37.535 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:37.532+0000 7fee0523c700 1 --2- 172.21.15.81:0/938801500 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fede802ff60 0x7fede8032420 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:37.536 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:37.532+0000 7fee0523c700 1 --2- 172.21.15.81:0/938801500 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fee000f6510 0x7fee001066d0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:37.536 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:37.532+0000 7fee0523c700 1 -- 172.21.15.81:0/938801500 >> 172.21.15.81:0/938801500 conn(0x7fee000f1c70 msgr2=0x7fee000f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:34:37.536 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:37.532+0000 7fee0523c700 1 -- 172.21.15.81:0/938801500 shutdown_connections 2023-11-15T06:34:37.536 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:37.532+0000 7fee0523c700 1 -- 172.21.15.81:0/938801500 wait complete. 2023-11-15T06:34:37.537 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:34:38.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:37 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:34:37.531347+0000 mon.smithi062 (mon.0) 315 : audit [DBG] from='client.? 172.21.15.81:0/938801500' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:34:38.934 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:34:38.935 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:34:39.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:38 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:34:37.862455+0000 mgr.smithi062.rwurtk (mgr.14164) 270 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:34:40.152 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:40.148+0000 7f1bd1996700 1 -- 172.21.15.81:0/893248386 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1bcc0f78e0 msgr2=0x7f1bcc0f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:40.152 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:40.148+0000 7f1bd1996700 1 --2- 172.21.15.81:0/893248386 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1bcc0f78e0 0x7f1bcc0f7d00 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f1bc0009fb0 tx=0x7f1bc0009b30).stop 2023-11-15T06:34:40.152 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:40.148+0000 7f1bd1996700 1 -- 172.21.15.81:0/893248386 shutdown_connections 2023-11-15T06:34:40.152 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:40.148+0000 7f1bd1996700 1 --2- 172.21.15.81:0/893248386 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1bcc0f78e0 0x7f1bcc0f7d00 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:40.153 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:40.148+0000 7f1bd1996700 1 -- 172.21.15.81:0/893248386 >> 172.21.15.81:0/893248386 conn(0x7f1bcc0f0120 msgr2=0x7f1bcc0f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:34:40.153 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:40.149+0000 7f1bd1996700 1 -- 172.21.15.81:0/893248386 shutdown_connections 2023-11-15T06:34:40.153 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:40.149+0000 7f1bd1996700 1 -- 172.21.15.81:0/893248386 wait complete. 2023-11-15T06:34:40.153 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:40.149+0000 7f1bd1996700 1 Processor -- start 2023-11-15T06:34:40.153 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:40.149+0000 7f1bd1996700 1 -- start start 2023-11-15T06:34:40.154 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:40.149+0000 7f1bd1996700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1bcc0f78e0 0x7f1bcc0f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:34:40.154 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:40.149+0000 7f1bd1996700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bcc0f5630 con 0x7f1bcc0f78e0 2023-11-15T06:34:40.154 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:40.150+0000 7f1bd0994700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1bcc0f78e0 0x7f1bcc0f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:34:40.154 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:40.150+0000 7f1bd0994700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1bcc0f78e0 0x7f1bcc0f71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:59968/0 (socket says 172.21.15.81:59968) 2023-11-15T06:34:40.154 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:40.150+0000 7f1bd0994700 1 -- 172.21.15.81:0/1462123316 learned_addr learned my addr 172.21.15.81:0/1462123316 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:34:40.156 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:40.150+0000 7f1bd0994700 1 -- 172.21.15.81:0/1462123316 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bc0005040 con 0x7f1bcc0f78e0 2023-11-15T06:34:40.156 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:40.150+0000 7f1bd0994700 1 --2- 172.21.15.81:0/1462123316 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1bcc0f78e0 0x7f1bcc0f71b0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f1bc0013510 tx=0x7f1bc000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:34:40.156 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:40.150+0000 7f1bc97fa700 1 -- 172.21.15.81:0/1462123316 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1bc0009320 con 0x7f1bcc0f78e0 2023-11-15T06:34:40.157 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:40.150+0000 7f1bd1996700 1 -- 172.21.15.81:0/1462123316 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bcc0f57d0 con 0x7f1bcc0f78e0 2023-11-15T06:34:40.157 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:40.150+0000 7f1bc97fa700 1 -- 172.21.15.81:0/1462123316 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1bc0013f00 con 0x7f1bcc0f78e0 2023-11-15T06:34:40.157 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:40.150+0000 7f1bc97fa700 1 -- 172.21.15.81:0/1462123316 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1bc0014dc0 con 0x7f1bcc0f78e0 2023-11-15T06:34:40.157 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:40.150+0000 7f1bd1996700 1 -- 172.21.15.81:0/1462123316 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bcc0f5c60 con 0x7f1bcc0f78e0 2023-11-15T06:34:40.157 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:40.151+0000 7f1bc97fa700 1 -- 172.21.15.81:0/1462123316 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1bc00095f0 con 0x7f1bcc0f78e0 2023-11-15T06:34:40.158 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:40.151+0000 7f1bd1996700 1 -- 172.21.15.81:0/1462123316 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bcc059100 con 0x7f1bcc0f78e0 2023-11-15T06:34:40.158 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:40.151+0000 7f1bc97fa700 1 --2- 172.21.15.81:0/1462123316 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f1bbc02ff60 0x7f1bbc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:34:40.158 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:40.151+0000 7f1bc97fa700 1 -- 172.21.15.81:0/1462123316 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1bc0041000 con 0x7f1bcc0f78e0 2023-11-15T06:34:40.158 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:40.152+0000 7f1bcbfff700 1 --2- 172.21.15.81:0/1462123316 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f1bbc02ff60 0x7f1bbc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:34:40.158 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:40.152+0000 7f1bcbfff700 1 --2- 172.21.15.81:0/1462123316 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f1bbc02ff60 0x7f1bbc032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f1bb80096a0 tx=0x7f1bb8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:34:40.159 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:40.156+0000 7f1bc97fa700 1 -- 172.21.15.81:0/1462123316 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1bc0014420 con 0x7f1bcc0f78e0 2023-11-15T06:34:40.362 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:40.360+0000 7f1bd1996700 1 -- 172.21.15.81:0/1462123316 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1bcc0fbce0 con 0x7f1bcc0f78e0 2023-11-15T06:34:40.363 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:40.361+0000 7f1bc97fa700 1 -- 172.21.15.81:0/1462123316 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1bc0010230 con 0x7f1bcc0f78e0 2023-11-15T06:34:40.363 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:34:40.364 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:34:40.364 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:40.362+0000 7f1bd1996700 1 -- 172.21.15.81:0/1462123316 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f1bbc02ff60 msgr2=0x7f1bbc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:40.365 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:40.362+0000 7f1bd1996700 1 --2- 172.21.15.81:0/1462123316 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f1bbc02ff60 0x7f1bbc032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f1bb80096a0 tx=0x7f1bb8006b40).stop 2023-11-15T06:34:40.365 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:40.362+0000 7f1bd1996700 1 -- 172.21.15.81:0/1462123316 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1bcc0f78e0 msgr2=0x7f1bcc0f71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:40.365 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:40.362+0000 7f1bd1996700 1 --2- 172.21.15.81:0/1462123316 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1bcc0f78e0 0x7f1bcc0f71b0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f1bc0013510 tx=0x7f1bc000b040).stop 2023-11-15T06:34:40.365 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:40.362+0000 7f1bd1996700 1 -- 172.21.15.81:0/1462123316 shutdown_connections 2023-11-15T06:34:40.365 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:40.362+0000 7f1bd1996700 1 --2- 172.21.15.81:0/1462123316 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f1bbc02ff60 0x7f1bbc032420 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:40.366 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:40.362+0000 7f1bd1996700 1 --2- 172.21.15.81:0/1462123316 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1bcc0f78e0 0x7f1bcc0f71b0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:40.366 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:40.362+0000 7f1bd1996700 1 -- 172.21.15.81:0/1462123316 >> 172.21.15.81:0/1462123316 conn(0x7f1bcc0f0120 msgr2=0x7f1bcc0f0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:34:40.366 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:40.362+0000 7f1bd1996700 1 -- 172.21.15.81:0/1462123316 shutdown_connections 2023-11-15T06:34:40.366 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:40.362+0000 7f1bd1996700 1 -- 172.21.15.81:0/1462123316 wait complete. 2023-11-15T06:34:40.367 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:34:41.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:40 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:34:39.862770+0000 mgr.smithi062.rwurtk (mgr.14164) 271 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:34:41.419 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:40 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:34:40.361562+0000 mon.smithi062 (mon.0) 316 : audit [DBG] from='client.? 172.21.15.81:0/1462123316' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:34:41.784 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:34:41.785 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:34:43.061 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:43.058+0000 7f9059246700 1 -- 172.21.15.81:0/3673093429 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90540f5410 msgr2=0x7f90540f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:43.061 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:43.058+0000 7f9059246700 1 --2- 172.21.15.81:0/3673093429 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90540f5410 0x7f90540f5830 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f9048009fb0 tx=0x7f9048009b30).stop 2023-11-15T06:34:43.061 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:43.058+0000 7f9059246700 1 -- 172.21.15.81:0/3673093429 shutdown_connections 2023-11-15T06:34:43.062 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:43.058+0000 7f9059246700 1 --2- 172.21.15.81:0/3673093429 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90540f5410 0x7f90540f5830 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:43.062 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:43.058+0000 7f9059246700 1 -- 172.21.15.81:0/3673093429 >> 172.21.15.81:0/3673093429 conn(0x7f90540f0b70 msgr2=0x7f90540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:34:43.062 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:43.058+0000 7f9059246700 1 -- 172.21.15.81:0/3673093429 shutdown_connections 2023-11-15T06:34:43.062 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:43.058+0000 7f9059246700 1 -- 172.21.15.81:0/3673093429 wait complete. 2023-11-15T06:34:43.064 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:43.059+0000 7f9059246700 1 Processor -- start 2023-11-15T06:34:43.064 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:43.059+0000 7f9059246700 1 -- start start 2023-11-15T06:34:43.064 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:43.059+0000 7f9059246700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90540f5410 0x7f9054106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:34:43.065 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:43.059+0000 7f9059246700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9054106c00 con 0x7f90540f5410 2023-11-15T06:34:43.065 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:43.060+0000 7f9052d9d700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90540f5410 0x7f9054106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:34:43.065 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:43.060+0000 7f9052d9d700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90540f5410 0x7f9054106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:45058/0 (socket says 172.21.15.81:45058) 2023-11-15T06:34:43.065 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:43.060+0000 7f9052d9d700 1 -- 172.21.15.81:0/1155184228 learned_addr learned my addr 172.21.15.81:0/1155184228 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:34:43.066 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:43.060+0000 7f9052d9d700 1 -- 172.21.15.81:0/1155184228 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9048005040 con 0x7f90540f5410 2023-11-15T06:34:43.066 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:43.060+0000 7f9052d9d700 1 --2- 172.21.15.81:0/1155184228 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90540f5410 0x7f9054106710 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f9048013510 tx=0x7f904800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:34:43.066 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:43.060+0000 7f903b7fe700 1 -- 172.21.15.81:0/1155184228 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9048009320 con 0x7f90540f5410 2023-11-15T06:34:43.066 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:43.060+0000 7f9059246700 1 -- 172.21.15.81:0/1155184228 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9054106e00 con 0x7f90540f5410 2023-11-15T06:34:43.067 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:43.060+0000 7f9059246700 1 -- 172.21.15.81:0/1155184228 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9054107290 con 0x7f90540f5410 2023-11-15T06:34:43.067 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:43.060+0000 7f903b7fe700 1 -- 172.21.15.81:0/1155184228 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9048013f00 con 0x7f90540f5410 2023-11-15T06:34:43.067 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:43.060+0000 7f903b7fe700 1 -- 172.21.15.81:0/1155184228 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9048014d60 con 0x7f90540f5410 2023-11-15T06:34:43.067 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:43.061+0000 7f903b7fe700 1 -- 172.21.15.81:0/1155184228 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f90480263a0 con 0x7f90540f5410 2023-11-15T06:34:43.067 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:43.061+0000 7f9059246700 1 -- 172.21.15.81:0/1155184228 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9054045ad0 con 0x7f90540f5410 2023-11-15T06:34:43.067 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:43.061+0000 7f903b7fe700 1 --2- 172.21.15.81:0/1155184228 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f903c02ff60 0x7f903c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:34:43.068 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:43.061+0000 7f903b7fe700 1 -- 172.21.15.81:0/1155184228 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9048041400 con 0x7f90540f5410 2023-11-15T06:34:43.068 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:43.061+0000 7f905259c700 1 --2- 172.21.15.81:0/1155184228 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f903c02ff60 0x7f903c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:34:43.068 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:43.062+0000 7f905259c700 1 --2- 172.21.15.81:0/1155184228 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f903c02ff60 0x7f903c032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f90440096a0 tx=0x7f9044006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:34:43.069 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:43.066+0000 7f903b7fe700 1 -- 172.21.15.81:0/1155184228 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9048012330 con 0x7f90540f5410 2023-11-15T06:34:43.270 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:43.268+0000 7f9059246700 1 -- 172.21.15.81:0/1155184228 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f90540fae60 con 0x7f90540f5410 2023-11-15T06:34:43.271 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:43.269+0000 7f903b7fe700 1 -- 172.21.15.81:0/1155184228 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9048012330 con 0x7f90540f5410 2023-11-15T06:34:43.271 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:34:43.272 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:34:43.273 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:43.271+0000 7f9059246700 1 -- 172.21.15.81:0/1155184228 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f903c02ff60 msgr2=0x7f903c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:43.273 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:43.271+0000 7f9059246700 1 --2- 172.21.15.81:0/1155184228 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f903c02ff60 0x7f903c032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f90440096a0 tx=0x7f9044006b40).stop 2023-11-15T06:34:43.274 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:43.271+0000 7f9059246700 1 -- 172.21.15.81:0/1155184228 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90540f5410 msgr2=0x7f9054106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:43.274 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:43.271+0000 7f9059246700 1 --2- 172.21.15.81:0/1155184228 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90540f5410 0x7f9054106710 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f9048013510 tx=0x7f904800b040).stop 2023-11-15T06:34:43.274 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:43.271+0000 7f9059246700 1 -- 172.21.15.81:0/1155184228 shutdown_connections 2023-11-15T06:34:43.274 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:43.271+0000 7f9059246700 1 --2- 172.21.15.81:0/1155184228 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f903c02ff60 0x7f903c032420 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:43.274 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:43.271+0000 7f9059246700 1 --2- 172.21.15.81:0/1155184228 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90540f5410 0x7f9054106710 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:43.275 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:43.271+0000 7f9059246700 1 -- 172.21.15.81:0/1155184228 >> 172.21.15.81:0/1155184228 conn(0x7f90540f0b70 msgr2=0x7f905405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:34:43.275 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:43.271+0000 7f9059246700 1 -- 172.21.15.81:0/1155184228 shutdown_connections 2023-11-15T06:34:43.275 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:43.271+0000 7f9059246700 1 -- 172.21.15.81:0/1155184228 wait complete. 2023-11-15T06:34:43.276 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:34:43.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:42 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:34:41.863018+0000 mgr.smithi062.rwurtk (mgr.14164) 272 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:34:44.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:43 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:34:43.269855+0000 mon.smithi062 (mon.0) 317 : audit [DBG] from='client.? 172.21.15.81:0/1155184228' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:34:44.652 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:34:44.653 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:34:45.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:44 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:34:43.863421+0000 mgr.smithi062.rwurtk (mgr.14164) 273 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:34:45.911 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:45.907+0000 7f82d319b700 1 -- 172.21.15.81:0/184132794 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82cc0f5410 msgr2=0x7f82cc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:45.911 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:45.907+0000 7f82d319b700 1 --2- 172.21.15.81:0/184132794 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82cc0f5410 0x7f82cc0f5830 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f82c0009fb0 tx=0x7f82c0009b30).stop 2023-11-15T06:34:45.911 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:45.907+0000 7f82d319b700 1 -- 172.21.15.81:0/184132794 shutdown_connections 2023-11-15T06:34:45.912 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:45.907+0000 7f82d319b700 1 --2- 172.21.15.81:0/184132794 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82cc0f5410 0x7f82cc0f5830 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:45.912 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:45.907+0000 7f82d319b700 1 -- 172.21.15.81:0/184132794 >> 172.21.15.81:0/184132794 conn(0x7f82cc0f0b70 msgr2=0x7f82cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:34:45.912 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:45.907+0000 7f82d319b700 1 -- 172.21.15.81:0/184132794 shutdown_connections 2023-11-15T06:34:45.912 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:45.907+0000 7f82d319b700 1 -- 172.21.15.81:0/184132794 wait complete. 2023-11-15T06:34:45.912 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:45.908+0000 7f82d319b700 1 Processor -- start 2023-11-15T06:34:45.913 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:45.908+0000 7f82d319b700 1 -- start start 2023-11-15T06:34:45.913 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:45.908+0000 7f82d319b700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82cc0f5410 0x7f82cc109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:34:45.913 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:45.908+0000 7f82d319b700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82cc060240 con 0x7f82cc0f5410 2023-11-15T06:34:45.913 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:45.908+0000 7f82d0f37700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82cc0f5410 0x7f82cc109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:34:45.913 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:45.908+0000 7f82d0f37700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82cc0f5410 0x7f82cc109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:45068/0 (socket says 172.21.15.81:45068) 2023-11-15T06:34:45.914 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:45.908+0000 7f82d0f37700 1 -- 172.21.15.81:0/1199513875 learned_addr learned my addr 172.21.15.81:0/1199513875 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:34:45.914 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:45.909+0000 7f82d0f37700 1 -- 172.21.15.81:0/1199513875 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82c0005040 con 0x7f82cc0f5410 2023-11-15T06:34:45.914 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:45.909+0000 7f82d0f37700 1 --2- 172.21.15.81:0/1199513875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82cc0f5410 0x7f82cc109430 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f82c0014000 tx=0x7f82c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:34:45.914 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:45.909+0000 7f82c97fa700 1 -- 172.21.15.81:0/1199513875 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82c00095f0 con 0x7f82cc0f5410 2023-11-15T06:34:45.915 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:45.909+0000 7f82d319b700 1 -- 172.21.15.81:0/1199513875 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82cc060440 con 0x7f82cc0f5410 2023-11-15T06:34:45.915 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:45.909+0000 7f82c97fa700 1 -- 172.21.15.81:0/1199513875 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f82c0007e70 con 0x7f82cc0f5410 2023-11-15T06:34:45.915 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:45.909+0000 7f82c97fa700 1 -- 172.21.15.81:0/1199513875 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82c0014ea0 con 0x7f82cc0f5410 2023-11-15T06:34:45.915 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:45.909+0000 7f82d319b700 1 -- 172.21.15.81:0/1199513875 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82cc109ae0 con 0x7f82cc0f5410 2023-11-15T06:34:45.916 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:45.910+0000 7f82c97fa700 1 -- 172.21.15.81:0/1199513875 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f82c0025070 con 0x7f82cc0f5410 2023-11-15T06:34:45.917 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:45.910+0000 7f82d319b700 1 -- 172.21.15.81:0/1199513875 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82cc045ad0 con 0x7f82cc0f5410 2023-11-15T06:34:45.917 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:45.910+0000 7f82c97fa700 1 --2- 172.21.15.81:0/1199513875 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f82bc02ff60 0x7f82bc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:34:45.917 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:45.910+0000 7f82c97fa700 1 -- 172.21.15.81:0/1199513875 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f82c0040950 con 0x7f82cc0f5410 2023-11-15T06:34:45.917 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:45.910+0000 7f82cbfff700 1 --2- 172.21.15.81:0/1199513875 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f82bc02ff60 0x7f82bc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:34:45.917 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:45.911+0000 7f82cbfff700 1 --2- 172.21.15.81:0/1199513875 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f82bc02ff60 0x7f82bc032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f82b80096a0 tx=0x7f82b8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:34:45.918 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:45.915+0000 7f82c97fa700 1 -- 172.21.15.81:0/1199513875 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f82c000f4e0 con 0x7f82cc0f5410 2023-11-15T06:34:46.121 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:46.118+0000 7f82d319b700 1 -- 172.21.15.81:0/1199513875 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f82cc0fae60 con 0x7f82cc0f5410 2023-11-15T06:34:46.121 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:46.119+0000 7f82c97fa700 1 -- 172.21.15.81:0/1199513875 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f82c000f300 con 0x7f82cc0f5410 2023-11-15T06:34:46.122 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:34:46.122 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:34:46.123 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:46.121+0000 7f82d319b700 1 -- 172.21.15.81:0/1199513875 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f82bc02ff60 msgr2=0x7f82bc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:46.123 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:46.121+0000 7f82d319b700 1 --2- 172.21.15.81:0/1199513875 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f82bc02ff60 0x7f82bc032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f82b80096a0 tx=0x7f82b8006b40).stop 2023-11-15T06:34:46.123 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:46.121+0000 7f82d319b700 1 -- 172.21.15.81:0/1199513875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82cc0f5410 msgr2=0x7f82cc109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:46.124 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:46.121+0000 7f82d319b700 1 --2- 172.21.15.81:0/1199513875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82cc0f5410 0x7f82cc109430 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f82c0014000 tx=0x7f82c000b040).stop 2023-11-15T06:34:46.124 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:46.121+0000 7f82d319b700 1 -- 172.21.15.81:0/1199513875 shutdown_connections 2023-11-15T06:34:46.124 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:46.121+0000 7f82d319b700 1 --2- 172.21.15.81:0/1199513875 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f82bc02ff60 0x7f82bc032420 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:46.125 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:46.121+0000 7f82d319b700 1 --2- 172.21.15.81:0/1199513875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82cc0f5410 0x7f82cc109430 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:46.125 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:46.121+0000 7f82d319b700 1 -- 172.21.15.81:0/1199513875 >> 172.21.15.81:0/1199513875 conn(0x7f82cc0f0b70 msgr2=0x7f82cc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:34:46.125 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:46.121+0000 7f82d319b700 1 -- 172.21.15.81:0/1199513875 shutdown_connections 2023-11-15T06:34:46.125 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:46.121+0000 7f82d319b700 1 -- 172.21.15.81:0/1199513875 wait complete. 2023-11-15T06:34:46.126 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:34:47.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:46 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:34:45.863829+0000 mgr.smithi062.rwurtk (mgr.14164) 274 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:34:47.419 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:46 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:34:46.119952+0000 mon.smithi062 (mon.0) 318 : audit [DBG] from='client.? 172.21.15.81:0/1199513875' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:34:47.524 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:34:47.524 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:34:48.683 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:48.680+0000 7f7df1efe700 1 -- 172.21.15.81:0/4179254691 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7dec0f6510 msgr2=0x7f7dec0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:48.683 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:48.680+0000 7f7df1efe700 1 --2- 172.21.15.81:0/4179254691 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7dec0f6510 0x7f7dec0f6930 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f7de0009fb0 tx=0x7f7de0009b30).stop 2023-11-15T06:34:48.683 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:48.680+0000 7f7df1efe700 1 -- 172.21.15.81:0/4179254691 shutdown_connections 2023-11-15T06:34:48.684 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:48.680+0000 7f7df1efe700 1 --2- 172.21.15.81:0/4179254691 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7dec0f6510 0x7f7dec0f6930 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:48.684 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:48.680+0000 7f7df1efe700 1 -- 172.21.15.81:0/4179254691 >> 172.21.15.81:0/4179254691 conn(0x7f7dec0f1c70 msgr2=0x7f7dec0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:34:48.684 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:48.680+0000 7f7df1efe700 1 -- 172.21.15.81:0/4179254691 shutdown_connections 2023-11-15T06:34:48.684 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:48.680+0000 7f7df1efe700 1 -- 172.21.15.81:0/4179254691 wait complete. 2023-11-15T06:34:48.684 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:48.681+0000 7f7df1efe700 1 Processor -- start 2023-11-15T06:34:48.686 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:48.681+0000 7f7df1efe700 1 -- start start 2023-11-15T06:34:48.686 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:48.681+0000 7f7df1efe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7dec0f6510 0x7f7dec06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:34:48.687 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:48.681+0000 7f7df1efe700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dec069e20 con 0x7f7dec0f6510 2023-11-15T06:34:48.687 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:48.681+0000 7f7deb7fe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7dec0f6510 0x7f7dec06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:34:48.687 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:48.681+0000 7f7deb7fe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7dec0f6510 0x7f7dec06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:45096/0 (socket says 172.21.15.81:45096) 2023-11-15T06:34:48.687 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:48.681+0000 7f7deb7fe700 1 -- 172.21.15.81:0/4282252485 learned_addr learned my addr 172.21.15.81:0/4282252485 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:34:48.687 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:48.682+0000 7f7deb7fe700 1 -- 172.21.15.81:0/4282252485 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7de0005040 con 0x7f7dec0f6510 2023-11-15T06:34:48.688 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:48.682+0000 7f7deb7fe700 1 --2- 172.21.15.81:0/4282252485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7dec0f6510 0x7f7dec06b9a0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f7de0013510 tx=0x7f7de000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:34:48.688 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:48.682+0000 7f7dd3fff700 1 -- 172.21.15.81:0/4282252485 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7de0009320 con 0x7f7dec0f6510 2023-11-15T06:34:48.688 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:48.682+0000 7f7df1efe700 1 -- 172.21.15.81:0/4282252485 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dec069fc0 con 0x7f7dec0f6510 2023-11-15T06:34:48.689 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:48.682+0000 7f7dd3fff700 1 -- 172.21.15.81:0/4282252485 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7de0013f00 con 0x7f7dec0f6510 2023-11-15T06:34:48.689 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:48.682+0000 7f7dd3fff700 1 -- 172.21.15.81:0/4282252485 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7de0014d90 con 0x7f7dec0f6510 2023-11-15T06:34:48.689 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:48.682+0000 7f7df1efe700 1 -- 172.21.15.81:0/4282252485 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dec06a450 con 0x7f7dec0f6510 2023-11-15T06:34:48.689 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:48.683+0000 7f7dd3fff700 1 -- 172.21.15.81:0/4282252485 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7de00095f0 con 0x7f7dec0f6510 2023-11-15T06:34:48.689 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:48.683+0000 7f7df1efe700 1 -- 172.21.15.81:0/4282252485 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dec059100 con 0x7f7dec0f6510 2023-11-15T06:34:48.690 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:48.683+0000 7f7dd3fff700 1 --2- 172.21.15.81:0/4282252485 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f7dd402ff60 0x7f7dd4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:34:48.690 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:48.683+0000 7f7dd3fff700 1 -- 172.21.15.81:0/4282252485 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7de00410a0 con 0x7f7dec0f6510 2023-11-15T06:34:48.690 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:48.683+0000 7f7deaffd700 1 --2- 172.21.15.81:0/4282252485 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f7dd402ff60 0x7f7dd4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:34:48.690 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:48.684+0000 7f7deaffd700 1 --2- 172.21.15.81:0/4282252485 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f7dd402ff60 0x7f7dd4032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f7ddc0096a0 tx=0x7f7ddc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:34:48.691 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:48.688+0000 7f7dd3fff700 1 -- 172.21.15.81:0/4282252485 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7de001d7f0 con 0x7f7dec0f6510 2023-11-15T06:34:48.897 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:48.895+0000 7f7df1efe700 1 -- 172.21.15.81:0/4282252485 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7dec0fbf60 con 0x7f7dec0f6510 2023-11-15T06:34:48.898 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:48.896+0000 7f7dd3fff700 1 -- 172.21.15.81:0/4282252485 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7de0010230 con 0x7f7dec0f6510 2023-11-15T06:34:48.899 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:34:48.900 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:34:48.901 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:48.897+0000 7f7df1efe700 1 -- 172.21.15.81:0/4282252485 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f7dd402ff60 msgr2=0x7f7dd4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:48.901 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:48.897+0000 7f7df1efe700 1 --2- 172.21.15.81:0/4282252485 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f7dd402ff60 0x7f7dd4032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f7ddc0096a0 tx=0x7f7ddc006b40).stop 2023-11-15T06:34:48.903 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:48.897+0000 7f7df1efe700 1 -- 172.21.15.81:0/4282252485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7dec0f6510 msgr2=0x7f7dec06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:48.903 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:48.897+0000 7f7df1efe700 1 --2- 172.21.15.81:0/4282252485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7dec0f6510 0x7f7dec06b9a0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f7de0013510 tx=0x7f7de000b040).stop 2023-11-15T06:34:48.904 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:48.897+0000 7f7df1efe700 1 -- 172.21.15.81:0/4282252485 shutdown_connections 2023-11-15T06:34:48.904 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:48.897+0000 7f7df1efe700 1 --2- 172.21.15.81:0/4282252485 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f7dd402ff60 0x7f7dd4032420 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:48.904 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:48.897+0000 7f7df1efe700 1 --2- 172.21.15.81:0/4282252485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7dec0f6510 0x7f7dec06b9a0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:48.905 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:48.897+0000 7f7df1efe700 1 -- 172.21.15.81:0/4282252485 >> 172.21.15.81:0/4282252485 conn(0x7f7dec0f1c70 msgr2=0x7f7dec0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:34:48.905 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:48.898+0000 7f7df1efe700 1 -- 172.21.15.81:0/4282252485 shutdown_connections 2023-11-15T06:34:48.905 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:48.898+0000 7f7df1efe700 1 -- 172.21.15.81:0/4282252485 wait complete. 2023-11-15T06:34:48.906 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:34:48.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:48 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:34:47.864170+0000 mgr.smithi062.rwurtk (mgr.14164) 275 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:34:49.918 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:49 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:34:48.896658+0000 mon.smithi062 (mon.0) 319 : audit [DBG] from='client.? 172.21.15.81:0/4282252485' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:34:50.381 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:34:50.381 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:34:51.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:50 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:34:49.864484+0000 mgr.smithi062.rwurtk (mgr.14164) 276 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:34:51.673 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:51.670+0000 7fd00a2a2700 1 -- 172.21.15.81:0/174733841 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd004060220 msgr2=0x7fd0040f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:51.674 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:51.670+0000 7fd00a2a2700 1 --2- 172.21.15.81:0/174733841 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd004060220 0x7fd0040f9150 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7fcff8009fb0 tx=0x7fcff8009b30).stop 2023-11-15T06:34:51.674 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:51.670+0000 7fd00a2a2700 1 -- 172.21.15.81:0/174733841 shutdown_connections 2023-11-15T06:34:51.674 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:51.670+0000 7fd00a2a2700 1 --2- 172.21.15.81:0/174733841 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd004060220 0x7fd0040f9150 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:51.674 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:51.670+0000 7fd00a2a2700 1 -- 172.21.15.81:0/174733841 >> 172.21.15.81:0/174733841 conn(0x7fd0040f0b70 msgr2=0x7fd0040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:34:51.674 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:51.670+0000 7fd00a2a2700 1 -- 172.21.15.81:0/174733841 shutdown_connections 2023-11-15T06:34:51.675 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:51.670+0000 7fd00a2a2700 1 -- 172.21.15.81:0/174733841 wait complete. 2023-11-15T06:34:51.675 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:51.670+0000 7fd00a2a2700 1 Processor -- start 2023-11-15T06:34:51.675 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:51.671+0000 7fd00a2a2700 1 -- start start 2023-11-15T06:34:51.675 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:51.671+0000 7fd00a2a2700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd004060220 0x7fd0041066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:34:51.675 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:51.671+0000 7fd00a2a2700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd004106bd0 con 0x7fd004060220 2023-11-15T06:34:51.676 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:51.671+0000 7fd003fff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd004060220 0x7fd0041066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:34:51.676 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:51.671+0000 7fd003fff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd004060220 0x7fd0041066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:45106/0 (socket says 172.21.15.81:45106) 2023-11-15T06:34:51.676 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:51.671+0000 7fd003fff700 1 -- 172.21.15.81:0/1192677651 learned_addr learned my addr 172.21.15.81:0/1192677651 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:34:51.676 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:51.671+0000 7fd003fff700 1 -- 172.21.15.81:0/1192677651 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcff8005040 con 0x7fd004060220 2023-11-15T06:34:51.676 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:51.671+0000 7fd003fff700 1 --2- 172.21.15.81:0/1192677651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd004060220 0x7fd0041066e0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7fcff8013510 tx=0x7fcff800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:34:51.678 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:51.672+0000 7fd000ff9700 1 -- 172.21.15.81:0/1192677651 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcff8009320 con 0x7fd004060220 2023-11-15T06:34:51.678 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:51.672+0000 7fd00a2a2700 1 -- 172.21.15.81:0/1192677651 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd004106dd0 con 0x7fd004060220 2023-11-15T06:34:51.678 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:51.672+0000 7fd000ff9700 1 -- 172.21.15.81:0/1192677651 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcff8013f00 con 0x7fd004060220 2023-11-15T06:34:51.678 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:51.672+0000 7fd000ff9700 1 -- 172.21.15.81:0/1192677651 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcff8014dc0 con 0x7fd004060220 2023-11-15T06:34:51.679 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:51.672+0000 7fd00a2a2700 1 -- 172.21.15.81:0/1192677651 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd004107260 con 0x7fd004060220 2023-11-15T06:34:51.679 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:51.672+0000 7fd000ff9700 1 -- 172.21.15.81:0/1192677651 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcff80095f0 con 0x7fd004060220 2023-11-15T06:34:51.679 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:51.672+0000 7fd00a2a2700 1 -- 172.21.15.81:0/1192677651 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd004045ad0 con 0x7fd004060220 2023-11-15T06:34:51.679 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:51.673+0000 7fd000ff9700 1 --2- 172.21.15.81:0/1192677651 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fcfec02ff60 0x7fcfec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:34:51.679 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:51.673+0000 7fd000ff9700 1 -- 172.21.15.81:0/1192677651 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcff80410c0 con 0x7fd004060220 2023-11-15T06:34:51.680 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:51.673+0000 7fd0037fe700 1 --2- 172.21.15.81:0/1192677651 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fcfec02ff60 0x7fcfec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:34:51.680 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:51.673+0000 7fd0037fe700 1 --2- 172.21.15.81:0/1192677651 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fcfec02ff60 0x7fcfec032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fcff40096a0 tx=0x7fcff4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:34:51.680 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:51.677+0000 7fd000ff9700 1 -- 172.21.15.81:0/1192677651 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcff8014420 con 0x7fd004060220 2023-11-15T06:34:51.884 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:51.882+0000 7fd00a2a2700 1 -- 172.21.15.81:0/1192677651 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd004108110 con 0x7fd004060220 2023-11-15T06:34:51.885 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:51.883+0000 7fd000ff9700 1 -- 172.21.15.81:0/1192677651 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcff8010230 con 0x7fd004060220 2023-11-15T06:34:51.885 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:34:51.885 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:34:51.887 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:51.884+0000 7fd00a2a2700 1 -- 172.21.15.81:0/1192677651 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fcfec02ff60 msgr2=0x7fcfec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:51.887 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:51.884+0000 7fd00a2a2700 1 --2- 172.21.15.81:0/1192677651 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fcfec02ff60 0x7fcfec032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fcff40096a0 tx=0x7fcff4006b40).stop 2023-11-15T06:34:51.887 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:51.884+0000 7fd00a2a2700 1 -- 172.21.15.81:0/1192677651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd004060220 msgr2=0x7fd0041066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:51.887 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:51.884+0000 7fd00a2a2700 1 --2- 172.21.15.81:0/1192677651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd004060220 0x7fd0041066e0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7fcff8013510 tx=0x7fcff800b040).stop 2023-11-15T06:34:51.888 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:51.884+0000 7fd00a2a2700 1 -- 172.21.15.81:0/1192677651 shutdown_connections 2023-11-15T06:34:51.888 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:51.884+0000 7fd00a2a2700 1 --2- 172.21.15.81:0/1192677651 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fcfec02ff60 0x7fcfec032420 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:51.888 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:51.884+0000 7fd00a2a2700 1 --2- 172.21.15.81:0/1192677651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd004060220 0x7fd0041066e0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:51.888 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:51.884+0000 7fd00a2a2700 1 -- 172.21.15.81:0/1192677651 >> 172.21.15.81:0/1192677651 conn(0x7fd0040f0b70 msgr2=0x7fd0040f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:34:51.888 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:51.885+0000 7fd00a2a2700 1 -- 172.21.15.81:0/1192677651 shutdown_connections 2023-11-15T06:34:51.888 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:51.885+0000 7fd00a2a2700 1 -- 172.21.15.81:0/1192677651 wait complete. 2023-11-15T06:34:51.889 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:34:52.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:51 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:34:51.883482+0000 mon.smithi062 (mon.0) 320 : audit [DBG] from='client.? 172.21.15.81:0/1192677651' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:34:53.360 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:34:53.360 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:34:53.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:52 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:34:51.864796+0000 mgr.smithi062.rwurtk (mgr.14164) 277 : cluster [DBG] pgmap v263: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:34:54.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:53 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:34:53.926059+0000 mon.smithi062 (mon.0) 321 : audit [INF] from='mgr.14164 172.21.15.62:0/147479783' entity='mgr.smithi062.rwurtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi062.rwurtk/trash_purge_schedule"}]: dispatch 2023-11-15T06:34:54.419 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:53 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:34:53.926290+0000 mon.smithi062 (mon.0) 322 : audit [INF] from='mgr.14164 172.21.15.62:0/147479783' entity='mgr.smithi062.rwurtk' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi062.rwurtk/mirror_snapshot_schedule"}]: dispatch 2023-11-15T06:34:54.676 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:54.673+0000 7f958e5d4700 1 -- 172.21.15.81:0/3316405739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95880f5410 msgr2=0x7f95880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:54.676 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:54.673+0000 7f958e5d4700 1 --2- 172.21.15.81:0/3316405739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95880f5410 0x7f95880f5830 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f957c009fb0 tx=0x7f957c009b30).stop 2023-11-15T06:34:54.676 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:54.673+0000 7f958e5d4700 1 -- 172.21.15.81:0/3316405739 shutdown_connections 2023-11-15T06:34:54.677 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:54.673+0000 7f958e5d4700 1 --2- 172.21.15.81:0/3316405739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95880f5410 0x7f95880f5830 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:54.677 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:54.673+0000 7f958e5d4700 1 -- 172.21.15.81:0/3316405739 >> 172.21.15.81:0/3316405739 conn(0x7f95880f0b70 msgr2=0x7f95880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:34:54.677 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:54.673+0000 7f958e5d4700 1 -- 172.21.15.81:0/3316405739 shutdown_connections 2023-11-15T06:34:54.677 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:54.673+0000 7f958e5d4700 1 -- 172.21.15.81:0/3316405739 wait complete. 2023-11-15T06:34:54.678 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:54.674+0000 7f958e5d4700 1 Processor -- start 2023-11-15T06:34:54.678 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:54.674+0000 7f958e5d4700 1 -- start start 2023-11-15T06:34:54.679 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:54.674+0000 7f958e5d4700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95880f5410 0x7f9588105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:34:54.680 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:54.674+0000 7f958e5d4700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9588105b20 con 0x7f95880f5410 2023-11-15T06:34:54.680 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:54.674+0000 7f9587fff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95880f5410 0x7f9588105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:34:54.680 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:54.674+0000 7f9587fff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95880f5410 0x7f9588105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:54614/0 (socket says 172.21.15.81:54614) 2023-11-15T06:34:54.680 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:54.674+0000 7f9587fff700 1 -- 172.21.15.81:0/4132039836 learned_addr learned my addr 172.21.15.81:0/4132039836 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:34:54.681 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:54.675+0000 7f9587fff700 1 -- 172.21.15.81:0/4132039836 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f957c005040 con 0x7f95880f5410 2023-11-15T06:34:54.681 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:54.675+0000 7f9587fff700 1 --2- 172.21.15.81:0/4132039836 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95880f5410 0x7f9588105630 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f957c013510 tx=0x7f957c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:34:54.681 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:54.675+0000 7f9584ff9700 1 -- 172.21.15.81:0/4132039836 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f957c009320 con 0x7f95880f5410 2023-11-15T06:34:54.681 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:54.675+0000 7f958e5d4700 1 -- 172.21.15.81:0/4132039836 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9588105d20 con 0x7f95880f5410 2023-11-15T06:34:54.681 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:54.675+0000 7f9584ff9700 1 -- 172.21.15.81:0/4132039836 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f957c013f00 con 0x7f95880f5410 2023-11-15T06:34:54.682 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:54.675+0000 7f9584ff9700 1 -- 172.21.15.81:0/4132039836 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f957c014d90 con 0x7f95880f5410 2023-11-15T06:34:54.682 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:54.675+0000 7f958e5d4700 1 -- 172.21.15.81:0/4132039836 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95881061b0 con 0x7f95880f5410 2023-11-15T06:34:54.682 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:54.676+0000 7f9584ff9700 1 -- 172.21.15.81:0/4132039836 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f957c0095f0 con 0x7f95880f5410 2023-11-15T06:34:54.682 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:54.676+0000 7f958e5d4700 1 -- 172.21.15.81:0/4132039836 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9588045ad0 con 0x7f95880f5410 2023-11-15T06:34:54.683 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:54.676+0000 7f9584ff9700 1 --2- 172.21.15.81:0/4132039836 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f957002ff60 0x7f9570032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:34:54.683 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:54.676+0000 7f9584ff9700 1 -- 172.21.15.81:0/4132039836 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f957c040fc0 con 0x7f95880f5410 2023-11-15T06:34:54.683 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:54.676+0000 7f95877fe700 1 --2- 172.21.15.81:0/4132039836 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f957002ff60 0x7f9570032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:34:54.683 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:54.677+0000 7f95877fe700 1 --2- 172.21.15.81:0/4132039836 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f957002ff60 0x7f9570032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f95780096a0 tx=0x7f9578006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:34:54.684 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:54.681+0000 7f9584ff9700 1 -- 172.21.15.81:0/4132039836 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f957c014420 con 0x7f95880f5410 2023-11-15T06:34:54.890 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:54.887+0000 7f958e5d4700 1 -- 172.21.15.81:0/4132039836 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f95880fae60 con 0x7f95880f5410 2023-11-15T06:34:54.891 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:54.888+0000 7f9584ff9700 1 -- 172.21.15.81:0/4132039836 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f957c010230 con 0x7f95880f5410 2023-11-15T06:34:54.891 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:34:54.891 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:34:54.892 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:54.890+0000 7f958e5d4700 1 -- 172.21.15.81:0/4132039836 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f957002ff60 msgr2=0x7f9570032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:54.892 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:54.890+0000 7f958e5d4700 1 --2- 172.21.15.81:0/4132039836 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f957002ff60 0x7f9570032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f95780096a0 tx=0x7f9578006b40).stop 2023-11-15T06:34:54.892 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:54.890+0000 7f958e5d4700 1 -- 172.21.15.81:0/4132039836 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95880f5410 msgr2=0x7f9588105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:54.893 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:54.890+0000 7f958e5d4700 1 --2- 172.21.15.81:0/4132039836 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95880f5410 0x7f9588105630 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f957c013510 tx=0x7f957c00b040).stop 2023-11-15T06:34:54.893 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:54.890+0000 7f958e5d4700 1 -- 172.21.15.81:0/4132039836 shutdown_connections 2023-11-15T06:34:54.893 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:54.890+0000 7f958e5d4700 1 --2- 172.21.15.81:0/4132039836 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f957002ff60 0x7f9570032420 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:54.893 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:54.890+0000 7f958e5d4700 1 --2- 172.21.15.81:0/4132039836 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95880f5410 0x7f9588105630 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:54.893 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:54.890+0000 7f958e5d4700 1 -- 172.21.15.81:0/4132039836 >> 172.21.15.81:0/4132039836 conn(0x7f95880f0b70 msgr2=0x7f95880f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:34:54.894 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:54.890+0000 7f958e5d4700 1 -- 172.21.15.81:0/4132039836 shutdown_connections 2023-11-15T06:34:54.894 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:54.890+0000 7f958e5d4700 1 -- 172.21.15.81:0/4132039836 wait complete. 2023-11-15T06:34:54.894 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:34:55.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:54 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:34:53.865216+0000 mgr.smithi062.rwurtk (mgr.14164) 278 : cluster [DBG] pgmap v264: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:34:55.419 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:54 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:34:54.889100+0000 mon.smithi062 (mon.0) 323 : audit [DBG] from='client.? 172.21.15.81:0/4132039836' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:34:56.325 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:34:56.326 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:34:57.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:56 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:34:55.865627+0000 mgr.smithi062.rwurtk (mgr.14164) 279 : cluster [DBG] pgmap v265: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:34:57.604 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:57.601+0000 7f3eaace7700 1 -- 172.21.15.81:0/2296024440 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ea40f6510 msgr2=0x7f3ea40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:57.605 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:57.601+0000 7f3eaace7700 1 --2- 172.21.15.81:0/2296024440 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ea40f6510 0x7f3ea40f6930 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f3e98009fb0 tx=0x7f3e98009b30).stop 2023-11-15T06:34:57.605 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:57.601+0000 7f3eaace7700 1 -- 172.21.15.81:0/2296024440 shutdown_connections 2023-11-15T06:34:57.605 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:57.601+0000 7f3eaace7700 1 --2- 172.21.15.81:0/2296024440 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ea40f6510 0x7f3ea40f6930 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:57.606 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:57.601+0000 7f3eaace7700 1 -- 172.21.15.81:0/2296024440 >> 172.21.15.81:0/2296024440 conn(0x7f3ea40f1c70 msgr2=0x7f3ea40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:34:57.606 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:57.601+0000 7f3eaace7700 1 -- 172.21.15.81:0/2296024440 shutdown_connections 2023-11-15T06:34:57.606 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:57.601+0000 7f3eaace7700 1 -- 172.21.15.81:0/2296024440 wait complete. 2023-11-15T06:34:57.606 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:57.601+0000 7f3eaace7700 1 Processor -- start 2023-11-15T06:34:57.606 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:57.601+0000 7f3eaace7700 1 -- start start 2023-11-15T06:34:57.607 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:57.602+0000 7f3eaace7700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ea40f6510 0x7f3ea4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:34:57.607 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:57.602+0000 7f3eaace7700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea4106c20 con 0x7f3ea40f6510 2023-11-15T06:34:57.608 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:57.602+0000 7f3ea8a83700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ea40f6510 0x7f3ea4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:34:57.609 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:57.602+0000 7f3ea8a83700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ea40f6510 0x7f3ea4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:54634/0 (socket says 172.21.15.81:54634) 2023-11-15T06:34:57.609 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:57.602+0000 7f3ea8a83700 1 -- 172.21.15.81:0/1570775740 learned_addr learned my addr 172.21.15.81:0/1570775740 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:34:57.609 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:57.602+0000 7f3ea8a83700 1 -- 172.21.15.81:0/1570775740 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e98005040 con 0x7f3ea40f6510 2023-11-15T06:34:57.610 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:57.602+0000 7f3ea8a83700 1 --2- 172.21.15.81:0/1570775740 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ea40f6510 0x7f3ea4106730 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f3e98013510 tx=0x7f3e9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:34:57.610 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:57.603+0000 7f3ea17fa700 1 -- 172.21.15.81:0/1570775740 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e98009320 con 0x7f3ea40f6510 2023-11-15T06:34:57.610 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:57.603+0000 7f3eaace7700 1 -- 172.21.15.81:0/1570775740 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ea4106e20 con 0x7f3ea40f6510 2023-11-15T06:34:57.610 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:57.603+0000 7f3ea17fa700 1 -- 172.21.15.81:0/1570775740 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3e98013f00 con 0x7f3ea40f6510 2023-11-15T06:34:57.610 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:57.603+0000 7f3ea17fa700 1 -- 172.21.15.81:0/1570775740 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e98014d90 con 0x7f3ea40f6510 2023-11-15T06:34:57.611 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:57.603+0000 7f3eaace7700 1 -- 172.21.15.81:0/1570775740 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ea41072b0 con 0x7f3ea40f6510 2023-11-15T06:34:57.611 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:57.603+0000 7f3ea17fa700 1 -- 172.21.15.81:0/1570775740 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3e980263a0 con 0x7f3ea40f6510 2023-11-15T06:34:57.611 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:57.604+0000 7f3ea17fa700 1 --2- 172.21.15.81:0/1570775740 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f3e9402ffb0 0x7f3e94032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:34:57.611 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:57.604+0000 7f3ea17fa700 1 -- 172.21.15.81:0/1570775740 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3e98041380 con 0x7f3ea40f6510 2023-11-15T06:34:57.611 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:57.604+0000 7f3eaace7700 1 -- 172.21.15.81:0/1570775740 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ea4045ad0 con 0x7f3ea40f6510 2023-11-15T06:34:57.612 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:57.604+0000 7f3ea3fff700 1 --2- 172.21.15.81:0/1570775740 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f3e9402ffb0 0x7f3e94032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:34:57.612 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:57.604+0000 7f3ea3fff700 1 --2- 172.21.15.81:0/1570775740 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f3e9402ffb0 0x7f3e94032470 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f3e900096a0 tx=0x7f3e90006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:34:57.616 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:57.609+0000 7f3ea17fa700 1 -- 172.21.15.81:0/1570775740 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3e98012920 con 0x7f3ea40f6510 2023-11-15T06:34:57.814 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:57.811+0000 7f3eaace7700 1 -- 172.21.15.81:0/1570775740 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3ea40fc050 con 0x7f3ea40f6510 2023-11-15T06:34:57.815 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:57.813+0000 7f3ea17fa700 1 -- 172.21.15.81:0/1570775740 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3e9802b030 con 0x7f3ea40f6510 2023-11-15T06:34:57.815 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:34:57.816 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:34:57.817 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:57.813+0000 7f3eaace7700 1 -- 172.21.15.81:0/1570775740 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f3e9402ffb0 msgr2=0x7f3e94032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:57.817 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:57.813+0000 7f3eaace7700 1 --2- 172.21.15.81:0/1570775740 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f3e9402ffb0 0x7f3e94032470 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f3e900096a0 tx=0x7f3e90006b40).stop 2023-11-15T06:34:57.817 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:57.813+0000 7f3eaace7700 1 -- 172.21.15.81:0/1570775740 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ea40f6510 msgr2=0x7f3ea4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:34:57.817 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:57.813+0000 7f3eaace7700 1 --2- 172.21.15.81:0/1570775740 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ea40f6510 0x7f3ea4106730 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f3e98013510 tx=0x7f3e9800b040).stop 2023-11-15T06:34:57.818 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:57.814+0000 7f3eaace7700 1 -- 172.21.15.81:0/1570775740 shutdown_connections 2023-11-15T06:34:57.818 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:57.814+0000 7f3eaace7700 1 --2- 172.21.15.81:0/1570775740 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f3e9402ffb0 0x7f3e94032470 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:57.818 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:57.814+0000 7f3eaace7700 1 --2- 172.21.15.81:0/1570775740 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3ea40f6510 0x7f3ea4106730 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:34:57.818 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:57.814+0000 7f3eaace7700 1 -- 172.21.15.81:0/1570775740 >> 172.21.15.81:0/1570775740 conn(0x7f3ea40f1c70 msgr2=0x7f3ea40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:34:57.818 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:57.814+0000 7f3eaace7700 1 -- 172.21.15.81:0/1570775740 shutdown_connections 2023-11-15T06:34:57.819 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:34:57.814+0000 7f3eaace7700 1 -- 172.21.15.81:0/1570775740 wait complete. 2023-11-15T06:34:57.819 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:34:59.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:58 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:34:57.813815+0000 mon.smithi062 (mon.0) 324 : audit [DBG] from='client.? 172.21.15.81:0/1570775740' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:34:59.169 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:34:58 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:34:57.865995+0000 mgr.smithi062.rwurtk (mgr.14164) 280 : cluster [DBG] pgmap v266: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:34:59.232 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:34:59.233 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:35:00.431 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:00.427+0000 7f8fe3fff700 1 -- 172.21.15.81:0/100812884 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe4060140 msgr2=0x7f8fe40f8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:35:00.431 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:00.427+0000 7f8fe3fff700 1 --2- 172.21.15.81:0/100812884 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe4060140 0x7f8fe40f8fc0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f8fd4009fb0 tx=0x7f8fd4009b30).stop 2023-11-15T06:35:00.432 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:00.427+0000 7f8fe3fff700 1 -- 172.21.15.81:0/100812884 shutdown_connections 2023-11-15T06:35:00.432 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:00.427+0000 7f8fe3fff700 1 --2- 172.21.15.81:0/100812884 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe4060140 0x7f8fe40f8fc0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:35:00.432 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:00.427+0000 7f8fe3fff700 1 -- 172.21.15.81:0/100812884 >> 172.21.15.81:0/100812884 conn(0x7f8fe40f09a0 msgr2=0x7f8fe40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:35:00.432 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:00.427+0000 7f8fe3fff700 1 -- 172.21.15.81:0/100812884 shutdown_connections 2023-11-15T06:35:00.432 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:00.427+0000 7f8fe3fff700 1 -- 172.21.15.81:0/100812884 wait complete. 2023-11-15T06:35:00.433 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:00.427+0000 7f8fe3fff700 1 Processor -- start 2023-11-15T06:35:00.433 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:00.427+0000 7f8fe3fff700 1 -- start start 2023-11-15T06:35:00.433 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:00.428+0000 7f8fe3fff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe4060140 0x7f8fe40f6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:35:00.433 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:00.428+0000 7f8fe3fff700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe40f4ec0 con 0x7f8fe4060140 2023-11-15T06:35:00.433 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:00.428+0000 7f8fe2ffd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe4060140 0x7f8fe40f6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:35:00.434 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:00.428+0000 7f8fe2ffd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe4060140 0x7f8fe40f6a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:54650/0 (socket says 172.21.15.81:54650) 2023-11-15T06:35:00.434 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:00.428+0000 7f8fe2ffd700 1 -- 172.21.15.81:0/808658814 learned_addr learned my addr 172.21.15.81:0/808658814 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:35:00.434 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:00.428+0000 7f8fe2ffd700 1 -- 172.21.15.81:0/808658814 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd4005040 con 0x7f8fe4060140 2023-11-15T06:35:00.434 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:00.428+0000 7f8fe2ffd700 1 --2- 172.21.15.81:0/808658814 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe4060140 0x7f8fe40f6a40 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f8fd4013510 tx=0x7f8fd400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:35:00.435 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:00.428+0000 7f8fcb7fe700 1 -- 172.21.15.81:0/808658814 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8fd4009320 con 0x7f8fe4060140 2023-11-15T06:35:00.436 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:00.428+0000 7f8fe3fff700 1 -- 172.21.15.81:0/808658814 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fe40f5060 con 0x7f8fe4060140 2023-11-15T06:35:00.436 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:00.428+0000 7f8fcb7fe700 1 -- 172.21.15.81:0/808658814 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8fd4013f00 con 0x7f8fe4060140 2023-11-15T06:35:00.436 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:00.428+0000 7f8fcb7fe700 1 -- 172.21.15.81:0/808658814 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8fd4014d90 con 0x7f8fe4060140 2023-11-15T06:35:00.436 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:00.428+0000 7f8fe3fff700 1 -- 172.21.15.81:0/808658814 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fe40f54f0 con 0x7f8fe4060140 2023-11-15T06:35:00.437 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:00.429+0000 7f8fcb7fe700 1 -- 172.21.15.81:0/808658814 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8fd40095f0 con 0x7f8fe4060140 2023-11-15T06:35:00.437 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:00.429+0000 7f8fe3fff700 1 -- 172.21.15.81:0/808658814 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fe4059100 con 0x7f8fe4060140 2023-11-15T06:35:00.437 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:00.429+0000 7f8fcb7fe700 1 --2- 172.21.15.81:0/808658814 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f8fcc02ff60 0x7f8fcc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:35:00.437 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:00.429+0000 7f8fcb7fe700 1 -- 172.21.15.81:0/808658814 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8fd4041080 con 0x7f8fe4060140 2023-11-15T06:35:00.437 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:00.430+0000 7f8fe27fc700 1 --2- 172.21.15.81:0/808658814 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f8fcc02ff60 0x7f8fcc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:35:00.438 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:00.430+0000 7f8fe27fc700 1 --2- 172.21.15.81:0/808658814 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f8fcc02ff60 0x7f8fcc032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f8fd80096a0 tx=0x7f8fd8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:35:00.438 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:00.434+0000 7f8fcb7fe700 1 -- 172.21.15.81:0/808658814 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8fd4014420 con 0x7f8fe4060140 2023-11-15T06:35:00.640 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:00.637+0000 7f8fe3fff700 1 -- 172.21.15.81:0/808658814 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8fe40fbce0 con 0x7f8fe4060140 2023-11-15T06:35:00.641 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:00.638+0000 7f8fcb7fe700 1 -- 172.21.15.81:0/808658814 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8fd4010230 con 0x7f8fe4060140 2023-11-15T06:35:00.641 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:35:00.642 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:35:00.643 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:00.640+0000 7f8fe3fff700 1 -- 172.21.15.81:0/808658814 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f8fcc02ff60 msgr2=0x7f8fcc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:35:00.643 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:00.640+0000 7f8fe3fff700 1 --2- 172.21.15.81:0/808658814 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f8fcc02ff60 0x7f8fcc032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f8fd80096a0 tx=0x7f8fd8006b40).stop 2023-11-15T06:35:00.643 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:00.640+0000 7f8fe3fff700 1 -- 172.21.15.81:0/808658814 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe4060140 msgr2=0x7f8fe40f6a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:35:00.644 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:00.640+0000 7f8fe3fff700 1 --2- 172.21.15.81:0/808658814 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe4060140 0x7f8fe40f6a40 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f8fd4013510 tx=0x7f8fd400b040).stop 2023-11-15T06:35:00.644 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:00.640+0000 7f8fe3fff700 1 -- 172.21.15.81:0/808658814 shutdown_connections 2023-11-15T06:35:00.644 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:00.640+0000 7f8fe3fff700 1 --2- 172.21.15.81:0/808658814 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f8fcc02ff60 0x7f8fcc032420 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:35:00.644 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:00.640+0000 7f8fe3fff700 1 --2- 172.21.15.81:0/808658814 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe4060140 0x7f8fe40f6a40 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:35:00.644 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:00.640+0000 7f8fe3fff700 1 -- 172.21.15.81:0/808658814 >> 172.21.15.81:0/808658814 conn(0x7f8fe40f09a0 msgr2=0x7f8fe40f14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:35:00.645 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:00.640+0000 7f8fe3fff700 1 -- 172.21.15.81:0/808658814 shutdown_connections 2023-11-15T06:35:00.645 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:00.640+0000 7f8fe3fff700 1 -- 172.21.15.81:0/808658814 wait complete. 2023-11-15T06:35:00.645 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:35:01.419 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:35:00 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:34:59.866312+0000 mgr.smithi062.rwurtk (mgr.14164) 281 : cluster [DBG] pgmap v267: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:35:01.419 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:35:00 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:35:00.640010+0000 mon.smithi062 (mon.0) 325 : audit [DBG] from='client.? 172.21.15.81:0/808658814' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:35:02.106 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:35:02.107 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:35:03.383 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:03.379+0000 7f645b846700 1 -- 172.21.15.81:0/3786031193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64540f5410 msgr2=0x7f64540f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:35:03.384 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:03.379+0000 7f645b846700 1 --2- 172.21.15.81:0/3786031193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64540f5410 0x7f64540f5830 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f6444009fb0 tx=0x7f6444009b30).stop 2023-11-15T06:35:03.384 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:03.379+0000 7f645b846700 1 -- 172.21.15.81:0/3786031193 shutdown_connections 2023-11-15T06:35:03.384 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:03.379+0000 7f645b846700 1 --2- 172.21.15.81:0/3786031193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64540f5410 0x7f64540f5830 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:35:03.384 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:03.379+0000 7f645b846700 1 -- 172.21.15.81:0/3786031193 >> 172.21.15.81:0/3786031193 conn(0x7f64540f0b70 msgr2=0x7f64540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:35:03.384 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:03.380+0000 7f645b846700 1 -- 172.21.15.81:0/3786031193 shutdown_connections 2023-11-15T06:35:03.385 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:03.380+0000 7f645b846700 1 -- 172.21.15.81:0/3786031193 wait complete. 2023-11-15T06:35:03.385 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:03.380+0000 7f645b846700 1 Processor -- start 2023-11-15T06:35:03.385 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:03.380+0000 7f645b846700 1 -- start start 2023-11-15T06:35:03.385 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:03.380+0000 7f645b846700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64540f5410 0x7f6454106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:35:03.386 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:03.380+0000 7f645b846700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6454106bf0 con 0x7f64540f5410 2023-11-15T06:35:03.387 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:03.380+0000 7f64595e2700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64540f5410 0x7f6454106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:35:03.388 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:03.381+0000 7f64595e2700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64540f5410 0x7f6454106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:55162/0 (socket says 172.21.15.81:55162) 2023-11-15T06:35:03.388 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:03.381+0000 7f64595e2700 1 -- 172.21.15.81:0/2872314941 learned_addr learned my addr 172.21.15.81:0/2872314941 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:35:03.388 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:03.381+0000 7f64595e2700 1 -- 172.21.15.81:0/2872314941 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6444005040 con 0x7f64540f5410 2023-11-15T06:35:03.388 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:03.381+0000 7f64595e2700 1 --2- 172.21.15.81:0/2872314941 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64540f5410 0x7f6454106700 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f6444013510 tx=0x7f644400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:35:03.388 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:03.381+0000 7f6449ffb700 1 -- 172.21.15.81:0/2872314941 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6444009320 con 0x7f64540f5410 2023-11-15T06:35:03.389 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:03.381+0000 7f645b846700 1 -- 172.21.15.81:0/2872314941 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6454106df0 con 0x7f64540f5410 2023-11-15T06:35:03.389 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:03.381+0000 7f6449ffb700 1 -- 172.21.15.81:0/2872314941 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6444013f00 con 0x7f64540f5410 2023-11-15T06:35:03.389 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:03.381+0000 7f6449ffb700 1 -- 172.21.15.81:0/2872314941 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6444014d90 con 0x7f64540f5410 2023-11-15T06:35:03.389 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:03.381+0000 7f645b846700 1 -- 172.21.15.81:0/2872314941 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6454107280 con 0x7f64540f5410 2023-11-15T06:35:03.389 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:03.382+0000 7f6449ffb700 1 -- 172.21.15.81:0/2872314941 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6444014420 con 0x7f64540f5410 2023-11-15T06:35:03.390 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:03.382+0000 7f645b846700 1 -- 172.21.15.81:0/2872314941 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6454045ad0 con 0x7f64540f5410 2023-11-15T06:35:03.390 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:03.382+0000 7f6449ffb700 1 --2- 172.21.15.81:0/2872314941 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f644002ff60 0x7f6440032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:35:03.390 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:03.382+0000 7f6449ffb700 1 -- 172.21.15.81:0/2872314941 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6444041110 con 0x7f64540f5410 2023-11-15T06:35:03.390 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:03.382+0000 7f6458de1700 1 --2- 172.21.15.81:0/2872314941 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f644002ff60 0x7f6440032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:35:03.390 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:03.383+0000 7f6458de1700 1 --2- 172.21.15.81:0/2872314941 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f644002ff60 0x7f6440032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f64500096a0 tx=0x7f6450006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:35:03.391 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:03.387+0000 7f6449ffb700 1 -- 172.21.15.81:0/2872314941 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f64440095f0 con 0x7f64540f5410 2023-11-15T06:35:03.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:35:02 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:35:01.866624+0000 mgr.smithi062.rwurtk (mgr.14164) 282 : cluster [DBG] pgmap v268: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:35:03.596 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:03.592+0000 7f645b846700 1 -- 172.21.15.81:0/2872314941 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f64540fbf80 con 0x7f64540f5410 2023-11-15T06:35:03.596 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:03.593+0000 7f6449ffb700 1 -- 172.21.15.81:0/2872314941 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f644401db70 con 0x7f64540f5410 2023-11-15T06:35:03.597 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:35:03.597 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:35:03.598 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:03.594+0000 7f645b846700 1 -- 172.21.15.81:0/2872314941 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f644002ff60 msgr2=0x7f6440032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:35:03.598 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:03.594+0000 7f645b846700 1 --2- 172.21.15.81:0/2872314941 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f644002ff60 0x7f6440032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f64500096a0 tx=0x7f6450006b40).stop 2023-11-15T06:35:03.598 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:03.595+0000 7f645b846700 1 -- 172.21.15.81:0/2872314941 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64540f5410 msgr2=0x7f6454106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:35:03.599 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:03.595+0000 7f645b846700 1 --2- 172.21.15.81:0/2872314941 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64540f5410 0x7f6454106700 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f6444013510 tx=0x7f644400b040).stop 2023-11-15T06:35:03.599 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:03.595+0000 7f645b846700 1 -- 172.21.15.81:0/2872314941 shutdown_connections 2023-11-15T06:35:03.599 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:03.595+0000 7f645b846700 1 --2- 172.21.15.81:0/2872314941 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7f644002ff60 0x7f6440032420 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:35:03.599 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:03.595+0000 7f645b846700 1 --2- 172.21.15.81:0/2872314941 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64540f5410 0x7f6454106700 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:35:03.599 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:03.595+0000 7f645b846700 1 -- 172.21.15.81:0/2872314941 >> 172.21.15.81:0/2872314941 conn(0x7f64540f0b70 msgr2=0x7f64540f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:35:03.600 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:03.595+0000 7f645b846700 1 -- 172.21.15.81:0/2872314941 shutdown_connections 2023-11-15T06:35:03.600 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:03.595+0000 7f645b846700 1 -- 172.21.15.81:0/2872314941 wait complete. 2023-11-15T06:35:03.600 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:35:04.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:35:03 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:35:03.594979+0000 mon.smithi062 (mon.0) 326 : audit [DBG] from='client.? 172.21.15.81:0/2872314941' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:35:04.998 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:35:04.998 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:35:05.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:35:04 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:35:03.866970+0000 mgr.smithi062.rwurtk (mgr.14164) 283 : cluster [DBG] pgmap v269: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:35:06.214 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:06.210+0000 7fd7d93d8700 1 -- 172.21.15.81:0/1734405592 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7d40f5410 msgr2=0x7fd7d40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:35:06.215 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:06.210+0000 7fd7d93d8700 1 --2- 172.21.15.81:0/1734405592 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7d40f5410 0x7fd7d40f5830 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7fd7c4009fb0 tx=0x7fd7c4009b30).stop 2023-11-15T06:35:06.215 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:06.210+0000 7fd7d93d8700 1 -- 172.21.15.81:0/1734405592 shutdown_connections 2023-11-15T06:35:06.215 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:06.210+0000 7fd7d93d8700 1 --2- 172.21.15.81:0/1734405592 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7d40f5410 0x7fd7d40f5830 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:35:06.215 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:06.210+0000 7fd7d93d8700 1 -- 172.21.15.81:0/1734405592 >> 172.21.15.81:0/1734405592 conn(0x7fd7d40f0b70 msgr2=0x7fd7d40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:35:06.216 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:06.211+0000 7fd7d93d8700 1 -- 172.21.15.81:0/1734405592 shutdown_connections 2023-11-15T06:35:06.216 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:06.211+0000 7fd7d93d8700 1 -- 172.21.15.81:0/1734405592 wait complete. 2023-11-15T06:35:06.216 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:06.211+0000 7fd7d93d8700 1 Processor -- start 2023-11-15T06:35:06.216 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:06.211+0000 7fd7d93d8700 1 -- start start 2023-11-15T06:35:06.216 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:06.211+0000 7fd7d93d8700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7d40f5410 0x7fd7d4109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:35:06.217 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:06.211+0000 7fd7d93d8700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d4060240 con 0x7fd7d40f5410 2023-11-15T06:35:06.217 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:06.212+0000 7fd7d2ffd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7d40f5410 0x7fd7d4109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:35:06.217 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:06.212+0000 7fd7d2ffd700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7d40f5410 0x7fd7d4109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:55180/0 (socket says 172.21.15.81:55180) 2023-11-15T06:35:06.217 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:06.212+0000 7fd7d2ffd700 1 -- 172.21.15.81:0/3396358849 learned_addr learned my addr 172.21.15.81:0/3396358849 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:35:06.219 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:06.212+0000 7fd7d2ffd700 1 -- 172.21.15.81:0/3396358849 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c4005040 con 0x7fd7d40f5410 2023-11-15T06:35:06.220 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:06.212+0000 7fd7d2ffd700 1 --2- 172.21.15.81:0/3396358849 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7d40f5410 0x7fd7d4109400 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fd7c4013510 tx=0x7fd7c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:35:06.220 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:06.212+0000 7fd7bb7fe700 1 -- 172.21.15.81:0/3396358849 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7c4009320 con 0x7fd7d40f5410 2023-11-15T06:35:06.220 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:06.212+0000 7fd7d93d8700 1 -- 172.21.15.81:0/3396358849 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7d4060440 con 0x7fd7d40f5410 2023-11-15T06:35:06.220 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:06.212+0000 7fd7bb7fe700 1 -- 172.21.15.81:0/3396358849 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd7c4013f00 con 0x7fd7d40f5410 2023-11-15T06:35:06.221 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:06.212+0000 7fd7bb7fe700 1 -- 172.21.15.81:0/3396358849 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7c4014d90 con 0x7fd7d40f5410 2023-11-15T06:35:06.221 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:06.212+0000 7fd7d93d8700 1 -- 172.21.15.81:0/3396358849 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7d4109ab0 con 0x7fd7d40f5410 2023-11-15T06:35:06.221 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:06.213+0000 7fd7bb7fe700 1 -- 172.21.15.81:0/3396358849 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd7c40095f0 con 0x7fd7d40f5410 2023-11-15T06:35:06.221 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:06.213+0000 7fd7d93d8700 1 -- 172.21.15.81:0/3396358849 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7d4045ad0 con 0x7fd7d40f5410 2023-11-15T06:35:06.221 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:06.213+0000 7fd7bb7fe700 1 --2- 172.21.15.81:0/3396358849 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fd7bc02ffb0 0x7fd7bc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:35:06.222 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:06.213+0000 7fd7bb7fe700 1 -- 172.21.15.81:0/3396358849 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd7c40410a0 con 0x7fd7d40f5410 2023-11-15T06:35:06.222 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:06.213+0000 7fd7d27fc700 1 --2- 172.21.15.81:0/3396358849 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fd7bc02ffb0 0x7fd7bc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:35:06.222 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:06.214+0000 7fd7d27fc700 1 --2- 172.21.15.81:0/3396358849 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fd7bc02ffb0 0x7fd7bc032470 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fd7c80096a0 tx=0x7fd7c8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:35:06.223 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:06.218+0000 7fd7bb7fe700 1 -- 172.21.15.81:0/3396358849 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd7c406d2b0 con 0x7fd7d40f5410 2023-11-15T06:35:06.427 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:06.424+0000 7fd7d93d8700 1 -- 172.21.15.81:0/3396358849 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd7d40fae60 con 0x7fd7d40f5410 2023-11-15T06:35:06.428 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:06.425+0000 7fd7bb7fe700 1 -- 172.21.15.81:0/3396358849 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd7c4014420 con 0x7fd7d40f5410 2023-11-15T06:35:06.428 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:35:06.428 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:35:06.430 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:06.426+0000 7fd7d93d8700 1 -- 172.21.15.81:0/3396358849 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fd7bc02ffb0 msgr2=0x7fd7bc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:35:06.430 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:06.426+0000 7fd7d93d8700 1 --2- 172.21.15.81:0/3396358849 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fd7bc02ffb0 0x7fd7bc032470 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fd7c80096a0 tx=0x7fd7c8006b40).stop 2023-11-15T06:35:06.430 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:06.426+0000 7fd7d93d8700 1 -- 172.21.15.81:0/3396358849 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7d40f5410 msgr2=0x7fd7d4109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:35:06.430 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:06.426+0000 7fd7d93d8700 1 --2- 172.21.15.81:0/3396358849 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7d40f5410 0x7fd7d4109400 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fd7c4013510 tx=0x7fd7c400b040).stop 2023-11-15T06:35:06.431 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:06.426+0000 7fd7d93d8700 1 -- 172.21.15.81:0/3396358849 shutdown_connections 2023-11-15T06:35:06.431 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:06.426+0000 7fd7d93d8700 1 --2- 172.21.15.81:0/3396358849 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fd7bc02ffb0 0x7fd7bc032470 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:35:06.431 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:06.426+0000 7fd7d93d8700 1 --2- 172.21.15.81:0/3396358849 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7d40f5410 0x7fd7d4109400 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:35:06.432 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:06.426+0000 7fd7d93d8700 1 -- 172.21.15.81:0/3396358849 >> 172.21.15.81:0/3396358849 conn(0x7fd7d40f0b70 msgr2=0x7fd7d40f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:35:06.432 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:06.426+0000 7fd7d93d8700 1 -- 172.21.15.81:0/3396358849 shutdown_connections 2023-11-15T06:35:06.432 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:06.426+0000 7fd7d93d8700 1 -- 172.21.15.81:0/3396358849 wait complete. 2023-11-15T06:35:06.433 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:35:07.418 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:35:06 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:35:05.867420+0000 mgr.smithi062.rwurtk (mgr.14164) 284 : cluster [DBG] pgmap v270: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:35:07.419 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:35:06 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:35:06.426520+0000 mon.smithi062 (mon.0) 327 : audit [DBG] from='client.? 172.21.15.81:0/3396358849' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:35:07.884 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:35:07.885 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:35:09.140 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:09.135+0000 7fb6feb2c700 1 -- 172.21.15.81:0/2183509620 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb6f8060220 msgr2=0x7fb6f80f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:35:09.141 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:09.135+0000 7fb6feb2c700 1 --2- 172.21.15.81:0/2183509620 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb6f8060220 0x7fb6f80f9150 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7fb6ec009fb0 tx=0x7fb6ec009b30).stop 2023-11-15T06:35:09.141 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:09.135+0000 7fb6feb2c700 1 -- 172.21.15.81:0/2183509620 shutdown_connections 2023-11-15T06:35:09.141 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:09.135+0000 7fb6feb2c700 1 --2- 172.21.15.81:0/2183509620 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb6f8060220 0x7fb6f80f9150 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:35:09.141 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:09.135+0000 7fb6feb2c700 1 -- 172.21.15.81:0/2183509620 >> 172.21.15.81:0/2183509620 conn(0x7fb6f80f0b70 msgr2=0x7fb6f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:35:09.142 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:09.135+0000 7fb6feb2c700 1 -- 172.21.15.81:0/2183509620 shutdown_connections 2023-11-15T06:35:09.142 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:09.135+0000 7fb6feb2c700 1 -- 172.21.15.81:0/2183509620 wait complete. 2023-11-15T06:35:09.142 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:09.136+0000 7fb6feb2c700 1 Processor -- start 2023-11-15T06:35:09.142 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:09.136+0000 7fb6feb2c700 1 -- start start 2023-11-15T06:35:09.142 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:09.136+0000 7fb6feb2c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb6f8060220 0x7fb6f80f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:35:09.143 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:09.136+0000 7fb6feb2c700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f80f5050 con 0x7fb6f8060220 2023-11-15T06:35:09.143 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:09.137+0000 7fb6fc8c8700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb6f8060220 0x7fb6f80f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:35:09.143 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:09.137+0000 7fb6fc8c8700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb6f8060220 0x7fb6f80f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.81:55192/0 (socket says 172.21.15.81:55192) 2023-11-15T06:35:09.143 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:09.137+0000 7fb6fc8c8700 1 -- 172.21.15.81:0/536561225 learned_addr learned my addr 172.21.15.81:0/536561225 (peer_addr_for_me v2:172.21.15.81:0/0) 2023-11-15T06:35:09.143 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:09.137+0000 7fb6fc8c8700 1 -- 172.21.15.81:0/536561225 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6ec005040 con 0x7fb6f8060220 2023-11-15T06:35:09.145 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:09.137+0000 7fb6fc8c8700 1 --2- 172.21.15.81:0/536561225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb6f8060220 0x7fb6f80f6bd0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7fb6ec013510 tx=0x7fb6ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:35:09.145 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:09.137+0000 7fb6f57fa700 1 -- 172.21.15.81:0/536561225 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6ec009320 con 0x7fb6f8060220 2023-11-15T06:35:09.145 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:09.137+0000 7fb6feb2c700 1 -- 172.21.15.81:0/536561225 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6f80f51f0 con 0x7fb6f8060220 2023-11-15T06:35:09.146 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:09.137+0000 7fb6f57fa700 1 -- 172.21.15.81:0/536561225 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb6ec013f00 con 0x7fb6f8060220 2023-11-15T06:35:09.146 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:09.137+0000 7fb6f57fa700 1 -- 172.21.15.81:0/536561225 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6ec014dc0 con 0x7fb6f8060220 2023-11-15T06:35:09.146 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:09.137+0000 7fb6feb2c700 1 -- 172.21.15.81:0/536561225 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6f80f5680 con 0x7fb6f8060220 2023-11-15T06:35:09.147 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:09.138+0000 7fb6f57fa700 1 -- 172.21.15.81:0/536561225 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb6ec01d3d0 con 0x7fb6f8060220 2023-11-15T06:35:09.148 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:09.138+0000 7fb6f57fa700 1 --2- 172.21.15.81:0/536561225 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fb6e802ff60 0x7fb6e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:35:09.149 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:09.138+0000 7fb6f57fa700 1 -- 172.21.15.81:0/536561225 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb6ec0410c0 con 0x7fb6f8060220 2023-11-15T06:35:09.149 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:09.138+0000 7fb6feb2c700 1 -- 172.21.15.81:0/536561225 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6f8059100 con 0x7fb6f8060220 2023-11-15T06:35:09.149 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:09.138+0000 7fb6f7fff700 1 --2- 172.21.15.81:0/536561225 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fb6e802ff60 0x7fb6e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:35:09.150 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:09.139+0000 7fb6f7fff700 1 --2- 172.21.15.81:0/536561225 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fb6e802ff60 0x7fb6e8032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fb6e40096a0 tx=0x7fb6e4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:35:09.150 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:09.143+0000 7fb6f57fa700 1 -- 172.21.15.81:0/536561225 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb6ec012920 con 0x7fb6f8060220 2023-11-15T06:35:09.168 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:35:08 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: cluster 2023-11-15T06:35:07.867767+0000 mgr.smithi062.rwurtk (mgr.14164) 285 : cluster [DBG] pgmap v271: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:35:09.350 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:09.347+0000 7fb6feb2c700 1 -- 172.21.15.81:0/536561225 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb6f80f5d80 con 0x7fb6f8060220 2023-11-15T06:35:09.351 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:09.347+0000 7fb6f57fa700 1 -- 172.21.15.81:0/536561225 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb6ec02b020 con 0x7fb6f8060220 2023-11-15T06:35:09.351 INFO:teuthology.orchestra.run.smithi081.stdout: 2023-11-15T06:35:09.352 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"fsid":"a3c2da76-837f-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:24:57.422579Z","created":"2023-11-15T06:24:57.422579Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi062","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:35:09.352 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:09.349+0000 7fb6feb2c700 1 -- 172.21.15.81:0/536561225 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fb6e802ff60 msgr2=0x7fb6e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:35:09.353 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:09.349+0000 7fb6feb2c700 1 --2- 172.21.15.81:0/536561225 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fb6e802ff60 0x7fb6e8032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fb6e40096a0 tx=0x7fb6e4006b40).stop 2023-11-15T06:35:09.353 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:09.349+0000 7fb6feb2c700 1 -- 172.21.15.81:0/536561225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb6f8060220 msgr2=0x7fb6f80f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:35:09.353 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:09.349+0000 7fb6feb2c700 1 --2- 172.21.15.81:0/536561225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb6f8060220 0x7fb6f80f6bd0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7fb6ec013510 tx=0x7fb6ec00b040).stop 2023-11-15T06:35:09.353 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:09.349+0000 7fb6feb2c700 1 -- 172.21.15.81:0/536561225 shutdown_connections 2023-11-15T06:35:09.354 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:09.349+0000 7fb6feb2c700 1 --2- 172.21.15.81:0/536561225 >> [v2:172.21.15.62:6800/2759863993,v1:172.21.15.62:6801/2759863993] conn(0x7fb6e802ff60 0x7fb6e8032420 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:35:09.354 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:09.349+0000 7fb6feb2c700 1 --2- 172.21.15.81:0/536561225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb6f8060220 0x7fb6f80f6bd0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:35:09.354 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:09.349+0000 7fb6feb2c700 1 -- 172.21.15.81:0/536561225 >> 172.21.15.81:0/536561225 conn(0x7fb6f80f0b70 msgr2=0x7fb6f80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:35:09.354 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:09.349+0000 7fb6feb2c700 1 -- 172.21.15.81:0/536561225 shutdown_connections 2023-11-15T06:35:09.354 INFO:teuthology.orchestra.run.smithi081.stderr:2023-11-15T06:35:09.349+0000 7fb6feb2c700 1 -- 172.21.15.81:0/536561225 wait complete. 2023-11-15T06:35:09.355 INFO:teuthology.orchestra.run.smithi081.stderr:dumped monmap epoch 1 2023-11-15T06:35:09.759 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2023-11-15T06:35:09.760 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-11-15T06:35:09.760 DEBUG:teuthology.orchestra.run.smithi062:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-11-15T06:35:09.786 DEBUG:teuthology.orchestra.run.smithi081:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-11-15T06:35:09.804 INFO:tasks.cephadm:Stopping all daemons... 2023-11-15T06:35:09.805 INFO:tasks.cephadm.mon.smithi062:Stopping mon.smithi062... 2023-11-15T06:35:09.805 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7@mon.smithi062 2023-11-15T06:35:09.948 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:35:09 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: audit 2023-11-15T06:35:09.349343+0000 mon.smithi062 (mon.0) 328 : audit [DBG] from='client.? 172.21.15.81:0/536561225' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:35:09.948 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:35:09 smithi062 systemd[1]: Stopping Ceph mon.smithi062 for a3c2da76-837f-11ee-8dbe-212e2dc638e7... 2023-11-15T06:35:09.948 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:35:09 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:35:09.944+0000 7fc747e8e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi062 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-11-15T06:35:10.198 INFO:journalctl@ceph.mon.smithi062.smithi062.stdout:Nov 15 06:35:09 smithi062 ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7-mon.smithi062[112362]: debug 2023-11-15T06:35:09.944+0000 7fc747e8e700 -1 mon.smithi062@0(leader) e1 *** Got Signal Terminated *** 2023-11-15T06:35:10.377 DEBUG:teuthology.orchestra.run.smithi062:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7@mon.smithi062.service' 2023-11-15T06:35:10.427 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-15T06:35:10.427 INFO:tasks.cephadm.mon.smithi062:Stopped mon.smithi062 2023-11-15T06:35:10.428 INFO:tasks.cephadm.mon.smithi081:Stopping mon.smithi081... 2023-11-15T06:35:10.428 DEBUG:teuthology.orchestra.run.smithi081:> sudo systemctl stop ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7@mon.smithi081 2023-11-15T06:35:10.463 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to stop ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7@mon.smithi081.service: Unit ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7@mon.smithi081.service not loaded. 2023-11-15T06:35:10.464 DEBUG:teuthology.orchestra.run:got remote process result: 5 2023-11-15T06:35:10.465 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi081" Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi081 with status 5: 'sudo systemctl stop ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7@mon.smithi081' 2023-11-15T06:35:10.465 INFO:tasks.cephadm:Archiving crash dumps... 2023-11-15T06:35:10.468 DEBUG:teuthology.misc:Transferring archived files from smithi062:/var/lib/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/crash to /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458591/remote/ubuntu@smithi062.front.sepia.ceph.com/crash 2023-11-15T06:35:10.470 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar c -f - -C /var/lib/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/crash -- . 2023-11-15T06:35:10.518 DEBUG:teuthology.misc:Transferring archived files from smithi081:/var/lib/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/crash to /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458591/remote/ubuntu@smithi081.front.sepia.ceph.com/crash 2023-11-15T06:35:10.520 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar c -f - -C /var/lib/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/crash -- . 2023-11-15T06:35:10.565 INFO:teuthology.orchestra.run.smithi081.stderr:tar: /var/lib/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/crash: Cannot open: No such file or directory 2023-11-15T06:35:10.566 INFO:teuthology.orchestra.run.smithi081.stderr:tar: Error is not recoverable: exiting now 2023-11-15T06:35:10.567 INFO:tasks.cephadm:Checking cluster log for badness... 2023-11-15T06:35:10.567 DEBUG:teuthology.orchestra.run.smithi062:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-11-15T06:35:10.600 INFO:teuthology.orchestra.run.smithi062.stderr:grep: /var/log/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/ceph.log: No such file or directory 2023-11-15T06:35:10.601 INFO:tasks.cephadm:Compressing logs... 2023-11-15T06:35:10.602 DEBUG:teuthology.orchestra.run.smithi062:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-15T06:35:10.644 DEBUG:teuthology.orchestra.run.smithi081:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-15T06:35:10.671 INFO:teuthology.orchestra.run.smithi062.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-11-15T06:35:10.672 INFO:teuthology.orchestra.run.smithi062.stderr:: No such file or directory 2023-11-15T06:35:10.672 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/ceph-volume.log 2023-11-15T06:35:10.672 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi062.rwurtk.log 2023-11-15T06:35:10.672 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/ceph-mon.smithi062.log 2023-11-15T06:35:10.674 INFO:teuthology.orchestra.run.smithi081.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-11-15T06:35:10.675 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-11-15T06:35:10.675 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/ceph-volume.log: /var/log/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi062.rwurtk.log: /var/log/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/ceph-mon.smithi062.log: 90.2% -- replaced with /var/log/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/ceph-volume.log.gz 2023-11-15T06:35:10.676 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/cephadm.log: 96.2% -- replaced with /var/log/ceph/cephadm.log.gz 2023-11-15T06:35:10.677 INFO:teuthology.orchestra.run.smithi081.stderr: 2023-11-15T06:35:10.677 INFO:teuthology.orchestra.run.smithi081.stderr:real 0m0.014s 2023-11-15T06:35:10.678 INFO:teuthology.orchestra.run.smithi081.stderr:user 0m0.012s 2023-11-15T06:35:10.678 INFO:teuthology.orchestra.run.smithi081.stderr:sys 0m0.010s 2023-11-15T06:35:10.679 INFO:teuthology.orchestra.run.smithi062.stderr: 92.2% -- replaced with /var/log/ceph/cephadm.log.gz 2023-11-15T06:35:10.752 INFO:teuthology.orchestra.run.smithi062.stderr: 92.6% -- replaced with /var/log/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi062.rwurtk.log.gz 2023-11-15T06:35:10.867 INFO:teuthology.orchestra.run.smithi062.stderr: 90.5% -- replaced with /var/log/ceph/a3c2da76-837f-11ee-8dbe-212e2dc638e7/ceph-mon.smithi062.log.gz 2023-11-15T06:35:10.869 INFO:teuthology.orchestra.run.smithi062.stderr: 2023-11-15T06:35:10.870 INFO:teuthology.orchestra.run.smithi062.stderr:real 0m0.209s 2023-11-15T06:35:10.870 INFO:teuthology.orchestra.run.smithi062.stderr:user 0m0.245s 2023-11-15T06:35:10.870 INFO:teuthology.orchestra.run.smithi062.stderr:sys 0m0.024s 2023-11-15T06:35:10.871 INFO:tasks.cephadm:Archiving logs... 2023-11-15T06:35:10.872 DEBUG:teuthology.misc:Transferring archived files from smithi062:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458591/remote/ubuntu@smithi062.front.sepia.ceph.com/log 2023-11-15T06:35:10.873 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-15T06:35:10.995 DEBUG:teuthology.misc:Transferring archived files from smithi081:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458591/remote/ubuntu@smithi081.front.sepia.ceph.com/log 2023-11-15T06:35:10.996 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-15T06:35:11.025 INFO:tasks.cephadm:Removing cluster... 2023-11-15T06:35:11.026 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 --force 2023-11-15T06:35:46.667 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a3c2da76-837f-11ee-8dbe-212e2dc638e7 --force 2023-11-15T06:35:46.969 INFO:tasks.cephadm:Removing cephadm ... 2023-11-15T06:35:46.969 DEBUG:teuthology.orchestra.run.smithi062:> rm -rf /home/ubuntu/cephtest/cephadm 2023-11-15T06:35:46.994 DEBUG:teuthology.orchestra.run.smithi081:> rm -rf /home/ubuntu/cephtest/cephadm 2023-11-15T06:35:47.013 INFO:tasks.cephadm:Teardown complete 2023-11-15T06:35:47.013 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi081 with status 5: 'sudo systemctl stop ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7@mon.smithi081' 2023-11-15T06:35:47.093 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=c3a6bb356a7b40e891ba3cdf868e7c46 Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi081 with status 5: 'sudo systemctl stop ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7@mon.smithi081' 2023-11-15T06:35:47.099 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-11-15T06:35:47.110 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-11-15T06:35:47.150 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-11-15T06:35:47.151 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-15T06:35:47.165 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-15T06:35:47.247 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-11-15T06:35:47.257 INFO:teuthology.task.clock:Checking final clock skew... 2023-11-15T06:35:47.258 DEBUG:teuthology.orchestra.run.smithi062:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-15T06:35:47.261 DEBUG:teuthology.orchestra.run.smithi081:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-15T06:35:47.276 INFO:teuthology.orchestra.run.smithi062.stderr:bash: ntpq: command not found 2023-11-15T06:35:47.278 INFO:teuthology.orchestra.run.smithi081.stderr:bash: ntpq: command not found 2023-11-15T06:35:47.281 INFO:teuthology.orchestra.run.smithi062.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-15T06:35:47.281 INFO:teuthology.orchestra.run.smithi062.stdout:=============================================================================== 2023-11-15T06:35:47.281 INFO:teuthology.orchestra.run.smithi062.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 56 +1915us[+1918us] +/- 102ms 2023-11-15T06:35:47.281 INFO:teuthology.orchestra.run.smithi062.stdout:^* hv02.front.sepia.ceph.com 3 6 377 55 -781us[ -778us] +/- 50ms 2023-11-15T06:35:47.282 INFO:teuthology.orchestra.run.smithi062.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 57 -203us[ -200us] +/- 75ms 2023-11-15T06:35:47.282 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T06:35:47.284 INFO:teuthology.orchestra.run.smithi081.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-15T06:35:47.284 INFO:teuthology.orchestra.run.smithi081.stdout:=============================================================================== 2023-11-15T06:35:47.284 INFO:teuthology.orchestra.run.smithi081.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 54 +1894us[+1903us] +/- 102ms 2023-11-15T06:35:47.284 INFO:teuthology.orchestra.run.smithi081.stdout:^* hv02.front.sepia.ceph.com 3 6 377 53 -802us[ -793us] +/- 50ms 2023-11-15T06:35:47.285 INFO:teuthology.orchestra.run.smithi081.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 56 -218us[ -210us] +/- 75ms 2023-11-15T06:35:47.285 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T06:35:47.286 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-11-15T06:35:47.295 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-11-15T06:35:47.295 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-11-15T06:35:47.344 DEBUG:teuthology.orchestra.run.smithi062:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-15T06:35:47.435 DEBUG:teuthology.orchestra.run.smithi081:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-15T06:35:47.526 DEBUG:teuthology.orchestra.run.smithi062:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-15T06:35:47.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T06:35:47.559 DEBUG:teuthology.orchestra.run.smithi081:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-15T06:35:47.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T06:35:47.592 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-11-15T06:35:47.602 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-11-15T06:35:47.635 INFO:teuthology.task.internal:Duration was 1359.541387 seconds 2023-11-15T06:35:47.636 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-11-15T06:35:47.686 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-11-15T06:35:47.688 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-15T06:35:47.690 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-15T06:35:47.735 INFO:teuthology.orchestra.run.smithi062.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-15T06:35:47.737 INFO:teuthology.orchestra.run.smithi081.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-15T06:35:48.140 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-11-15T06:35:48.141 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi062.front.sepia.ceph.com 2023-11-15T06:35:48.141 DEBUG:teuthology.orchestra.run.smithi062:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-15T06:35:48.164 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi081.front.sepia.ceph.com 2023-11-15T06:35:48.165 DEBUG:teuthology.orchestra.run.smithi081:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-15T06:35:48.188 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-11-15T06:35:48.189 DEBUG:teuthology.orchestra.run.smithi062:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-15T06:35:48.208 DEBUG:teuthology.orchestra.run.smithi081:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-15T06:35:48.274 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-11-15T06:35:48.274 DEBUG:teuthology.orchestra.run.smithi062:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-15T06:35:48.277 DEBUG:teuthology.orchestra.run.smithi081:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-15T06:35:48.363 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-11-15T06:35:48.374 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-11-15T06:35:48.375 DEBUG:teuthology.orchestra.run.smithi062:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-15T06:35:48.403 DEBUG:teuthology.orchestra.run.smithi081:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-15T06:35:48.431 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-11-15T06:35:48.445 DEBUG:teuthology.orchestra.run.smithi062:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-15T06:35:48.449 DEBUG:teuthology.orchestra.run.smithi081:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-15T06:35:48.473 INFO:teuthology.orchestra.run.smithi062.stdout:kernel.core_pattern = core 2023-11-15T06:35:48.498 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern = core 2023-11-15T06:35:48.548 DEBUG:teuthology.orchestra.run.smithi062:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-15T06:35:48.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T06:35:48.567 DEBUG:teuthology.orchestra.run.smithi081:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-15T06:35:48.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T06:35:48.606 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-11-15T06:35:48.620 INFO:teuthology.task.internal:Transferring archived files... 2023-11-15T06:35:48.621 DEBUG:teuthology.misc:Transferring archived files from smithi062:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458591/remote/smithi062 2023-11-15T06:35:48.622 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-15T06:35:48.663 DEBUG:teuthology.misc:Transferring archived files from smithi081:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458591/remote/smithi081 2023-11-15T06:35:48.664 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-15T06:35:48.704 INFO:teuthology.task.internal:Removing archive directory... 2023-11-15T06:35:48.704 DEBUG:teuthology.orchestra.run.smithi062:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-15T06:35:48.707 DEBUG:teuthology.orchestra.run.smithi081:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-15T06:35:48.754 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-11-15T06:35:48.767 INFO:teuthology.task.internal:Not uploading archives. 2023-11-15T06:35:48.768 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-11-15T06:35:48.794 INFO:teuthology.task.internal:Tidying up after the test... 2023-11-15T06:35:48.795 DEBUG:teuthology.orchestra.run.smithi062:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-15T06:35:48.797 DEBUG:teuthology.orchestra.run.smithi081:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-15T06:35:48.813 INFO:teuthology.orchestra.run.smithi062.stdout: 393225 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 15 06:35 /home/ubuntu/cephtest 2023-11-15T06:35:48.815 INFO:teuthology.orchestra.run.smithi081.stdout: 525365 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 15 06:35 /home/ubuntu/cephtest 2023-11-15T06:35:48.816 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-11-15T06:35:48.845 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-11-15T06:35:48.945 INFO:teuthology.nuke:Checking targets against current locks 2023-11-15T06:35:48.970 DEBUG:teuthology.nuke:shortname: smithi062 2023-11-15T06:35:48.971 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-15T06:35:48.997 DEBUG:teuthology.nuke:shortname: smithi081 2023-11-15T06:35:48.998 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-15T06:35:49.001 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi062.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458591', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-15 06:01:05.773227', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEO4f9qXeLsyaEg8zm5TX1dLSdZfIsPBopSJGvx+tD1N5Rn3euSdRmOB7TSf8U9/zLF6g0j1B3WLxLEBSgYDXz8='} 2023-11-15T06:35:49.027 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi081.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458591', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-15 06:01:05.771625', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBM5LBq+RfAdX5BWtHj8tKPv/mmy/VE0f2udeNzpMPJ0BW8X2ba+xgk0g2pU/d6UbD/pXqkBqqI6OCS0Vli8ETM='} 2023-11-15T06:35:49.049 INFO:teuthology.orchestra.console.smithi062:Power off 2023-11-15T06:35:49.050 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-15T06:35:49.073 DEBUG:teuthology.orchestra.console.smithi062:power off output: Chassis Power Control: Down/Off 2023-11-15T06:35:49.074 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T06:35:49.089 INFO:teuthology.orchestra.console.smithi081:Power off 2023-11-15T06:35:49.090 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-15T06:35:49.100 DEBUG:teuthology.orchestra.console.smithi062:check power output: Chassis Power is on 2023-11-15T06:35:49.123 DEBUG:teuthology.orchestra.console.smithi081:power off output: Chassis Power Control: Down/Off 2023-11-15T06:35:49.124 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T06:35:49.155 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is on 2023-11-15T06:35:53.103 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T06:35:53.157 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T06:35:53.214 DEBUG:teuthology.orchestra.console.smithi062:check power output: Chassis Power is on 2023-11-15T06:35:53.270 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is on 2023-11-15T06:35:57.218 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T06:35:57.271 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T06:35:57.330 DEBUG:teuthology.orchestra.console.smithi062:check power output: Chassis Power is off 2023-11-15T06:35:57.383 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is off 2023-11-15T06:35:57.431 INFO:teuthology.orchestra.console.smithi062:Power off completed 2023-11-15T06:35:57.485 INFO:teuthology.orchestra.console.smithi081:Power off completed 2023-11-15T06:35:57.590 INFO:teuthology.run:Summary data: description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1359.541386604309 failure_reason: 'Command failed on smithi081 with status 5: ''sudo systemctl stop ceph-a3c2da76-837f-11ee-8dbe-212e2dc638e7@mon.smithi081''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=c3a6bb356a7b40e891ba3cdf868e7c46 status: fail success: false 2023-11-15T06:35:57.591 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-15T06:35:57.675 INFO:teuthology.run:FAIL