2023-11-17T05:03:57.703 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-17T05:03:57.704 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} --name yuriw-2023-11-16_22:29:26-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi --verbose --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-11-16_22:29:26-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7461279 -- /home/teuthworker/archive/yuriw-2023-11-16_22:29:26-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7461279/orig.config.yaml 2023-11-17T05:03:57.756 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-17T05:03:57.851 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-11-16_22:29:26-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7461279 branch: wip-yuri3-testing-2023-11-14-1227-pacific description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: null first_in_suite: false job_id: '7461279' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: yuriw-2023-11-16_22:29:26-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri3-testing-2023-11-14-1227-pacific ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: ed3c8e926b9de93026c8a7c1d86be5d6821b52ac ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: ed3c8e926b9de93026c8a7c1d86be5d6821b52ac selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri3-testing-2023-11-14-1227-pacific sha1: ed3c8e926b9de93026c8a7c1d86be5d6821b52ac owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 7739 sha1: ed3c8e926b9de93026c8a7c1d86be5d6821b52ac sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri3-testing-2023-11-14-1227-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: ed3c8e926b9de93026c8a7c1d86be5d6821b52ac targets: smithi003.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJJV4WNOwMl33g9HixRkIr7BY3VEXWagoMBrVhRFOUoQxvg4q6RSgGThArhqlOU1kBuDVezziTIIBVgPH2l9pks= smithi160.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPpODOB/solzOsCcE4102HDppGKiXexWxeqSM3BUajX1T17h4kOCuJlrIQASsUkOzbECYHuaChFy5SnFYynEQPM= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - pexec: clients: - sudo modprobe -r ceph - sudo modprobe ceph disable_send_metrics=on - install: exclude_packages: - ceph-volume tag: v16.2.4 - print: '**** done install task...' - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6899cd26fceddb2fec83dc1a1349394b28c8998e timestamp: 2023-11-16_22:29:26 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1224587 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2023-11-17T05:03:57.852 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa; will attempt to use it 2023-11-17T05:03:57.852 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa/tasks 2023-11-17T05:03:57.852 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-11-17T05:03:57.854 INFO:teuthology.task.internal:Checking packages... 2023-11-17T05:03:57.877 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'ed3c8e926b9de93026c8a7c1d86be5d6821b52ac' 2023-11-17T05:03:57.878 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-17T05:03:57.878 INFO:teuthology.packaging:ref: None 2023-11-17T05:03:57.878 INFO:teuthology.packaging:tag: None 2023-11-17T05:03:57.878 INFO:teuthology.packaging:branch: wip-yuri3-testing-2023-11-14-1227-pacific 2023-11-17T05:03:57.878 INFO:teuthology.packaging:sha1: ed3c8e926b9de93026c8a7c1d86be5d6821b52ac 2023-11-17T05:03:57.878 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri3-testing-2023-11-14-1227-pacific 2023-11-17T05:03:58.117 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-171.ged3c8e92 2023-11-17T05:03:58.118 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-11-17T05:03:58.124 INFO:teuthology.task.internal:no buildpackages task found 2023-11-17T05:03:58.124 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-11-17T05:03:58.133 INFO:teuthology.task.internal:Saving configuration 2023-11-17T05:03:58.151 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-11-17T05:03:58.158 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-17T05:03:58.184 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi003.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-16_22:29:26-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7461279', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-17 04:58:22.977822', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJJV4WNOwMl33g9HixRkIr7BY3VEXWagoMBrVhRFOUoQxvg4q6RSgGThArhqlOU1kBuDVezziTIIBVgPH2l9pks='} 2023-11-17T05:03:58.207 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi160.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-16_22:29:26-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7461279', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-17 04:58:22.975807', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPpODOB/solzOsCcE4102HDppGKiXexWxeqSM3BUajX1T17h4kOCuJlrIQASsUkOzbECYHuaChFy5SnFYynEQPM='} 2023-11-17T05:03:58.207 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-11-17T05:03:58.212 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2023-11-17T05:03:58.212 INFO:teuthology.task.internal:roles: ubuntu@smithi160.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2023-11-17T05:03:58.212 INFO:teuthology.run_tasks:Running task console_log... 2023-11-17T05:03:58.290 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc7dd093f70>, signals=[15]) 2023-11-17T05:03:58.337 INFO:teuthology.run_tasks:Running task internal.connect... 2023-11-17T05:03:58.343 INFO:teuthology.task.internal:Opening connections... 2023-11-17T05:03:58.343 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi003.front.sepia.ceph.com 2023-11-17T05:03:58.345 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-17T05:03:58.416 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi160.front.sepia.ceph.com 2023-11-17T05:03:58.417 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-17T05:03:58.487 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-11-17T05:03:58.493 DEBUG:teuthology.orchestra.run.smithi003:> uname -m 2023-11-17T05:03:58.520 INFO:teuthology.orchestra.run.smithi003.stdout:x86_64 2023-11-17T05:03:58.521 DEBUG:teuthology.orchestra.run.smithi003:> cat /etc/os-release 2023-11-17T05:03:58.583 INFO:teuthology.orchestra.run.smithi003.stdout:NAME="CentOS Stream" 2023-11-17T05:03:58.583 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION="8" 2023-11-17T05:03:58.583 INFO:teuthology.orchestra.run.smithi003.stdout:ID="centos" 2023-11-17T05:03:58.583 INFO:teuthology.orchestra.run.smithi003.stdout:ID_LIKE="rhel fedora" 2023-11-17T05:03:58.583 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION_ID="8" 2023-11-17T05:03:58.583 INFO:teuthology.orchestra.run.smithi003.stdout:PLATFORM_ID="platform:el8" 2023-11-17T05:03:58.584 INFO:teuthology.orchestra.run.smithi003.stdout:PRETTY_NAME="CentOS Stream 8" 2023-11-17T05:03:58.584 INFO:teuthology.orchestra.run.smithi003.stdout:ANSI_COLOR="0;31" 2023-11-17T05:03:58.584 INFO:teuthology.orchestra.run.smithi003.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-11-17T05:03:58.584 INFO:teuthology.orchestra.run.smithi003.stdout:HOME_URL="https://centos.org/" 2023-11-17T05:03:58.584 INFO:teuthology.orchestra.run.smithi003.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-17T05:03:58.584 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-17T05:03:58.584 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-17T05:03:58.585 INFO:teuthology.lock.ops:Updating smithi003.front.sepia.ceph.com on lock server 2023-11-17T05:03:58.605 DEBUG:teuthology.orchestra.run.smithi160:> uname -m 2023-11-17T05:03:58.627 INFO:teuthology.orchestra.run.smithi160.stdout:x86_64 2023-11-17T05:03:58.627 DEBUG:teuthology.orchestra.run.smithi160:> cat /etc/os-release 2023-11-17T05:03:58.689 INFO:teuthology.orchestra.run.smithi160.stdout:NAME="CentOS Stream" 2023-11-17T05:03:58.689 INFO:teuthology.orchestra.run.smithi160.stdout:VERSION="8" 2023-11-17T05:03:58.689 INFO:teuthology.orchestra.run.smithi160.stdout:ID="centos" 2023-11-17T05:03:58.689 INFO:teuthology.orchestra.run.smithi160.stdout:ID_LIKE="rhel fedora" 2023-11-17T05:03:58.689 INFO:teuthology.orchestra.run.smithi160.stdout:VERSION_ID="8" 2023-11-17T05:03:58.690 INFO:teuthology.orchestra.run.smithi160.stdout:PLATFORM_ID="platform:el8" 2023-11-17T05:03:58.690 INFO:teuthology.orchestra.run.smithi160.stdout:PRETTY_NAME="CentOS Stream 8" 2023-11-17T05:03:58.690 INFO:teuthology.orchestra.run.smithi160.stdout:ANSI_COLOR="0;31" 2023-11-17T05:03:58.690 INFO:teuthology.orchestra.run.smithi160.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-11-17T05:03:58.690 INFO:teuthology.orchestra.run.smithi160.stdout:HOME_URL="https://centos.org/" 2023-11-17T05:03:58.690 INFO:teuthology.orchestra.run.smithi160.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-17T05:03:58.690 INFO:teuthology.orchestra.run.smithi160.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-17T05:03:58.690 INFO:teuthology.orchestra.run.smithi160.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-17T05:03:58.690 INFO:teuthology.lock.ops:Updating smithi160.front.sepia.ceph.com on lock server 2023-11-17T05:03:58.713 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-11-17T05:03:58.737 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-11-17T05:03:58.751 INFO:teuthology.task.internal:Checking for old test directory... 2023-11-17T05:03:58.751 DEBUG:teuthology.orchestra.run.smithi003:> test '!' -e /home/ubuntu/cephtest 2023-11-17T05:03:58.754 DEBUG:teuthology.orchestra.run.smithi160:> test '!' -e /home/ubuntu/cephtest 2023-11-17T05:03:58.774 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-11-17T05:03:58.779 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-11-17T05:03:58.779 DEBUG:teuthology.orchestra.run.smithi003:> test -z $(ls -A /var/lib/ceph) 2023-11-17T05:03:58.821 DEBUG:teuthology.orchestra.run.smithi160:> test -z $(ls -A /var/lib/ceph) 2023-11-17T05:03:58.852 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-11-17T05:03:58.933 INFO:teuthology.run_tasks:Running task kernel... 2023-11-17T05:03:58.973 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-11-17T05:03:58.973 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-11-17T05:03:58.973 DEBUG:teuthology.orchestra.run.smithi003:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-17T05:03:58.974 DEBUG:teuthology.orchestra.run.smithi160:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-17T05:03:58.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-17T05:03:58.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-17T05:03:58.997 DEBUG:teuthology.orchestra.run.smithi003:> uname -r 2023-11-17T05:03:58.997 DEBUG:teuthology.orchestra.run.smithi160:> uname -r 2023-11-17T05:03:59.059 INFO:teuthology.orchestra.run.smithi160.stdout:4.18.0-489.el8.x86_64 2023-11-17T05:03:59.059 INFO:teuthology.task.kernel:Running kernel on smithi160: 4.18.0-489.el8.x86_64 2023-11-17T05:03:59.059 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum install -y kernel 2023-11-17T05:03:59.060 INFO:teuthology.orchestra.run.smithi003.stdout:4.18.0-489.el8.x86_64 2023-11-17T05:03:59.060 INFO:teuthology.task.kernel:Running kernel on smithi003: 4.18.0-489.el8.x86_64 2023-11-17T05:03:59.061 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum install -y kernel 2023-11-17T05:04:08.534 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 8 - AppStream 4.9 MB/s | 34 MB 00:06 2023-11-17T05:04:09.798 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 8 - AppStream 4.1 MB/s | 34 MB 00:08 2023-11-17T05:04:17.367 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 8 - BaseOS 23 MB/s | 53 MB 00:02 2023-11-17T05:04:18.414 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 8 - BaseOS 22 MB/s | 53 MB 00:02 2023-11-17T05:04:25.842 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 8 - Extras 53 kB/s | 18 kB 00:00 2023-11-17T05:04:26.185 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 8 - Extras common packages 45 kB/s | 6.9 kB 00:00 2023-11-17T05:04:26.288 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 8 - Extras 57 kB/s | 18 kB 00:00 2023-11-17T05:04:26.511 INFO:teuthology.orchestra.run.smithi003.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2023-11-17T05:04:26.581 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 8 - Extras common packages 63 kB/s | 6.9 kB 00:00 2023-11-17T05:04:26.892 INFO:teuthology.orchestra.run.smithi160.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2023-11-17T05:04:27.758 INFO:teuthology.orchestra.run.smithi160.stdout:Extra Packages for Enterprise Linux 22 MB/s | 16 MB 00:00 2023-11-17T05:04:27.778 INFO:teuthology.orchestra.run.smithi003.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2023-11-17T05:04:31.633 INFO:teuthology.orchestra.run.smithi160.stdout:lab-extras 420 kB/s | 24 kB 00:00 2023-11-17T05:04:31.845 INFO:teuthology.orchestra.run.smithi003.stdout:lab-extras 426 kB/s | 24 kB 00:00 2023-11-17T05:04:35.241 INFO:teuthology.orchestra.run.smithi160.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-11-17T05:04:35.241 INFO:teuthology.orchestra.run.smithi160.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-11-17T05:04:35.512 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-11-17T05:04:35.513 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:04:35.514 INFO:teuthology.orchestra.run.smithi160.stdout: Package Architecture Version Repository Size 2023-11-17T05:04:35.514 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:04:35.514 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-11-17T05:04:35.516 INFO:teuthology.orchestra.run.smithi160.stdout: kernel x86_64 4.18.0-521.el8 baseos 10 M 2023-11-17T05:04:35.517 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2023-11-17T05:04:35.517 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-core x86_64 4.18.0-521.el8 baseos 43 M 2023-11-17T05:04:35.517 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-modules x86_64 4.18.0-521.el8 baseos 36 M 2023-11-17T05:04:35.517 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:04:35.517 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-11-17T05:04:35.517 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:04:35.517 INFO:teuthology.orchestra.run.smithi160.stdout:Install 3 Packages 2023-11-17T05:04:35.517 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:04:35.538 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 89 M 2023-11-17T05:04:35.538 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 96 M 2023-11-17T05:04:35.538 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-11-17T05:04:35.703 INFO:teuthology.orchestra.run.smithi003.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-11-17T05:04:35.703 INFO:teuthology.orchestra.run.smithi003.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-11-17T05:04:35.991 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-17T05:04:35.992 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:04:35.992 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2023-11-17T05:04:35.992 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:04:35.992 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-11-17T05:04:35.992 INFO:teuthology.orchestra.run.smithi003.stdout: kernel x86_64 4.18.0-521.el8 baseos 10 M 2023-11-17T05:04:35.992 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2023-11-17T05:04:35.992 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-core x86_64 4.18.0-521.el8 baseos 43 M 2023-11-17T05:04:35.992 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules x86_64 4.18.0-521.el8 baseos 36 M 2023-11-17T05:04:35.993 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:04:35.993 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-11-17T05:04:35.993 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:04:35.993 INFO:teuthology.orchestra.run.smithi003.stdout:Install 3 Packages 2023-11-17T05:04:35.993 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:04:35.993 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 89 M 2023-11-17T05:04:35.993 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 96 M 2023-11-17T05:04:35.994 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-11-17T05:04:36.719 INFO:teuthology.orchestra.run.smithi160.stdout:(1/3): kernel-4.18.0-521.el8.x86_64.rpm 8.6 MB/s | 10 MB 00:01 2023-11-17T05:04:37.554 INFO:teuthology.orchestra.run.smithi003.stdout:(1/3): kernel-4.18.0-521.el8.x86_64.rpm 7.3 MB/s | 10 MB 00:01 2023-11-17T05:04:37.868 INFO:teuthology.orchestra.run.smithi160.stdout:(2/3): kernel-modules-4.18.0-521.el8.x86_64.rpm 16 MB/s | 36 MB 00:02 2023-11-17T05:04:38.519 INFO:teuthology.orchestra.run.smithi160.stdout:(3/3): kernel-core-4.18.0-521.el8.x86_64.rpm 15 MB/s | 43 MB 00:02 2023-11-17T05:04:38.519 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-11-17T05:04:38.520 INFO:teuthology.orchestra.run.smithi160.stdout:Total 30 MB/s | 89 MB 00:03 2023-11-17T05:04:38.679 INFO:teuthology.orchestra.run.smithi003.stdout:(2/3): kernel-modules-4.18.0-521.el8.x86_64.rpm 14 MB/s | 36 MB 00:02 2023-11-17T05:04:39.198 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-11-17T05:04:39.280 INFO:teuthology.orchestra.run.smithi003.stdout:(3/3): kernel-core-4.18.0-521.el8.x86_64.rpm 14 MB/s | 43 MB 00:03 2023-11-17T05:04:39.280 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-11-17T05:04:39.280 INFO:teuthology.orchestra.run.smithi003.stdout:Total 27 MB/s | 89 MB 00:03 2023-11-17T05:04:39.617 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-11-17T05:04:39.617 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-11-17T05:04:39.984 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-11-17T05:04:40.431 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-11-17T05:04:40.431 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-11-17T05:04:40.844 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-11-17T05:04:40.865 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-11-17T05:04:41.734 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-11-17T05:04:41.763 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-11-17T05:04:43.577 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-11-17T05:04:44.180 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-17T05:04:44.707 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-11-17T05:04:45.244 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-17T05:04:47.746 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-17T05:04:48.095 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-17T05:04:49.090 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-17T05:04:49.493 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-17T05:04:51.523 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-17T05:04:51.804 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : kernel-4.18.0-521.el8.x86_64 3/3 2023-11-17T05:04:53.224 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-17T05:04:53.528 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-4.18.0-521.el8.x86_64 3/3 2023-11-17T05:05:13.693 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 3/3 2023-11-17T05:05:13.718 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-17T05:05:14.051 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-4.18.0-521.el8.x86_64 3/3 2023-11-17T05:05:14.052 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : kernel-4.18.0-521.el8.x86_64 1/3 2023-11-17T05:05:14.052 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : kernel-core-4.18.0-521.el8.x86_64 2/3 2023-11-17T05:05:14.766 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-17T05:05:14.768 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:05:14.769 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-11-17T05:05:14.773 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-4.18.0-521.el8.x86_64 kernel-core-4.18.0-521.el8.x86_64 2023-11-17T05:05:14.773 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-modules-4.18.0-521.el8.x86_64 2023-11-17T05:05:14.773 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:05:14.773 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-11-17T05:05:14.928 DEBUG:teuthology.orchestra.run.smithi160:> rpm -q kernel | sort -rV | head -n 1 2023-11-17T05:05:15.195 INFO:teuthology.orchestra.run.smithi160.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-17T05:05:15.196 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-17T05:05:15.196 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-521.el8.x86_64 2023-11-17T05:05:15.196 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-17T05:05:15.196 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-17T05:05:15.196 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2023-11-17T05:05:15.196 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi160.front.sepia.ceph.com, path=None, version=distro) 2023-11-17T05:05:15.196 DEBUG:teuthology.orchestra.run.smithi160:> rpm -q kernel | sort -rV | head -n 1 2023-11-17T05:05:15.405 INFO:teuthology.orchestra.run.smithi160.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-17T05:05:15.405 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-17T05:05:15.405 DEBUG:teuthology.orchestra.run.smithi160:> sudo rpm -qi grub2-tools 2023-11-17T05:05:15.440 INFO:teuthology.orchestra.run.smithi160.stdout:Name : grub2-tools 2023-11-17T05:05:15.440 INFO:teuthology.orchestra.run.smithi160.stdout:Epoch : 1 2023-11-17T05:05:15.440 INFO:teuthology.orchestra.run.smithi160.stdout:Version : 2.02 2023-11-17T05:05:15.440 INFO:teuthology.orchestra.run.smithi160.stdout:Release : 129.el8 2023-11-17T05:05:15.440 INFO:teuthology.orchestra.run.smithi160.stdout:Architecture: x86_64 2023-11-17T05:05:15.440 INFO:teuthology.orchestra.run.smithi160.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-11-17T05:05:15.440 INFO:teuthology.orchestra.run.smithi160.stdout:Group : System Environment/Base 2023-11-17T05:05:15.440 INFO:teuthology.orchestra.run.smithi160.stdout:Size : 9265586 2023-11-17T05:05:15.440 INFO:teuthology.orchestra.run.smithi160.stdout:License : GPLv3+ 2023-11-17T05:05:15.440 INFO:teuthology.orchestra.run.smithi160.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-11-17T05:05:15.441 INFO:teuthology.orchestra.run.smithi160.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-11-17T05:05:15.441 INFO:teuthology.orchestra.run.smithi160.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-11-17T05:05:15.441 INFO:teuthology.orchestra.run.smithi160.stdout:Build Host : kbuilder.bsys.centos.org 2023-11-17T05:05:15.441 INFO:teuthology.orchestra.run.smithi160.stdout:Relocations : (not relocatable) 2023-11-17T05:05:15.441 INFO:teuthology.orchestra.run.smithi160.stdout:Packager : CentOS BuildSystem 2023-11-17T05:05:15.441 INFO:teuthology.orchestra.run.smithi160.stdout:Vendor : CentOS 2023-11-17T05:05:15.441 INFO:teuthology.orchestra.run.smithi160.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-17T05:05:15.441 INFO:teuthology.orchestra.run.smithi160.stdout:Summary : Support tools for GRUB. 2023-11-17T05:05:15.441 INFO:teuthology.orchestra.run.smithi160.stdout:Description : 2023-11-17T05:05:15.441 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:05:15.441 INFO:teuthology.orchestra.run.smithi160.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-17T05:05:15.441 INFO:teuthology.orchestra.run.smithi160.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-17T05:05:15.441 INFO:teuthology.orchestra.run.smithi160.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-17T05:05:15.441 INFO:teuthology.orchestra.run.smithi160.stdout:hardware devices. 2023-11-17T05:05:15.442 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:05:15.442 INFO:teuthology.orchestra.run.smithi160.stdout:This subpackage provides tools for support of all platforms. 2023-11-17T05:05:15.442 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-17T05:05:15.443 INFO:teuthology.task.kernel:Updating grub on smithi160 to boot 4.18.0-521.el8.x86_64 2023-11-17T05:05:15.443 DEBUG:teuthology.orchestra.run.smithi160:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-17T05:05:16.923 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 3/3 2023-11-17T05:05:16.946 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-17T05:05:17.099 INFO:teuthology.orchestra.run.smithi160.stderr:Generating grub configuration file ... 2023-11-17T05:05:17.292 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-4.18.0-521.el8.x86_64 3/3 2023-11-17T05:05:17.292 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-4.18.0-521.el8.x86_64 1/3 2023-11-17T05:05:17.292 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-core-4.18.0-521.el8.x86_64 2/3 2023-11-17T05:05:18.118 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-17T05:05:18.118 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:05:18.118 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-11-17T05:05:18.147 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-4.18.0-521.el8.x86_64 kernel-core-4.18.0-521.el8.x86_64 2023-11-17T05:05:18.147 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules-4.18.0-521.el8.x86_64 2023-11-17T05:05:18.147 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:05:18.147 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-17T05:05:18.276 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q kernel | sort -rV | head -n 1 2023-11-17T05:05:18.497 INFO:teuthology.orchestra.run.smithi003.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-17T05:05:18.497 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-17T05:05:18.497 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-521.el8.x86_64 2023-11-17T05:05:18.497 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-17T05:05:18.497 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-17T05:05:18.497 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2023-11-17T05:05:18.497 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi003.front.sepia.ceph.com, path=None, version=distro) 2023-11-17T05:05:18.498 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q kernel | sort -rV | head -n 1 2023-11-17T05:05:18.716 INFO:teuthology.orchestra.run.smithi003.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-17T05:05:18.716 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-17T05:05:18.717 DEBUG:teuthology.orchestra.run.smithi003:> sudo rpm -qi grub2-tools 2023-11-17T05:05:18.751 INFO:teuthology.orchestra.run.smithi003.stdout:Name : grub2-tools 2023-11-17T05:05:18.751 INFO:teuthology.orchestra.run.smithi003.stdout:Epoch : 1 2023-11-17T05:05:18.751 INFO:teuthology.orchestra.run.smithi003.stdout:Version : 2.02 2023-11-17T05:05:18.751 INFO:teuthology.orchestra.run.smithi003.stdout:Release : 129.el8 2023-11-17T05:05:18.751 INFO:teuthology.orchestra.run.smithi003.stdout:Architecture: x86_64 2023-11-17T05:05:18.751 INFO:teuthology.orchestra.run.smithi003.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-11-17T05:05:18.751 INFO:teuthology.orchestra.run.smithi003.stdout:Group : System Environment/Base 2023-11-17T05:05:18.751 INFO:teuthology.orchestra.run.smithi003.stdout:Size : 9265586 2023-11-17T05:05:18.752 INFO:teuthology.orchestra.run.smithi003.stdout:License : GPLv3+ 2023-11-17T05:05:18.752 INFO:teuthology.orchestra.run.smithi003.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-11-17T05:05:18.752 INFO:teuthology.orchestra.run.smithi003.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-11-17T05:05:18.752 INFO:teuthology.orchestra.run.smithi003.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-11-17T05:05:18.752 INFO:teuthology.orchestra.run.smithi003.stdout:Build Host : kbuilder.bsys.centos.org 2023-11-17T05:05:18.752 INFO:teuthology.orchestra.run.smithi003.stdout:Relocations : (not relocatable) 2023-11-17T05:05:18.752 INFO:teuthology.orchestra.run.smithi003.stdout:Packager : CentOS BuildSystem 2023-11-17T05:05:18.752 INFO:teuthology.orchestra.run.smithi003.stdout:Vendor : CentOS 2023-11-17T05:05:18.752 INFO:teuthology.orchestra.run.smithi003.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-17T05:05:18.752 INFO:teuthology.orchestra.run.smithi003.stdout:Summary : Support tools for GRUB. 2023-11-17T05:05:18.752 INFO:teuthology.orchestra.run.smithi003.stdout:Description : 2023-11-17T05:05:18.752 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:05:18.752 INFO:teuthology.orchestra.run.smithi003.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-17T05:05:18.753 INFO:teuthology.orchestra.run.smithi003.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-17T05:05:18.753 INFO:teuthology.orchestra.run.smithi003.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-17T05:05:18.753 INFO:teuthology.orchestra.run.smithi003.stdout:hardware devices. 2023-11-17T05:05:18.753 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:05:18.753 INFO:teuthology.orchestra.run.smithi003.stdout:This subpackage provides tools for support of all platforms. 2023-11-17T05:05:18.754 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-17T05:05:18.754 INFO:teuthology.task.kernel:Updating grub on smithi003 to boot 4.18.0-521.el8.x86_64 2023-11-17T05:05:18.754 DEBUG:teuthology.orchestra.run.smithi003:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-17T05:05:19.432 INFO:teuthology.orchestra.run.smithi160.stderr:done 2023-11-17T05:05:19.437 DEBUG:teuthology.orchestra.run.smithi160:> mktemp 2023-11-17T05:05:19.480 INFO:teuthology.orchestra.run.smithi160.stdout:/tmp/tmp.eFml13goUl 2023-11-17T05:05:19.481 DEBUG:teuthology.orchestra.run.smithi160:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.eFml13goUl 2023-11-17T05:05:19.555 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod 0666 /tmp/tmp.eFml13goUl 2023-11-17T05:05:19.706 DEBUG:teuthology.orchestra.remote:smithi160:/tmp/tmp.eFml13goUl is 6KB 2023-11-17T05:05:19.757 DEBUG:teuthology.orchestra.run.smithi160:> rm -fr /tmp/tmp.eFml13goUl 2023-11-17T05:05:19.775 DEBUG:teuthology.orchestra.run.smithi160:> sudo /bin/ls /boot/loader/entries || true 2023-11-17T05:05:19.841 INFO:teuthology.orchestra.run.smithi160.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-11-17T05:05:19.841 INFO:teuthology.orchestra.run.smithi160.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-11-17T05:05:19.841 INFO:teuthology.orchestra.run.smithi160.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-11-17T05:05:19.841 INFO:teuthology.orchestra.run.smithi160.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64.conf 2023-11-17T05:05:19.842 DEBUG:teuthology.orchestra.run.smithi160:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64 2023-11-17T05:05:19.994 DEBUG:teuthology.orchestra.run.smithi160:> sudo shutdown -r now 2023-11-17T05:05:20.284 INFO:teuthology.orchestra.run.smithi003.stderr:Generating grub configuration file ... 2023-11-17T05:05:22.461 INFO:teuthology.orchestra.run.smithi003.stderr:done 2023-11-17T05:05:22.463 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2023-11-17T05:05:22.495 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.fWIHYJG7RH 2023-11-17T05:05:22.495 DEBUG:teuthology.orchestra.run.smithi003:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.fWIHYJG7RH 2023-11-17T05:05:22.568 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.fWIHYJG7RH 2023-11-17T05:05:31.866 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.fWIHYJG7RH is 6KB 2023-11-17T05:06:07.284 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2023-11-17T05:06:07.335 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2023-11-17T05:06:07.337 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-17T05:06:07.413 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.fWIHYJG7RH 2023-11-17T05:06:07.436 DEBUG:teuthology.orchestra.run.smithi003:> sudo /bin/ls /boot/loader/entries || true 2023-11-17T05:06:07.509 INFO:teuthology.orchestra.run.smithi003.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-11-17T05:06:07.510 INFO:teuthology.orchestra.run.smithi003.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-11-17T05:06:07.510 INFO:teuthology.orchestra.run.smithi003.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-11-17T05:06:07.510 INFO:teuthology.orchestra.run.smithi003.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64.conf 2023-11-17T05:06:07.511 DEBUG:teuthology.orchestra.run.smithi003:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64 2023-11-17T05:06:07.708 DEBUG:teuthology.orchestra.run.smithi003:> sudo shutdown -r now 2023-11-17T05:06:25.790 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2023-11-17T05:06:34.798 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2023-11-17T05:06:34.799 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-17T05:06:37.714 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2023-11-17T05:06:37.714 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-11-17T05:06:37.715 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-17T05:06:53.179 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2023-11-17T05:07:05.190 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2023-11-17T05:07:05.191 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-17T05:07:05.513 DEBUG:teuthology.orchestra.run.smithi160:> true 2023-11-17T05:07:05.852 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi160.front.sepia.ceph.com' 2023-11-17T05:07:05.852 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-521.el8.x86_64"... 2023-11-17T05:07:05.852 DEBUG:teuthology.orchestra.run.smithi160:> uname -r 2023-11-17T05:07:05.871 INFO:teuthology.orchestra.run.smithi160.stdout:4.18.0-521.el8.x86_64 2023-11-17T05:07:05.871 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-521.el8.x86_64 vs 4.18.0-521.el8.x86_64 2023-11-17T05:07:05.872 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-17T05:07:05.872 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-17T05:07:06.872 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2023-11-17T05:07:06.873 DEBUG:teuthology.orchestra.run.smithi160:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-17T05:07:06.946 INFO:teuthology.orchestra.run.smithi160.stdout:ttyS1 2023-11-17T05:07:06.957 DEBUG:teuthology.parallel:result is None 2023-11-17T05:07:37.717 DEBUG:teuthology.orchestra.remote:timed out 2023-11-17T05:07:46.718 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-11-17T05:07:46.719 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-17T05:07:47.010 DEBUG:teuthology.orchestra.run.smithi003:> true 2023-11-17T05:07:47.267 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-11-17T05:07:47.267 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-521.el8.x86_64"... 2023-11-17T05:07:47.268 DEBUG:teuthology.orchestra.run.smithi003:> uname -r 2023-11-17T05:07:47.290 INFO:teuthology.orchestra.run.smithi003.stdout:4.18.0-521.el8.x86_64 2023-11-17T05:07:47.290 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-521.el8.x86_64 vs 4.18.0-521.el8.x86_64 2023-11-17T05:07:47.291 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-17T05:07:47.291 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-17T05:07:48.293 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2023-11-17T05:07:48.293 DEBUG:teuthology.orchestra.run.smithi003:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-17T05:07:48.357 INFO:teuthology.orchestra.run.smithi003.stdout:ttyS1 2023-11-17T05:07:48.373 DEBUG:teuthology.parallel:result is None 2023-11-17T05:07:48.374 INFO:teuthology.run_tasks:Running task internal.base... 2023-11-17T05:07:48.382 INFO:teuthology.task.internal:Creating test directory... 2023-11-17T05:07:48.382 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-17T05:07:48.386 DEBUG:teuthology.orchestra.run.smithi160:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-17T05:07:48.412 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-11-17T05:07:48.419 INFO:teuthology.run_tasks:Running task internal.archive... 2023-11-17T05:07:48.426 INFO:teuthology.task.internal:Creating archive directory... 2023-11-17T05:07:48.426 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-17T05:07:48.448 DEBUG:teuthology.orchestra.run.smithi160:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-17T05:07:48.491 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-11-17T05:07:48.503 INFO:teuthology.task.internal:Enabling coredump saving... 2023-11-17T05:07:48.503 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-17T05:07:48.520 DEBUG:teuthology.orchestra.run.smithi160:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-17T05:07:48.560 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-17T05:07:48.572 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-17T05:07:48.574 INFO:teuthology.orchestra.run.smithi160.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-17T05:07:48.585 INFO:teuthology.orchestra.run.smithi160.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-17T05:07:48.586 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-11-17T05:07:48.593 INFO:teuthology.task.internal:Configuring sudo... 2023-11-17T05:07:48.594 DEBUG:teuthology.orchestra.run.smithi003:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-17T05:07:48.616 DEBUG:teuthology.orchestra.run.smithi160:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-17T05:07:48.646 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-11-17T05:07:48.657 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-11-17T05:07:48.657 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-17T05:07:48.690 DEBUG:teuthology.orchestra.run.smithi160:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-17T05:07:48.710 DEBUG:teuthology.orchestra.run.smithi003:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-17T05:07:48.792 DEBUG:teuthology.orchestra.run.smithi003:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-17T05:07:48.848 DEBUG:teuthology.orchestra.run.smithi003:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-17T05:07:48.936 DEBUG:teuthology.orchestra.run.smithi003:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-17T05:07:48.979 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-11-17T05:07:48.979 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-17T05:07:49.049 DEBUG:teuthology.orchestra.run.smithi160:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-17T05:07:49.094 DEBUG:teuthology.orchestra.run.smithi160:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-17T05:07:49.154 DEBUG:teuthology.orchestra.run.smithi160:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-17T05:07:49.243 DEBUG:teuthology.orchestra.run.smithi160:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-17T05:07:49.287 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-11-17T05:07:49.287 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-17T05:07:49.361 DEBUG:teuthology.orchestra.run.smithi003:> sudo service rsyslog restart 2023-11-17T05:07:49.366 DEBUG:teuthology.orchestra.run.smithi160:> sudo service rsyslog restart 2023-11-17T05:07:49.432 INFO:teuthology.orchestra.run.smithi003.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-17T05:07:49.487 INFO:teuthology.orchestra.run.smithi160.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-17T05:07:49.933 INFO:teuthology.run_tasks:Running task internal.timer... 2023-11-17T05:07:49.941 INFO:teuthology.task.internal:Starting timer... 2023-11-17T05:07:49.941 INFO:teuthology.run_tasks:Running task pcp... 2023-11-17T05:07:49.956 INFO:teuthology.run_tasks:Running task selinux... 2023-11-17T05:07:49.978 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-11-17T05:07:49.978 DEBUG:teuthology.orchestra.run.smithi003:> sudo service auditd rotate 2023-11-17T05:07:50.060 INFO:teuthology.orchestra.run.smithi003.stdout:Rotating logs: 2023-11-17T05:07:50.062 DEBUG:teuthology.orchestra.run.smithi160:> sudo service auditd rotate 2023-11-17T05:07:50.149 INFO:teuthology.orchestra.run.smithi160.stdout:Rotating logs: 2023-11-17T05:07:50.150 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-11-17T05:07:50.150 DEBUG:teuthology.orchestra.run.smithi003:> /usr/sbin/getenforce 2023-11-17T05:07:50.183 INFO:teuthology.orchestra.run.smithi003.stdout:Permissive 2023-11-17T05:07:50.183 DEBUG:teuthology.orchestra.run.smithi160:> /usr/sbin/getenforce 2023-11-17T05:07:50.209 INFO:teuthology.orchestra.run.smithi160.stdout:Permissive 2023-11-17T05:07:50.210 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi003.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi160.front.sepia.ceph.com': 'permissive'} 2023-11-17T05:07:50.210 DEBUG:teuthology.orchestra.run.smithi003:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-17T05:07:50.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-17T05:07:50.256 DEBUG:teuthology.orchestra.run.smithi160:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-17T05:07:50.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-17T05:07:50.284 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-11-17T05:07:50.285 DEBUG:teuthology.orchestra.run.smithi003:> sudo /usr/sbin/setenforce permissive 2023-11-17T05:07:50.330 DEBUG:teuthology.orchestra.run.smithi160:> sudo /usr/sbin/setenforce permissive 2023-11-17T05:07:50.358 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-11-17T05:07:50.369 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-11-17T05:07:50.386 INFO:teuthology.repo_utils:Fetching main from origin 2023-11-17T05:07:50.540 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-11-17T05:07:50.569 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-11-17T05:07:50.570 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi003.front.sepia.ceph.com,smithi160.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-11-17T05:17:18.490 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi003.front.sepia.ceph.com'), Remote(name='ubuntu@smithi160.front.sepia.ceph.com')] 2023-11-17T05:18:02.735 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-11-17T05:18:02.761 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-17T05:18:03.258 DEBUG:teuthology.orchestra.run.smithi003:> true 2023-11-17T05:18:03.332 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-11-17T05:18:03.333 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2023-11-17T05:18:03.333 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-17T05:18:03.406 DEBUG:teuthology.orchestra.run.smithi160:> true 2023-11-17T05:18:03.436 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi160.front.sepia.ceph.com' 2023-11-17T05:18:03.436 INFO:teuthology.run_tasks:Running task clock... 2023-11-17T05:18:03.447 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-11-17T05:18:03.447 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-17T05:18:03.448 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-17T05:18:03.450 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-17T05:18:03.450 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-17T05:18:03.487 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-17T05:18:03.505 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-17T05:18:03.511 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-17T05:18:03.526 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-17T05:18:03.592 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: ntpd: command not found 2023-11-17T05:18:03.602 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: ntpd: command not found 2023-11-17T05:18:03.605 INFO:teuthology.orchestra.run.smithi003.stdout:506 Cannot talk to daemon 2023-11-17T05:18:03.613 INFO:teuthology.orchestra.run.smithi160.stdout:506 Cannot talk to daemon 2023-11-17T05:18:03.622 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-17T05:18:03.627 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-17T05:18:03.637 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-17T05:18:03.640 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-17T05:18:03.703 INFO:teuthology.orchestra.run.smithi160.stderr:bash: ntpq: command not found 2023-11-17T05:18:03.707 INFO:teuthology.orchestra.run.smithi160.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-17T05:18:03.707 INFO:teuthology.orchestra.run.smithi160.stdout:=============================================================================== 2023-11-17T05:18:03.707 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-17T05:18:03.707 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-17T05:18:03.707 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-17T05:18:03.707 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-17T05:18:03.709 INFO:teuthology.orchestra.run.smithi003.stderr:bash: ntpq: command not found 2023-11-17T05:18:03.712 INFO:teuthology.orchestra.run.smithi003.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-17T05:18:03.712 INFO:teuthology.orchestra.run.smithi003.stdout:=============================================================================== 2023-11-17T05:18:03.712 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-17T05:18:03.712 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-17T05:18:03.712 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-17T05:18:03.713 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-17T05:18:03.714 INFO:teuthology.run_tasks:Running task pexec... 2023-11-17T05:18:03.723 INFO:teuthology.task.pexec:Executing custom commands... 2023-11-17T05:18:03.724 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi003.front.sepia.ceph.com 2023-11-17T05:18:03.724 DEBUG:teuthology.orchestra.run.smithi003:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-17T05:18:03.724 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi160.front.sepia.ceph.com 2023-11-17T05:18:03.725 DEBUG:teuthology.orchestra.run.smithi160:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-17T05:18:04.639 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:03:16 ago on Fri 17 Nov 2023 05:14:48 AM UTC. 2023-11-17T05:18:04.770 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:30 ago on Fri 17 Nov 2023 05:15:34 AM UTC. 2023-11-17T05:18:06.774 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-11-17T05:18:06.775 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:18:06.776 INFO:teuthology.orchestra.run.smithi160.stdout: Package Architecture Version Repository Size 2023-11-17T05:18:06.776 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:18:06.776 INFO:teuthology.orchestra.run.smithi160.stdout:Resetting modules: 2023-11-17T05:18:06.776 INFO:teuthology.orchestra.run.smithi160.stdout: container-tools 2023-11-17T05:18:06.776 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:18:06.776 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-11-17T05:18:06.777 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:18:06.777 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:18:07.032 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-11-17T05:18:07.143 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-17T05:18:07.144 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:18:07.144 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2023-11-17T05:18:07.144 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:18:07.145 INFO:teuthology.orchestra.run.smithi003.stdout:Resetting modules: 2023-11-17T05:18:07.145 INFO:teuthology.orchestra.run.smithi003.stdout: container-tools 2023-11-17T05:18:07.145 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:18:07.145 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-11-17T05:18:07.145 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:18:07.145 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:18:07.415 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-17T05:18:07.977 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:03:19 ago on Fri 17 Nov 2023 05:14:48 AM UTC. 2023-11-17T05:18:08.512 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:34 ago on Fri 17 Nov 2023 05:15:34 AM UTC. 2023-11-17T05:18:10.088 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-11-17T05:18:11.772 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-17T05:18:11.773 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:18:11.773 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2023-11-17T05:18:11.773 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:18:11.773 INFO:teuthology.orchestra.run.smithi003.stdout:Installing group/module packages: 2023-11-17T05:18:11.773 INFO:teuthology.orchestra.run.smithi003.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2023-11-17T05:18:11.774 INFO:teuthology.orchestra.run.smithi003.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2023-11-17T05:18:11.774 INFO:teuthology.orchestra.run.smithi003.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2023-11-17T05:18:11.776 INFO:teuthology.orchestra.run.smithi003.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2023-11-17T05:18:11.810 INFO:teuthology.orchestra.run.smithi003.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2023-11-17T05:18:11.810 INFO:teuthology.orchestra.run.smithi003.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2023-11-17T05:18:11.811 INFO:teuthology.orchestra.run.smithi003.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2023-11-17T05:18:11.811 INFO:teuthology.orchestra.run.smithi003.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2023-11-17T05:18:11.811 INFO:teuthology.orchestra.run.smithi003.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2023-11-17T05:18:11.811 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2023-11-17T05:18:11.811 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-11-17T05:18:11.811 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-11-17T05:18:11.812 INFO:teuthology.orchestra.run.smithi003.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-11-17T05:18:11.812 INFO:teuthology.orchestra.run.smithi003.stdout:Installing weak dependencies: 2023-11-17T05:18:11.812 INFO:teuthology.orchestra.run.smithi003.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-11-17T05:18:11.826 INFO:teuthology.orchestra.run.smithi003.stdout:Installing module profiles: 2023-11-17T05:18:11.826 INFO:teuthology.orchestra.run.smithi003.stdout: container-tools/common 2023-11-17T05:18:11.826 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:18:11.826 INFO:teuthology.orchestra.run.smithi003.stdout:Enabling module streams: 2023-11-17T05:18:11.826 INFO:teuthology.orchestra.run.smithi003.stdout: container-tools rhel8 2023-11-17T05:18:11.826 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:18:11.827 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-11-17T05:18:11.827 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:18:11.827 INFO:teuthology.orchestra.run.smithi003.stdout:Install 13 Packages 2023-11-17T05:18:11.827 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:18:11.827 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 27 M 2023-11-17T05:18:11.827 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 89 M 2023-11-17T05:18:11.828 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-11-17T05:18:11.830 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:18:11.830 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2023-11-17T05:18:11.830 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:18:11.831 INFO:teuthology.orchestra.run.smithi160.stdout:Installing group/module packages: 2023-11-17T05:18:11.831 INFO:teuthology.orchestra.run.smithi160.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2023-11-17T05:18:11.831 INFO:teuthology.orchestra.run.smithi160.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2023-11-17T05:18:11.831 INFO:teuthology.orchestra.run.smithi160.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2023-11-17T05:18:11.831 INFO:teuthology.orchestra.run.smithi160.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2023-11-17T05:18:11.831 INFO:teuthology.orchestra.run.smithi160.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2023-11-17T05:18:11.832 INFO:teuthology.orchestra.run.smithi160.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2023-11-17T05:18:11.832 INFO:teuthology.orchestra.run.smithi160.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2023-11-17T05:18:11.832 INFO:teuthology.orchestra.run.smithi160.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2023-11-17T05:18:11.832 INFO:teuthology.orchestra.run.smithi160.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2023-11-17T05:18:11.832 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2023-11-17T05:18:11.832 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-11-17T05:18:11.833 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-11-17T05:18:11.833 INFO:teuthology.orchestra.run.smithi160.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-11-17T05:18:11.833 INFO:teuthology.orchestra.run.smithi160.stdout:Installing weak dependencies: 2023-11-17T05:18:11.833 INFO:teuthology.orchestra.run.smithi160.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-11-17T05:18:11.833 INFO:teuthology.orchestra.run.smithi160.stdout:Installing module profiles: 2023-11-17T05:18:11.833 INFO:teuthology.orchestra.run.smithi160.stdout: container-tools/common 2023-11-17T05:18:11.834 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:18:11.834 INFO:teuthology.orchestra.run.smithi160.stdout:Enabling module streams: 2023-11-17T05:18:11.907 INFO:teuthology.orchestra.run.smithi160.stdout: container-tools rhel8 2023-11-17T05:18:11.908 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:18:11.908 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-11-17T05:18:11.908 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:18:11.908 INFO:teuthology.orchestra.run.smithi160.stdout:Install 13 Packages 2023-11-17T05:18:11.908 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:18:11.908 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 27 M 2023-11-17T05:18:11.909 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 89 M 2023-11-17T05:18:11.909 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-11-17T05:18:11.909 INFO:teuthology.orchestra.run.smithi160.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 2.7 MB/s | 750 kB 00:00 2023-11-17T05:18:11.909 INFO:teuthology.orchestra.run.smithi160.stdout:(2/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 3.2 MB/s | 1.0 MB 00:00 2023-11-17T05:18:11.909 INFO:teuthology.orchestra.run.smithi160.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 648 kB/s | 38 kB 00:00 2023-11-17T05:18:11.909 INFO:teuthology.orchestra.run.smithi160.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 4.0 MB/s | 241 kB 00:00 2023-11-17T05:18:11.910 INFO:teuthology.orchestra.run.smithi160.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 2.1 MB/s | 178 kB 00:00 2023-11-17T05:18:11.910 INFO:teuthology.orchestra.run.smithi160.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 371 kB/s | 25 kB 00:00 2023-11-17T05:18:11.910 INFO:teuthology.orchestra.run.smithi160.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.2 MB/s | 94 kB 00:00 2023-11-17T05:18:11.910 INFO:teuthology.orchestra.run.smithi160.stdout:(8/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 12 MB/s | 3.9 MB 00:00 2023-11-17T05:18:11.910 INFO:teuthology.orchestra.run.smithi160.stdout:(9/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 12 MB/s | 9.6 MB 00:00 2023-11-17T05:18:11.910 INFO:teuthology.orchestra.run.smithi160.stdout:(10/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 11 MB/s | 2.5 MB 00:00 2023-11-17T05:18:11.911 INFO:teuthology.orchestra.run.smithi160.stdout:(11/13): udica-0.2.6-20.module_el8+712+4cd1bd69 630 kB/s | 53 kB 00:00 2023-11-17T05:18:11.911 INFO:teuthology.orchestra.run.smithi160.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 822 kB/s | 41 kB 00:00 2023-11-17T05:18:12.014 INFO:teuthology.orchestra.run.smithi160.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 15 MB/s | 8.1 MB 00:00 2023-11-17T05:18:12.014 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-11-17T05:18:12.014 INFO:teuthology.orchestra.run.smithi160.stdout:Total 15 MB/s | 27 MB 00:01 2023-11-17T05:18:12.285 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-11-17T05:18:12.306 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-11-17T05:18:12.306 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-11-17T05:18:12.513 INFO:teuthology.orchestra.run.smithi003.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 2.8 MB/s | 750 kB 00:00 2023-11-17T05:18:12.546 INFO:teuthology.orchestra.run.smithi003.stdout:(2/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 3.4 MB/s | 1.0 MB 00:00 2023-11-17T05:18:12.563 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-11-17T05:18:12.566 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-11-17T05:18:12.574 INFO:teuthology.orchestra.run.smithi003.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 649 kB/s | 38 kB 00:00 2023-11-17T05:18:12.622 INFO:teuthology.orchestra.run.smithi003.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 3.1 MB/s | 241 kB 00:00 2023-11-17T05:18:12.681 INFO:teuthology.orchestra.run.smithi003.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 3.0 MB/s | 178 kB 00:00 2023-11-17T05:18:12.739 INFO:teuthology.orchestra.run.smithi003.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 426 kB/s | 25 kB 00:00 2023-11-17T05:18:12.806 INFO:teuthology.orchestra.run.smithi003.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.4 MB/s | 94 kB 00:00 2023-11-17T05:18:12.873 INFO:teuthology.orchestra.run.smithi003.stdout:(8/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 13 MB/s | 3.9 MB 00:00 2023-11-17T05:18:13.015 INFO:teuthology.orchestra.run.smithi003.stdout:(9/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 13 MB/s | 9.6 MB 00:00 2023-11-17T05:18:13.082 INFO:teuthology.orchestra.run.smithi003.stdout:(10/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 12 MB/s | 2.5 MB 00:00 2023-11-17T05:18:13.107 INFO:teuthology.orchestra.run.smithi003.stdout:(11/13): udica-0.2.6-20.module_el8+712+4cd1bd69 572 kB/s | 53 kB 00:00 2023-11-17T05:18:13.141 INFO:teuthology.orchestra.run.smithi003.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 700 kB/s | 41 kB 00:00 2023-11-17T05:18:13.262 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-11-17T05:18:13.308 INFO:teuthology.orchestra.run.smithi003.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 16 MB/s | 8.1 MB 00:00 2023-11-17T05:18:13.308 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-11-17T05:18:13.308 INFO:teuthology.orchestra.run.smithi003.stdout:Total 12 MB/s | 27 MB 00:02 2023-11-17T05:18:13.513 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-11-17T05:18:13.605 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-11-17T05:18:13.630 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-11-17T05:18:13.630 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-11-17T05:18:13.674 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-11-17T05:18:13.892 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-11-17T05:18:13.923 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-11-17T05:18:13.926 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-11-17T05:18:14.109 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-17T05:18:14.642 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-11-17T05:18:14.665 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2023-11-17T05:18:14.850 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-11-17T05:18:15.010 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-11-17T05:18:15.213 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-11-17T05:18:15.444 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-17T05:18:15.644 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-17T05:18:15.904 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2023-11-17T05:18:16.000 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2023-11-17T05:18:16.095 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2023-11-17T05:18:16.381 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2023-11-17T05:18:16.786 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2023-11-17T05:18:17.059 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-17T05:18:17.293 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2023-11-17T05:18:17.506 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2023-11-17T05:18:17.677 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-17T05:18:17.773 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2023-11-17T05:18:17.883 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2023-11-17T05:18:18.090 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-17T05:18:18.177 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2023-11-17T05:18:18.536 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-17T05:18:18.536 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2023-11-17T05:18:18.536 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2023-11-17T05:18:18.536 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2023-11-17T05:18:18.536 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-17T05:18:18.537 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2023-11-17T05:18:18.537 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-17T05:18:18.537 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2023-11-17T05:18:18.537 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-11-17T05:18:18.537 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-11-17T05:18:18.537 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2023-11-17T05:18:18.537 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-17T05:18:18.537 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2023-11-17T05:18:19.124 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-17T05:18:19.359 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2023-11-17T05:18:19.360 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-11-17T05:18:19.361 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:18:19.361 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-11-17T05:18:19.361 INFO:teuthology.orchestra.run.smithi160.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-17T05:18:19.361 INFO:teuthology.orchestra.run.smithi160.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-17T05:18:19.361 INFO:teuthology.orchestra.run.smithi160.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2023-11-17T05:18:19.361 INFO:teuthology.orchestra.run.smithi160.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-11-17T05:18:19.361 INFO:teuthology.orchestra.run.smithi160.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2023-11-17T05:18:19.361 INFO:teuthology.orchestra.run.smithi160.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2023-11-17T05:18:19.361 INFO:teuthology.orchestra.run.smithi160.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2023-11-17T05:18:19.361 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-11-17T05:18:19.361 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-11-17T05:18:19.361 INFO:teuthology.orchestra.run.smithi160.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2023-11-17T05:18:19.361 INFO:teuthology.orchestra.run.smithi160.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2023-11-17T05:18:19.361 INFO:teuthology.orchestra.run.smithi160.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2023-11-17T05:18:19.362 INFO:teuthology.orchestra.run.smithi160.stdout: yajl-2.1.0-12.el8.x86_64 2023-11-17T05:18:19.362 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:18:19.362 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-11-17T05:18:19.549 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-17T05:18:19.651 DEBUG:teuthology.parallel:result is None 2023-11-17T05:18:20.022 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-17T05:18:20.039 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2023-11-17T05:18:20.040 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2023-11-17T05:18:20.040 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2023-11-17T05:18:20.040 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-17T05:18:20.040 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2023-11-17T05:18:20.040 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-17T05:18:20.040 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2023-11-17T05:18:20.040 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-11-17T05:18:20.040 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-11-17T05:18:20.040 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2023-11-17T05:18:20.040 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-17T05:18:20.040 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2023-11-17T05:18:20.908 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-11-17T05:18:20.908 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:18:20.908 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-11-17T05:18:20.908 INFO:teuthology.orchestra.run.smithi003.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-17T05:18:20.908 INFO:teuthology.orchestra.run.smithi003.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-17T05:18:20.908 INFO:teuthology.orchestra.run.smithi003.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2023-11-17T05:18:20.908 INFO:teuthology.orchestra.run.smithi003.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-11-17T05:18:20.908 INFO:teuthology.orchestra.run.smithi003.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2023-11-17T05:18:20.908 INFO:teuthology.orchestra.run.smithi003.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2023-11-17T05:18:20.909 INFO:teuthology.orchestra.run.smithi003.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2023-11-17T05:18:20.909 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-11-17T05:18:20.909 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-11-17T05:18:20.909 INFO:teuthology.orchestra.run.smithi003.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2023-11-17T05:18:20.909 INFO:teuthology.orchestra.run.smithi003.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2023-11-17T05:18:20.909 INFO:teuthology.orchestra.run.smithi003.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2023-11-17T05:18:20.909 INFO:teuthology.orchestra.run.smithi003.stdout: yajl-2.1.0-12.el8.x86_64 2023-11-17T05:18:20.909 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:18:20.909 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-17T05:18:21.180 DEBUG:teuthology.parallel:result is None 2023-11-17T05:18:21.181 INFO:teuthology.run_tasks:Running task pexec... 2023-11-17T05:18:21.190 INFO:teuthology.task.pexec:Executing custom commands... 2023-11-17T05:18:21.191 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi003.front.sepia.ceph.com 2023-11-17T05:18:21.191 DEBUG:teuthology.orchestra.run.smithi003:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-17T05:18:21.191 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi160.front.sepia.ceph.com 2023-11-17T05:18:21.191 DEBUG:teuthology.orchestra.run.smithi160:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-17T05:18:21.357 DEBUG:teuthology.parallel:result is None 2023-11-17T05:18:21.438 DEBUG:teuthology.parallel:result is None 2023-11-17T05:18:21.438 INFO:teuthology.run_tasks:Running task install... 2023-11-17T05:18:21.447 DEBUG:teuthology.task.install:project ceph 2023-11-17T05:18:21.448 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'ed3c8e926b9de93026c8a7c1d86be5d6821b52ac'}} 2023-11-17T05:18:21.448 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'flavor': 'default', 'sha1': 'ed3c8e926b9de93026c8a7c1d86be5d6821b52ac'} 2023-11-17T05:18:21.448 INFO:teuthology.task.install:Using flavor: default 2023-11-17T05:18:21.456 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-11-17T05:18:21.459 INFO:teuthology.task.install:extra packages: [] 2023-11-17T05:18:21.459 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ed3c8e926b9de93026c8a7c1d86be5d6821b52ac', 'tag': 'v16.2.4', 'wait_for_package': False} 2023-11-17T05:18:21.459 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-17T05:18:21.459 INFO:teuthology.packaging:ref: None 2023-11-17T05:18:21.460 INFO:teuthology.packaging:tag: v16.2.4 2023-11-17T05:18:21.460 INFO:teuthology.packaging:branch: None 2023-11-17T05:18:21.460 INFO:teuthology.packaging:sha1: ed3c8e926b9de93026c8a7c1d86be5d6821b52ac 2023-11-17T05:18:21.668 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-17T05:18:21.668 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-17T05:18:21.671 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ed3c8e926b9de93026c8a7c1d86be5d6821b52ac', 'tag': 'v16.2.4', 'wait_for_package': False} 2023-11-17T05:18:21.671 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-17T05:18:21.671 INFO:teuthology.packaging:ref: None 2023-11-17T05:18:21.671 INFO:teuthology.packaging:tag: v16.2.4 2023-11-17T05:18:21.671 INFO:teuthology.packaging:branch: None 2023-11-17T05:18:21.671 INFO:teuthology.packaging:sha1: ed3c8e926b9de93026c8a7c1d86be5d6821b52ac 2023-11-17T05:18:21.901 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-17T05:18:21.902 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-17T05:18:22.073 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2023-11-17T05:18:22.073 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2023-11-17T05:18:22.108 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-17T05:18:22.108 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-11-17T05:18:22.109 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-17T05:18:22.139 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2023-11-17T05:18:22.139 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2023-11-17T05:18:22.142 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-11-17T05:18:22.143 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-17T05:18:22.143 INFO:teuthology.packaging:ref: None 2023-11-17T05:18:22.143 INFO:teuthology.packaging:tag: v16.2.4 2023-11-17T05:18:22.143 INFO:teuthology.packaging:branch: None 2023-11-17T05:18:22.143 INFO:teuthology.packaging:sha1: ed3c8e926b9de93026c8a7c1d86be5d6821b52ac 2023-11-17T05:18:22.143 DEBUG:teuthology.orchestra.run.smithi160:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-17T05:18:22.168 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-17T05:18:22.169 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-11-17T05:18:22.169 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-17T05:18:22.207 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-11-17T05:18:22.207 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-17T05:18:22.207 INFO:teuthology.packaging:ref: None 2023-11-17T05:18:22.207 INFO:teuthology.packaging:tag: v16.2.4 2023-11-17T05:18:22.207 INFO:teuthology.packaging:branch: None 2023-11-17T05:18:22.207 INFO:teuthology.packaging:sha1: ed3c8e926b9de93026c8a7c1d86be5d6821b52ac 2023-11-17T05:18:22.207 DEBUG:teuthology.orchestra.run.smithi003:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-17T05:18:22.214 DEBUG:teuthology.orchestra.run.smithi160:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-17T05:18:22.283 DEBUG:teuthology.orchestra.run.smithi003:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-17T05:18:22.294 DEBUG:teuthology.orchestra.run.smithi160:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-17T05:18:22.363 INFO:teuthology.orchestra.run.smithi160.stdout:check_obsoletes = 1 2023-11-17T05:18:22.365 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum clean all 2023-11-17T05:18:22.372 DEBUG:teuthology.orchestra.run.smithi003:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-17T05:18:22.411 INFO:teuthology.orchestra.run.smithi003.stdout:check_obsoletes = 1 2023-11-17T05:18:22.412 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum clean all 2023-11-17T05:18:22.649 INFO:teuthology.orchestra.run.smithi160.stdout:76 files removed 2023-11-17T05:18:22.679 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-radosgw 2023-11-17T05:18:22.725 INFO:teuthology.orchestra.run.smithi003.stdout:76 files removed 2023-11-17T05:18:22.761 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-radosgw 2023-11-17T05:18:23.163 INFO:teuthology.orchestra.run.smithi160.stdout:ceph packages for x86_64 951 kB/s | 74 kB 00:00 2023-11-17T05:18:23.305 INFO:teuthology.orchestra.run.smithi160.stdout:ceph noarch packages 145 kB/s | 15 kB 00:00 2023-11-17T05:18:23.306 INFO:teuthology.orchestra.run.smithi003.stdout:ceph packages for x86_64 959 kB/s | 74 kB 00:00 2023-11-17T05:18:23.397 INFO:teuthology.orchestra.run.smithi160.stdout:ceph source packages 28 kB/s | 1.7 kB 00:00 2023-11-17T05:18:23.465 INFO:teuthology.orchestra.run.smithi003.stdout:ceph noarch packages 131 kB/s | 15 kB 00:00 2023-11-17T05:18:23.583 INFO:teuthology.orchestra.run.smithi003.stdout:ceph source packages 21 kB/s | 1.7 kB 00:00 2023-11-17T05:18:24.806 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 8 - AppStream 25 MB/s | 34 MB 00:01 2023-11-17T05:18:25.141 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 8 - AppStream 22 MB/s | 34 MB 00:01 2023-11-17T05:18:32.256 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 8 - BaseOS 42 MB/s | 53 MB 00:01 2023-11-17T05:18:33.275 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 8 - BaseOS 40 MB/s | 53 MB 00:01 2023-11-17T05:18:40.098 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 8 - Extras 62 kB/s | 18 kB 00:00 2023-11-17T05:18:40.698 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 8 - Extras common packages 16 kB/s | 6.9 kB 00:00 2023-11-17T05:18:42.109 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 8 - PowerTools 5.3 MB/s | 6.6 MB 00:01 2023-11-17T05:18:42.183 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 8 - Extras 41 kB/s | 18 kB 00:00 2023-11-17T05:18:42.600 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 8 - Extras common packages 33 kB/s | 6.9 kB 00:00 2023-11-17T05:18:43.551 INFO:teuthology.orchestra.run.smithi160.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2023-11-17T05:18:43.968 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 8 - PowerTools 5.6 MB/s | 6.6 MB 00:01 2023-11-17T05:18:44.410 INFO:teuthology.orchestra.run.smithi160.stdout:Extra Packages for Enterprise Linux 23 MB/s | 16 MB 00:00 2023-11-17T05:18:45.537 INFO:teuthology.orchestra.run.smithi003.stdout:Copr repo for python3-asyncssh owned by ceph 28 kB/s | 3.5 kB 00:00 2023-11-17T05:18:48.203 INFO:teuthology.orchestra.run.smithi003.stdout:Extra Packages for Enterprise Linux 6.4 MB/s | 16 MB 00:02 2023-11-17T05:18:48.411 INFO:teuthology.orchestra.run.smithi160.stdout:lab-extras 426 kB/s | 24 kB 00:00 2023-11-17T05:18:52.504 INFO:teuthology.orchestra.run.smithi003.stdout:lab-extras 354 kB/s | 24 kB 00:00 2023-11-17T05:18:52.969 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-11-17T05:18:52.972 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:18:52.972 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2023-11-17T05:18:52.972 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:18:52.972 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-11-17T05:18:52.972 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2023-11-17T05:18:52.972 INFO:teuthology.orchestra.run.smithi160.stdout:Upgrading: 2023-11-17T05:18:52.973 INFO:teuthology.orchestra.run.smithi160.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-11-17T05:18:52.973 INFO:teuthology.orchestra.run.smithi160.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2023-11-17T05:18:52.973 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2023-11-17T05:18:52.973 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2023-11-17T05:18:52.973 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2023-11-17T05:18:52.973 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2023-11-17T05:18:52.973 INFO:teuthology.orchestra.run.smithi160.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2023-11-17T05:18:52.973 INFO:teuthology.orchestra.run.smithi160.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2023-11-17T05:18:52.973 INFO:teuthology.orchestra.run.smithi160.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-17T05:18:52.974 INFO:teuthology.orchestra.run.smithi160.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-11-17T05:18:52.974 INFO:teuthology.orchestra.run.smithi160.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2023-11-17T05:18:52.974 INFO:teuthology.orchestra.run.smithi160.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-17T05:18:52.974 INFO:teuthology.orchestra.run.smithi160.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2023-11-17T05:18:52.974 INFO:teuthology.orchestra.run.smithi160.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-11-17T05:18:52.974 INFO:teuthology.orchestra.run.smithi160.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2023-11-17T05:18:52.974 INFO:teuthology.orchestra.run.smithi160.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-11-17T05:18:52.974 INFO:teuthology.orchestra.run.smithi160.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2023-11-17T05:18:52.974 INFO:teuthology.orchestra.run.smithi160.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-17T05:18:52.974 INFO:teuthology.orchestra.run.smithi160.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-11-17T05:18:52.974 INFO:teuthology.orchestra.run.smithi160.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2023-11-17T05:18:52.974 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2023-11-17T05:18:52.975 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2023-11-17T05:18:52.975 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2023-11-17T05:18:52.975 INFO:teuthology.orchestra.run.smithi160.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-11-17T05:18:52.975 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2023-11-17T05:18:52.975 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2023-11-17T05:18:52.975 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2023-11-17T05:18:52.975 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:18:52.975 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-11-17T05:18:52.975 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:18:52.975 INFO:teuthology.orchestra.run.smithi160.stdout:Install 25 Packages 2023-11-17T05:18:52.976 INFO:teuthology.orchestra.run.smithi160.stdout:Upgrade 2 Packages 2023-11-17T05:18:52.976 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:18:52.976 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 59 M 2023-11-17T05:18:52.976 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-11-17T05:18:53.421 INFO:teuthology.orchestra.run.smithi160.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 48 MB/s | 6.0 MB 00:00 2023-11-17T05:18:53.446 INFO:teuthology.orchestra.run.smithi160.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 999 kB/s | 25 kB 00:00 2023-11-17T05:18:53.480 INFO:teuthology.orchestra.run.smithi160.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 23 MB/s | 801 kB 00:00 2023-11-17T05:18:53.547 INFO:teuthology.orchestra.run.smithi160.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 7.1 MB/s | 486 kB 00:00 2023-11-17T05:18:53.655 INFO:teuthology.orchestra.run.smithi160.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 38 MB/s | 4.1 MB 00:00 2023-11-17T05:18:53.680 INFO:teuthology.orchestra.run.smithi160.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.8 MB/s | 44 kB 00:00 2023-11-17T05:18:53.705 INFO:teuthology.orchestra.run.smithi160.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2023-11-17T05:18:53.731 INFO:teuthology.orchestra.run.smithi160.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.3 MB/s | 213 kB 00:00 2023-11-17T05:18:53.764 INFO:teuthology.orchestra.run.smithi160.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2023-11-17T05:18:53.798 INFO:teuthology.orchestra.run.smithi160.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 11 MB/s | 368 kB 00:00 2023-11-17T05:18:53.858 INFO:teuthology.orchestra.run.smithi160.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 1.9 MB/s | 114 kB 00:00 2023-11-17T05:18:53.999 INFO:teuthology.orchestra.run.smithi160.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 18 MB/s | 13 MB 00:00 2023-11-17T05:18:54.274 INFO:teuthology.orchestra.run.smithi160.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 24 MB/s | 24 MB 00:00 2023-11-17T05:18:54.299 INFO:teuthology.orchestra.run.smithi160.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 329 kB/s | 145 kB 00:00 2023-11-17T05:18:54.359 INFO:teuthology.orchestra.run.smithi160.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 752 kB/s | 44 kB 00:00 2023-11-17T05:18:54.383 INFO:teuthology.orchestra.run.smithi160.stdout:(16/27): librdkafka-0.11.4-3.el8.x86_64.rpm 922 kB/s | 354 kB 00:00 2023-11-17T05:18:54.475 INFO:teuthology.orchestra.run.smithi160.stdout:(17/27): daxctl-libs-71.1-7.el8.x86_64.rpm 361 kB/s | 42 kB 00:00 2023-11-17T05:18:54.501 INFO:teuthology.orchestra.run.smithi160.stdout:(18/27): librabbitmq-0.9.0-4.el8.x86_64.rpm 401 kB/s | 47 kB 00:00 2023-11-17T05:18:54.557 INFO:teuthology.orchestra.run.smithi160.stdout:(19/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 922 kB/s | 259 kB 00:00 2023-11-17T05:18:54.585 INFO:teuthology.orchestra.run.smithi160.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 959 kB/s | 80 kB 00:00 2023-11-17T05:18:54.676 INFO:teuthology.orchestra.run.smithi160.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 663 kB/s | 116 kB 00:00 2023-11-17T05:18:54.710 INFO:teuthology.orchestra.run.smithi160.stdout:(22/27): leveldb-1.22-1.el8.x86_64.rpm 1.4 MB/s | 181 kB 00:00 2023-11-17T05:18:54.744 INFO:teuthology.orchestra.run.smithi160.stdout:(23/27): gperftools-libs-2.7-9.el8.x86_64.rpm 1.6 MB/s | 306 kB 00:00 2023-11-17T05:18:54.760 INFO:teuthology.orchestra.run.smithi160.stdout:(24/27): liboath-2.6.2-3.el8.x86_64.rpm 699 kB/s | 59 kB 00:00 2023-11-17T05:18:54.781 INFO:teuthology.orchestra.run.smithi160.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 1.1 MB/s | 75 kB 00:00 2023-11-17T05:18:54.886 INFO:teuthology.orchestra.run.smithi160.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 26 MB/s | 3.7 MB 00:00 2023-11-17T05:18:54.945 INFO:teuthology.orchestra.run.smithi160.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 22 MB/s | 4.0 MB 00:00 2023-11-17T05:18:54.947 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-11-17T05:18:54.947 INFO:teuthology.orchestra.run.smithi160.stdout:Total 30 MB/s | 59 MB 00:01 2023-11-17T05:18:55.011 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-11-17T05:18:55.060 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-11-17T05:18:55.060 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-11-17T05:18:55.413 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-11-17T05:18:55.416 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-11-17T05:18:56.593 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-11-17T05:18:56.837 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2023-11-17T05:18:57.091 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2023-11-17T05:18:57.276 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-11-17T05:18:57.567 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-17T05:18:57.573 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:18:57.573 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2023-11-17T05:18:57.574 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:18:57.574 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-11-17T05:18:57.574 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2023-11-17T05:18:57.574 INFO:teuthology.orchestra.run.smithi003.stdout:Upgrading: 2023-11-17T05:18:57.574 INFO:teuthology.orchestra.run.smithi003.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-11-17T05:18:57.574 INFO:teuthology.orchestra.run.smithi003.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2023-11-17T05:18:57.575 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2023-11-17T05:18:57.575 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2023-11-17T05:18:57.575 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2023-11-17T05:18:57.575 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2023-11-17T05:18:57.575 INFO:teuthology.orchestra.run.smithi003.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2023-11-17T05:18:57.576 INFO:teuthology.orchestra.run.smithi003.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2023-11-17T05:18:57.576 INFO:teuthology.orchestra.run.smithi003.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-17T05:18:57.576 INFO:teuthology.orchestra.run.smithi003.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-11-17T05:18:57.577 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2023-11-17T05:18:57.577 INFO:teuthology.orchestra.run.smithi003.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-17T05:18:57.578 INFO:teuthology.orchestra.run.smithi003.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2023-11-17T05:18:57.578 INFO:teuthology.orchestra.run.smithi003.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-11-17T05:18:57.578 INFO:teuthology.orchestra.run.smithi003.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2023-11-17T05:18:57.578 INFO:teuthology.orchestra.run.smithi003.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-11-17T05:18:57.578 INFO:teuthology.orchestra.run.smithi003.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2023-11-17T05:18:57.581 INFO:teuthology.orchestra.run.smithi003.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-17T05:18:57.581 INFO:teuthology.orchestra.run.smithi003.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-11-17T05:18:57.581 INFO:teuthology.orchestra.run.smithi003.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2023-11-17T05:18:57.581 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2023-11-17T05:18:57.581 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2023-11-17T05:18:57.582 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2023-11-17T05:18:57.582 INFO:teuthology.orchestra.run.smithi003.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-11-17T05:18:57.582 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2023-11-17T05:18:57.582 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2023-11-17T05:18:57.582 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2023-11-17T05:18:57.582 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:18:57.582 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-11-17T05:18:57.582 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:18:57.582 INFO:teuthology.orchestra.run.smithi003.stdout:Install 25 Packages 2023-11-17T05:18:57.582 INFO:teuthology.orchestra.run.smithi003.stdout:Upgrade 2 Packages 2023-11-17T05:18:57.582 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:18:57.582 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 59 M 2023-11-17T05:18:57.582 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-11-17T05:18:57.682 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-11-17T05:18:57.839 INFO:teuthology.orchestra.run.smithi003.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 38 MB/s | 6.0 MB 00:00 2023-11-17T05:18:57.864 INFO:teuthology.orchestra.run.smithi003.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 1.0 MB/s | 25 kB 00:00 2023-11-17T05:18:57.897 INFO:teuthology.orchestra.run.smithi003.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 24 MB/s | 801 kB 00:00 2023-11-17T05:18:57.910 INFO:teuthology.orchestra.run.smithi160.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2023-11-17T05:18:57.931 INFO:teuthology.orchestra.run.smithi003.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2023-11-17T05:18:58.003 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2023-11-17T05:18:58.173 INFO:teuthology.orchestra.run.smithi003.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 17 MB/s | 4.1 MB 00:00 2023-11-17T05:18:58.190 INFO:teuthology.orchestra.run.smithi003.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 2.6 MB/s | 44 kB 00:00 2023-11-17T05:18:58.215 INFO:teuthology.orchestra.run.smithi003.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2023-11-17T05:18:58.240 INFO:teuthology.orchestra.run.smithi003.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.4 MB/s | 213 kB 00:00 2023-11-17T05:18:58.274 INFO:teuthology.orchestra.run.smithi003.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2023-11-17T05:18:58.308 INFO:teuthology.orchestra.run.smithi003.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 11 MB/s | 368 kB 00:00 2023-11-17T05:18:58.333 INFO:teuthology.orchestra.run.smithi003.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2023-11-17T05:18:58.359 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2023-11-17T05:18:58.508 INFO:teuthology.orchestra.run.smithi003.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 15 MB/s | 13 MB 00:00 2023-11-17T05:18:58.535 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-11-17T05:18:58.598 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-11-17T05:18:58.784 INFO:teuthology.orchestra.run.smithi003.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 22 MB/s | 24 MB 00:01 2023-11-17T05:18:58.809 INFO:teuthology.orchestra.run.smithi003.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 305 kB/s | 145 kB 00:00 2023-11-17T05:18:58.816 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/29 2023-11-17T05:18:58.867 INFO:teuthology.orchestra.run.smithi003.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 754 kB/s | 44 kB 00:00 2023-11-17T05:18:58.901 INFO:teuthology.orchestra.run.smithi003.stdout:(16/27): librdkafka-0.11.4-3.el8.x86_64.rpm 903 kB/s | 354 kB 00:00 2023-11-17T05:18:59.018 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/29 2023-11-17T05:18:59.118 INFO:teuthology.orchestra.run.smithi003.stdout:(17/27): daxctl-libs-71.1-7.el8.x86_64.rpm 168 kB/s | 42 kB 00:00 2023-11-17T05:18:59.143 INFO:teuthology.orchestra.run.smithi003.stdout:(18/27): librabbitmq-0.9.0-4.el8.x86_64.rpm 194 kB/s | 47 kB 00:00 2023-11-17T05:18:59.169 INFO:teuthology.orchestra.run.smithi003.stdout:(19/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 675 kB/s | 259 kB 00:00 2023-11-17T05:18:59.194 INFO:teuthology.orchestra.run.smithi003.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 1.0 MB/s | 80 kB 00:00 2023-11-17T05:18:59.440 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/29 2023-11-17T05:18:59.634 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-11-17T05:18:59.723 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-11-17T05:18:59.761 INFO:teuthology.orchestra.run.smithi003.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 188 kB/s | 116 kB 00:00 2023-11-17T05:18:59.948 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2023-11-17T05:18:59.961 INFO:teuthology.orchestra.run.smithi003.stdout:(22/27): leveldb-1.22-1.el8.x86_64.rpm 236 kB/s | 181 kB 00:00 2023-11-17T05:18:59.987 INFO:teuthology.orchestra.run.smithi003.stdout:(23/27): liboath-2.6.2-3.el8.x86_64.rpm 261 kB/s | 59 kB 00:00 2023-11-17T05:19:00.013 INFO:teuthology.orchestra.run.smithi003.stdout:(24/27): gperftools-libs-2.7-9.el8.x86_64.rpm 363 kB/s | 306 kB 00:00 2023-11-17T05:19:00.112 INFO:teuthology.orchestra.run.smithi003.stdout:(25/27): librados2-16.2.4-0.el8.x86_64.rpm 30 MB/s | 3.7 MB 00:00 2023-11-17T05:19:00.134 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2023-11-17T05:19:00.169 INFO:teuthology.orchestra.run.smithi003.stdout:(26/27): libunwind-1.3.1-3.el8.x86_64.rpm 363 kB/s | 75 kB 00:00 2023-11-17T05:19:00.226 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2023-11-17T05:19:00.237 INFO:teuthology.orchestra.run.smithi003.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 18 MB/s | 4.0 MB 00:00 2023-11-17T05:19:00.243 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-11-17T05:19:00.243 INFO:teuthology.orchestra.run.smithi003.stdout:Total 22 MB/s | 59 MB 00:02 2023-11-17T05:19:00.315 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-11-17T05:19:00.369 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-11-17T05:19:00.369 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-11-17T05:19:00.406 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2023-11-17T05:19:00.618 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2023-11-17T05:19:00.818 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2023-11-17T05:19:01.090 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-11-17T05:19:01.090 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-11-17T05:19:01.090 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2023-11-17T05:19:01.090 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-11-17T05:19:01.451 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-11-17T05:19:01.634 INFO:teuthology.orchestra.run.smithi160.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-11-17T05:19:01.767 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-11-17T05:19:01.938 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2023-11-17T05:19:02.152 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2023-11-17T05:19:02.290 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-11-17T05:19:02.359 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-11-17T05:19:02.432 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-11-17T05:19:02.500 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2023-11-17T05:19:02.659 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2023-11-17T05:19:02.743 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2023-11-17T05:19:02.817 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2023-11-17T05:19:02.942 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-11-17T05:19:03.025 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2023-11-17T05:19:03.402 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-11-17T05:19:03.418 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2023-11-17T05:19:03.608 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2023-11-17T05:19:03.895 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2023-11-17T05:19:04.102 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2023-11-17T05:19:04.142 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-11-17T05:19:04.193 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-11-17T05:19:04.411 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/29 2023-11-17T05:19:04.583 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/29 2023-11-17T05:19:05.038 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/29 2023-11-17T05:19:05.208 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-11-17T05:19:05.279 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-11-17T05:19:05.456 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2023-11-17T05:19:05.658 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2023-11-17T05:19:05.723 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2023-11-17T05:19:05.896 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2023-11-17T05:19:06.077 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2023-11-17T05:19:06.097 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-17T05:19:06.266 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2023-11-17T05:19:06.342 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-17T05:19:06.343 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2023-11-17T05:19:06.491 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-11-17T05:19:06.975 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-11-17T05:19:07.040 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-17T05:19:07.133 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-11-17T05:19:07.237 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-11-17T05:19:07.371 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-11-17T05:19:07.408 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2023-11-17T05:19:07.483 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-11-17T05:19:07.607 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2023-11-17T05:19:07.666 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-11-17T05:19:07.791 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-11-17T05:19:07.851 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-11-17T05:19:08.047 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2023-11-17T05:19:08.218 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2023-11-17T05:19:08.415 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2023-11-17T05:19:08.625 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2023-11-17T05:19:11.367 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-17T05:19:11.583 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-17T05:19:12.422 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-17T05:19:12.729 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-11-17T05:19:12.842 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-11-17T05:19:12.982 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-11-17T05:19:26.022 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-11-17T05:19:26.023 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /sys 2023-11-17T05:19:26.023 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /proc 2023-11-17T05:19:26.023 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /dev 2023-11-17T05:19:26.023 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /run 2023-11-17T05:19:26.023 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /mnt 2023-11-17T05:19:26.023 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /var/tmp 2023-11-17T05:19:26.023 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /home 2023-11-17T05:19:26.023 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /tmp 2023-11-17T05:19:26.023 INFO:teuthology.orchestra.run.smithi160.stdout:skipping the directory /dev 2023-11-17T05:19:26.023 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:19:27.335 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-11-17T05:19:27.433 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-11-17T05:19:27.433 INFO:teuthology.orchestra.run.smithi160.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-11-17T05:19:27.650 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-11-17T05:19:27.650 INFO:teuthology.orchestra.run.smithi160.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2023-11-17T05:19:28.368 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2023-11-17T05:19:28.369 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2023-11-17T05:19:28.369 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2023-11-17T05:19:28.369 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2023-11-17T05:19:28.369 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2023-11-17T05:19:28.369 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2023-11-17T05:19:28.369 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2023-11-17T05:19:28.369 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2023-11-17T05:19:28.369 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2023-11-17T05:19:28.369 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2023-11-17T05:19:28.369 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2023-11-17T05:19:28.369 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2023-11-17T05:19:28.370 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2023-11-17T05:19:28.370 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2023-11-17T05:19:28.370 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2023-11-17T05:19:28.370 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 15/29 2023-11-17T05:19:28.370 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2023-11-17T05:19:28.370 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2023-11-17T05:19:28.370 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2023-11-17T05:19:28.370 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 19/29 2023-11-17T05:19:28.370 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2023-11-17T05:19:28.370 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2023-11-17T05:19:28.370 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2023-11-17T05:19:28.370 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2023-11-17T05:19:28.371 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2023-11-17T05:19:28.371 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2023-11-17T05:19:28.371 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2023-11-17T05:19:28.371 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2023-11-17T05:19:28.371 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2023-11-17T05:19:29.583 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2023-11-17T05:19:29.583 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:19:29.583 INFO:teuthology.orchestra.run.smithi160.stdout:Upgraded: 2023-11-17T05:19:29.583 INFO:teuthology.orchestra.run.smithi160.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2023-11-17T05:19:29.583 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-11-17T05:19:29.583 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2023-11-17T05:19:29.583 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2023-11-17T05:19:29.583 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2023-11-17T05:19:29.583 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2023-11-17T05:19:29.584 INFO:teuthology.orchestra.run.smithi160.stdout: daxctl-libs-71.1-7.el8.x86_64 2023-11-17T05:19:29.584 INFO:teuthology.orchestra.run.smithi160.stdout: fmt-6.2.1-1.el8.x86_64 2023-11-17T05:19:29.584 INFO:teuthology.orchestra.run.smithi160.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-17T05:19:29.584 INFO:teuthology.orchestra.run.smithi160.stdout: leveldb-1.22-1.el8.x86_64 2023-11-17T05:19:29.584 INFO:teuthology.orchestra.run.smithi160.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2023-11-17T05:19:29.584 INFO:teuthology.orchestra.run.smithi160.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-17T05:19:29.584 INFO:teuthology.orchestra.run.smithi160.stdout: libpmemobj-1.6.1-1.el8.x86_64 2023-11-17T05:19:29.584 INFO:teuthology.orchestra.run.smithi160.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-11-17T05:19:29.584 INFO:teuthology.orchestra.run.smithi160.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2023-11-17T05:19:29.584 INFO:teuthology.orchestra.run.smithi160.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-17T05:19:29.584 INFO:teuthology.orchestra.run.smithi160.stdout: librgw2-2:16.2.4-0.el8.x86_64 2023-11-17T05:19:29.584 INFO:teuthology.orchestra.run.smithi160.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-17T05:19:29.584 INFO:teuthology.orchestra.run.smithi160.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-17T05:19:29.585 INFO:teuthology.orchestra.run.smithi160.stdout: ndctl-libs-71.1-7.el8.x86_64 2023-11-17T05:19:29.585 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2023-11-17T05:19:29.585 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2023-11-17T05:19:29.585 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2023-11-17T05:19:29.585 INFO:teuthology.orchestra.run.smithi160.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-17T05:19:29.585 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2023-11-17T05:19:29.585 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2023-11-17T05:19:29.585 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2023-11-17T05:19:29.585 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:19:29.585 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-11-17T05:19:29.810 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-test 2023-11-17T05:19:30.666 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:00:42 ago on Fri 17 Nov 2023 05:18:48 AM UTC. 2023-11-17T05:19:32.327 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-11-17T05:19:32.328 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:19:32.328 INFO:teuthology.orchestra.run.smithi160.stdout: Package Architecture Version Repository Size 2023-11-17T05:19:32.328 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:19:32.328 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-11-17T05:19:32.328 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2023-11-17T05:19:32.328 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2023-11-17T05:19:32.328 INFO:teuthology.orchestra.run.smithi160.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-11-17T05:19:32.328 INFO:teuthology.orchestra.run.smithi160.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2023-11-17T05:19:32.328 INFO:teuthology.orchestra.run.smithi160.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-11-17T05:19:32.328 INFO:teuthology.orchestra.run.smithi160.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-11-17T05:19:32.329 INFO:teuthology.orchestra.run.smithi160.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-11-17T05:19:32.329 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:19:32.329 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-11-17T05:19:32.329 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:19:32.329 INFO:teuthology.orchestra.run.smithi160.stdout:Install 6 Packages 2023-11-17T05:19:32.329 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:19:32.329 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 53 M 2023-11-17T05:19:32.330 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 229 M 2023-11-17T05:19:32.330 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-11-17T05:19:32.419 INFO:teuthology.orchestra.run.smithi160.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 3.4 MB/s | 169 kB 00:00 2023-11-17T05:19:32.586 INFO:teuthology.orchestra.run.smithi160.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 944 kB/s | 203 kB 00:00 2023-11-17T05:19:32.636 INFO:teuthology.orchestra.run.smithi160.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 863 kB/s | 187 kB 00:00 2023-11-17T05:19:32.678 INFO:teuthology.orchestra.run.smithi160.stdout:(4/6): socat-1.7.4.1-1.el8.x86_64.rpm 3.4 MB/s | 323 kB 00:00 2023-11-17T05:19:32.703 INFO:teuthology.orchestra.run.smithi160.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.0 MB/s | 71 kB 00:00 2023-11-17T05:19:33.187 INFO:teuthology.orchestra.run.smithi160.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 64 MB/s | 52 MB 00:00 2023-11-17T05:19:33.189 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-11-17T05:19:33.189 INFO:teuthology.orchestra.run.smithi160.stdout:Total 61 MB/s | 53 MB 00:00 2023-11-17T05:19:33.232 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-11-17T05:19:33.253 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-11-17T05:19:33.253 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-11-17T05:19:33.562 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-11-17T05:19:33.565 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-11-17T05:19:34.068 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-11-17T05:19:34.122 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-11-17T05:19:34.122 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /sys 2023-11-17T05:19:34.122 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /proc 2023-11-17T05:19:34.122 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /dev 2023-11-17T05:19:34.122 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /run 2023-11-17T05:19:34.122 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /mnt 2023-11-17T05:19:34.122 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /var/tmp 2023-11-17T05:19:34.122 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /home 2023-11-17T05:19:34.122 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /tmp 2023-11-17T05:19:34.122 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /dev 2023-11-17T05:19:34.123 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:19:34.298 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-11-17T05:19:34.515 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-11-17T05:19:34.704 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-17T05:19:34.783 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-17T05:19:34.978 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2023-11-17T05:19:35.137 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-11-17T05:19:35.584 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-11-17T05:19:35.666 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-11-17T05:19:35.666 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-11-17T05:19:35.865 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-11-17T05:19:35.865 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2023-11-17T05:19:36.602 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2023-11-17T05:19:36.602 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2023-11-17T05:19:36.602 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2023-11-17T05:19:36.602 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2023-11-17T05:19:36.602 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2023-11-17T05:19:36.602 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2023-11-17T05:19:36.602 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2023-11-17T05:19:36.602 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2023-11-17T05:19:36.603 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2023-11-17T05:19:36.603 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2023-11-17T05:19:36.603 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2023-11-17T05:19:36.603 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2023-11-17T05:19:36.603 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2023-11-17T05:19:36.603 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2023-11-17T05:19:36.603 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2023-11-17T05:19:36.603 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 15/29 2023-11-17T05:19:36.603 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2023-11-17T05:19:36.603 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2023-11-17T05:19:36.603 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2023-11-17T05:19:36.604 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 19/29 2023-11-17T05:19:36.604 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2023-11-17T05:19:36.604 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2023-11-17T05:19:36.604 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2023-11-17T05:19:36.604 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2023-11-17T05:19:36.604 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2023-11-17T05:19:36.604 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2023-11-17T05:19:36.604 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2023-11-17T05:19:36.604 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2023-11-17T05:19:36.604 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2023-11-17T05:19:37.911 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2023-11-17T05:19:37.911 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:19:37.912 INFO:teuthology.orchestra.run.smithi003.stdout:Upgraded: 2023-11-17T05:19:37.912 INFO:teuthology.orchestra.run.smithi003.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2023-11-17T05:19:37.912 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-11-17T05:19:37.912 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2023-11-17T05:19:37.912 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2023-11-17T05:19:37.912 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2023-11-17T05:19:37.913 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2023-11-17T05:19:37.913 INFO:teuthology.orchestra.run.smithi003.stdout: daxctl-libs-71.1-7.el8.x86_64 2023-11-17T05:19:37.913 INFO:teuthology.orchestra.run.smithi003.stdout: fmt-6.2.1-1.el8.x86_64 2023-11-17T05:19:37.913 INFO:teuthology.orchestra.run.smithi003.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-17T05:19:37.913 INFO:teuthology.orchestra.run.smithi003.stdout: leveldb-1.22-1.el8.x86_64 2023-11-17T05:19:37.913 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2023-11-17T05:19:37.914 INFO:teuthology.orchestra.run.smithi003.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-17T05:19:37.914 INFO:teuthology.orchestra.run.smithi003.stdout: libpmemobj-1.6.1-1.el8.x86_64 2023-11-17T05:19:37.914 INFO:teuthology.orchestra.run.smithi003.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-11-17T05:19:37.914 INFO:teuthology.orchestra.run.smithi003.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2023-11-17T05:19:37.914 INFO:teuthology.orchestra.run.smithi003.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-17T05:19:37.915 INFO:teuthology.orchestra.run.smithi003.stdout: librgw2-2:16.2.4-0.el8.x86_64 2023-11-17T05:19:37.915 INFO:teuthology.orchestra.run.smithi003.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-17T05:19:37.915 INFO:teuthology.orchestra.run.smithi003.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-17T05:19:37.915 INFO:teuthology.orchestra.run.smithi003.stdout: ndctl-libs-71.1-7.el8.x86_64 2023-11-17T05:19:37.915 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2023-11-17T05:19:37.916 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2023-11-17T05:19:37.916 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2023-11-17T05:19:37.916 INFO:teuthology.orchestra.run.smithi003.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-17T05:19:37.916 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2023-11-17T05:19:37.916 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2023-11-17T05:19:37.916 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2023-11-17T05:19:37.917 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:19:37.917 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-17T05:19:38.229 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-test 2023-11-17T05:19:39.217 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:47 ago on Fri 17 Nov 2023 05:18:52 AM UTC. 2023-11-17T05:19:40.014 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-11-17T05:19:40.285 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-11-17T05:19:40.572 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-11-17T05:19:40.573 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2023-11-17T05:19:40.573 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2023-11-17T05:19:40.573 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2023-11-17T05:19:40.573 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-11-17T05:19:40.573 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-11-17T05:19:41.070 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-17T05:19:41.075 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:19:41.075 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2023-11-17T05:19:41.075 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:19:41.075 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-11-17T05:19:41.075 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2023-11-17T05:19:41.075 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2023-11-17T05:19:41.076 INFO:teuthology.orchestra.run.smithi003.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-11-17T05:19:41.076 INFO:teuthology.orchestra.run.smithi003.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2023-11-17T05:19:41.076 INFO:teuthology.orchestra.run.smithi003.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-11-17T05:19:41.076 INFO:teuthology.orchestra.run.smithi003.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-11-17T05:19:41.076 INFO:teuthology.orchestra.run.smithi003.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-11-17T05:19:41.076 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:19:41.076 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-11-17T05:19:41.077 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:19:41.077 INFO:teuthology.orchestra.run.smithi003.stdout:Install 6 Packages 2023-11-17T05:19:41.077 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:19:41.077 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 53 M 2023-11-17T05:19:41.077 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 229 M 2023-11-17T05:19:41.077 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-11-17T05:19:41.194 INFO:teuthology.orchestra.run.smithi003.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 2.1 MB/s | 169 kB 00:00 2023-11-17T05:19:41.272 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-11-17T05:19:41.272 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:19:41.272 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-11-17T05:19:41.272 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2023-11-17T05:19:41.272 INFO:teuthology.orchestra.run.smithi160.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2023-11-17T05:19:41.273 INFO:teuthology.orchestra.run.smithi160.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2023-11-17T05:19:41.273 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:19:41.273 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-11-17T05:19:41.319 INFO:teuthology.orchestra.run.smithi003.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 989 kB/s | 203 kB 00:00 2023-11-17T05:19:41.377 INFO:teuthology.orchestra.run.smithi003.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2023-11-17T05:19:41.436 INFO:teuthology.orchestra.run.smithi003.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2023-11-17T05:19:41.461 INFO:teuthology.orchestra.run.smithi003.stdout:(5/6): socat-1.7.4.1-1.el8.x86_64.rpm 2.2 MB/s | 323 kB 00:00 2023-11-17T05:19:41.480 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph 2023-11-17T05:19:41.970 INFO:teuthology.orchestra.run.smithi003.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 61 MB/s | 52 MB 00:00 2023-11-17T05:19:41.972 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-11-17T05:19:41.972 INFO:teuthology.orchestra.run.smithi003.stdout:Total 59 MB/s | 53 MB 00:00 2023-11-17T05:19:42.018 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-11-17T05:19:42.043 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-11-17T05:19:42.044 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-11-17T05:19:42.347 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:00:54 ago on Fri 17 Nov 2023 05:18:48 AM UTC. 2023-11-17T05:19:42.381 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-11-17T05:19:42.384 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-11-17T05:19:42.887 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-11-17T05:19:43.119 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-11-17T05:19:43.308 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-11-17T05:19:43.635 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-17T05:19:43.719 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-17T05:19:43.895 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2023-11-17T05:19:44.021 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-11-17T05:19:44.024 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:19:44.024 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2023-11-17T05:19:44.025 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:19:44.025 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-11-17T05:19:44.025 INFO:teuthology.orchestra.run.smithi160.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2023-11-17T05:19:44.025 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2023-11-17T05:19:44.025 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2023-11-17T05:19:44.025 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2023-11-17T05:19:44.025 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2023-11-17T05:19:44.025 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2023-11-17T05:19:44.025 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2023-11-17T05:19:44.025 INFO:teuthology.orchestra.run.smithi160.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-17T05:19:44.026 INFO:teuthology.orchestra.run.smithi160.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-17T05:19:44.026 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-11-17T05:19:44.026 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-17T05:19:44.026 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-17T05:19:44.026 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-11-17T05:19:44.026 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-17T05:19:44.026 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-17T05:19:44.026 INFO:teuthology.orchestra.run.smithi160.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-17T05:19:44.026 INFO:teuthology.orchestra.run.smithi160.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-11-17T05:19:44.026 INFO:teuthology.orchestra.run.smithi160.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-11-17T05:19:44.026 INFO:teuthology.orchestra.run.smithi160.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-17T05:19:44.026 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-17T05:19:44.027 INFO:teuthology.orchestra.run.smithi160.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-17T05:19:44.027 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-11-17T05:19:44.027 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-11-17T05:19:44.027 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-11-17T05:19:44.027 INFO:teuthology.orchestra.run.smithi160.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-17T05:19:44.027 INFO:teuthology.orchestra.run.smithi160.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-17T05:19:44.027 INFO:teuthology.orchestra.run.smithi160.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-17T05:19:44.027 INFO:teuthology.orchestra.run.smithi160.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-17T05:19:44.027 INFO:teuthology.orchestra.run.smithi160.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-17T05:19:44.027 INFO:teuthology.orchestra.run.smithi160.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-17T05:19:44.027 INFO:teuthology.orchestra.run.smithi160.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-17T05:19:44.028 INFO:teuthology.orchestra.run.smithi160.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-11-17T05:19:44.028 INFO:teuthology.orchestra.run.smithi160.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-17T05:19:44.028 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:19:44.028 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-11-17T05:19:44.028 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:19:44.028 INFO:teuthology.orchestra.run.smithi160.stdout:Install 32 Packages 2023-11-17T05:19:44.028 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:19:44.029 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 31 M 2023-11-17T05:19:44.029 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 111 M 2023-11-17T05:19:44.029 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-11-17T05:19:44.035 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-11-17T05:19:44.338 INFO:teuthology.orchestra.run.smithi160.stdout:(1/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 33 MB/s | 2.4 MB 00:00 2023-11-17T05:19:44.363 INFO:teuthology.orchestra.run.smithi160.stdout:(2/32): ceph-16.2.4-0.el8.x86_64.rpm 65 kB/s | 6.3 kB 00:00 2023-11-17T05:19:44.433 INFO:teuthology.orchestra.run.smithi160.stdout:(3/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 9.4 MB/s | 1.6 MB 00:00 2023-11-17T05:19:44.472 INFO:teuthology.orchestra.run.smithi160.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 5.6 MB/s | 220 kB 00:00 2023-11-17T05:19:44.547 INFO:teuthology.orchestra.run.smithi160.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 24 MB/s | 5.0 MB 00:00 2023-11-17T05:19:44.806 INFO:teuthology.orchestra.run.smithi160.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 41 MB/s | 18 MB 00:00 2023-11-17T05:19:44.832 INFO:teuthology.orchestra.run.smithi160.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 137 kB/s | 39 kB 00:00 2023-11-17T05:19:44.857 INFO:teuthology.orchestra.run.smithi160.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 409 kB/s | 157 kB 00:00 2023-11-17T05:19:44.915 INFO:teuthology.orchestra.run.smithi160.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 682 kB/s | 57 kB 00:00 2023-11-17T05:19:44.974 INFO:teuthology.orchestra.run.smithi160.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 3.8 MB/s | 457 kB 00:00 2023-11-17T05:19:45.039 INFO:teuthology.orchestra.run.smithi160.stdout:(11/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 444 kB/s | 103 kB 00:00 2023-11-17T05:19:45.091 INFO:teuthology.orchestra.run.smithi160.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.4 MB/s | 249 kB 00:00 2023-11-17T05:19:45.183 INFO:teuthology.orchestra.run.smithi160.stdout:(13/32): python3-cryptography-3.2.1-6.el8.x86_6 3.0 MB/s | 641 kB 00:00 2023-11-17T05:19:45.210 INFO:teuthology.orchestra.run.smithi160.stdout:(14/32): python3-pycparser-2.14-14.el8.noarch.r 646 kB/s | 109 kB 00:00 2023-11-17T05:19:45.367 INFO:teuthology.orchestra.run.smithi160.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 158 kB/s | 44 kB 00:00 2023-11-17T05:19:45.442 INFO:teuthology.orchestra.run.smithi160.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 740 kB/s | 173 kB 00:00 2023-11-17T05:19:45.476 INFO:teuthology.orchestra.run.smithi160.stdout:(17/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 633 kB/s | 185 kB 00:00 2023-11-17T05:19:45.534 INFO:teuthology.orchestra.run.smithi160.stdout:(18/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.2 MB/s | 384 kB 00:00 2023-11-17T05:19:45.560 INFO:teuthology.orchestra.run.smithi160.stdout:(19/32): python3-jaraco-6.2-6.el8.noarch.rpm 95 kB/s | 11 kB 00:00 2023-11-17T05:19:45.585 INFO:teuthology.orchestra.run.smithi160.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 161 kB/s | 18 kB 00:00 2023-11-17T05:19:45.634 INFO:teuthology.orchestra.run.smithi160.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 488 kB/s | 49 kB 00:00 2023-11-17T05:19:45.669 INFO:teuthology.orchestra.run.smithi160.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 540 kB/s | 59 kB 00:00 2023-11-17T05:19:45.694 INFO:teuthology.orchestra.run.smithi160.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.5 MB/s | 283 kB 00:00 2023-11-17T05:19:45.711 INFO:teuthology.orchestra.run.smithi160.stdout:(24/32): python3-portend-2.6-1.el8.noarch.rpm 204 kB/s | 16 kB 00:00 2023-11-17T05:19:45.728 INFO:teuthology.orchestra.run.smithi160.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 318 kB/s | 19 kB 00:00 2023-11-17T05:19:45.762 INFO:teuthology.orchestra.run.smithi160.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 357 kB/s | 24 kB 00:00 2023-11-17T05:19:45.787 INFO:teuthology.orchestra.run.smithi160.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 387 kB/s | 29 kB 00:00 2023-11-17T05:19:45.813 INFO:teuthology.orchestra.run.smithi160.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 329 kB/s | 27 kB 00:00 2023-11-17T05:19:45.837 INFO:teuthology.orchestra.run.smithi160.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 3.3 MB/s | 255 kB 00:00 2023-11-17T05:19:45.863 INFO:teuthology.orchestra.run.smithi160.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 3.2 MB/s | 251 kB 00:00 2023-11-17T05:19:45.896 INFO:teuthology.orchestra.run.smithi160.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.0 MB/s | 85 kB 00:00 2023-11-17T05:19:45.946 INFO:teuthology.orchestra.run.smithi160.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 211 kB/s | 23 kB 00:00 2023-11-17T05:19:45.948 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-11-17T05:19:45.948 INFO:teuthology.orchestra.run.smithi160.stdout:Total 16 MB/s | 31 MB 00:01 2023-11-17T05:19:46.032 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-11-17T05:19:46.071 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-11-17T05:19:46.071 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-11-17T05:19:46.416 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-11-17T05:19:46.420 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-11-17T05:19:47.587 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-11-17T05:19:47.833 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-11-17T05:19:48.060 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-11-17T05:19:48.294 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-11-17T05:19:48.508 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-11-17T05:19:48.723 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-11-17T05:19:48.912 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-11-17T05:19:49.071 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-11-17T05:19:49.274 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-11-17T05:19:49.389 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-11-17T05:19:49.450 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-11-17T05:19:49.634 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-11-17T05:19:49.663 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-11-17T05:19:49.910 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-11-17T05:19:49.951 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-11-17T05:19:49.952 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2023-11-17T05:19:49.952 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2023-11-17T05:19:49.952 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2023-11-17T05:19:49.952 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-11-17T05:19:49.952 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-11-17T05:19:50.183 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-11-17T05:19:50.574 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-11-17T05:19:50.703 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-11-17T05:19:50.703 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:19:50.704 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-11-17T05:19:50.704 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2023-11-17T05:19:50.704 INFO:teuthology.orchestra.run.smithi003.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2023-11-17T05:19:50.704 INFO:teuthology.orchestra.run.smithi003.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2023-11-17T05:19:50.704 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:19:50.704 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-17T05:19:50.863 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-11-17T05:19:50.994 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph 2023-11-17T05:19:51.067 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-11-17T05:19:51.354 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-11-17T05:19:51.535 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-17T05:19:51.827 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-11-17T05:19:51.975 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:59 ago on Fri 17 Nov 2023 05:18:52 AM UTC. 2023-11-17T05:19:52.048 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-11-17T05:19:52.245 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-11-17T05:19:52.458 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-11-17T05:19:52.736 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-11-17T05:19:53.027 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-11-17T05:19:53.280 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-11-17T05:19:53.632 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-11-17T05:19:53.839 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-17T05:19:53.844 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:19:53.845 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2023-11-17T05:19:53.845 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:19:53.845 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-11-17T05:19:53.846 INFO:teuthology.orchestra.run.smithi003.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2023-11-17T05:19:53.846 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2023-11-17T05:19:53.847 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2023-11-17T05:19:53.847 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2023-11-17T05:19:53.847 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2023-11-17T05:19:53.848 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2023-11-17T05:19:53.848 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2023-11-17T05:19:53.848 INFO:teuthology.orchestra.run.smithi003.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-17T05:19:53.848 INFO:teuthology.orchestra.run.smithi003.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-17T05:19:53.848 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-11-17T05:19:53.849 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-17T05:19:53.849 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-17T05:19:53.850 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-11-17T05:19:53.851 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-17T05:19:53.851 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-17T05:19:53.851 INFO:teuthology.orchestra.run.smithi003.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-17T05:19:53.851 INFO:teuthology.orchestra.run.smithi003.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-11-17T05:19:53.851 INFO:teuthology.orchestra.run.smithi003.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-11-17T05:19:53.851 INFO:teuthology.orchestra.run.smithi003.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-17T05:19:53.851 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-17T05:19:53.851 INFO:teuthology.orchestra.run.smithi003.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-17T05:19:53.851 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-11-17T05:19:53.851 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-11-17T05:19:53.851 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-11-17T05:19:53.851 INFO:teuthology.orchestra.run.smithi003.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-17T05:19:53.852 INFO:teuthology.orchestra.run.smithi003.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-17T05:19:53.852 INFO:teuthology.orchestra.run.smithi003.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-17T05:19:53.852 INFO:teuthology.orchestra.run.smithi003.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-17T05:19:53.852 INFO:teuthology.orchestra.run.smithi003.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-17T05:19:53.852 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-17T05:19:53.852 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-17T05:19:53.852 INFO:teuthology.orchestra.run.smithi003.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-11-17T05:19:53.852 INFO:teuthology.orchestra.run.smithi003.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-17T05:19:53.852 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:19:53.852 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-11-17T05:19:53.852 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:19:53.852 INFO:teuthology.orchestra.run.smithi003.stdout:Install 32 Packages 2023-11-17T05:19:53.853 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:19:53.853 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 31 M 2023-11-17T05:19:53.853 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 111 M 2023-11-17T05:19:53.853 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-11-17T05:19:53.975 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-11-17T05:19:54.197 INFO:teuthology.orchestra.run.smithi003.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 142 kB/s | 6.3 kB 00:00 2023-11-17T05:19:54.247 INFO:teuthology.orchestra.run.smithi003.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2023-11-17T05:19:54.306 INFO:teuthology.orchestra.run.smithi003.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 16 MB/s | 2.4 MB 00:00 2023-11-17T05:19:54.339 INFO:teuthology.orchestra.run.smithi003.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 6.5 MB/s | 220 kB 00:00 2023-11-17T05:19:54.402 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2023-11-17T05:19:54.440 INFO:teuthology.orchestra.run.smithi003.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 21 MB/s | 5.0 MB 00:00 2023-11-17T05:19:54.672 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-11-17T05:19:54.698 INFO:teuthology.orchestra.run.smithi003.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 40 MB/s | 18 MB 00:00 2023-11-17T05:19:54.723 INFO:teuthology.orchestra.run.smithi003.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 137 kB/s | 39 kB 00:00 2023-11-17T05:19:54.749 INFO:teuthology.orchestra.run.smithi003.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 384 kB/s | 157 kB 00:00 2023-11-17T05:19:54.816 INFO:teuthology.orchestra.run.smithi003.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 621 kB/s | 57 kB 00:00 2023-11-17T05:19:54.908 INFO:teuthology.orchestra.run.smithi003.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.8 MB/s | 457 kB 00:00 2023-11-17T05:19:54.958 INFO:teuthology.orchestra.run.smithi003.stdout:(11/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 399 kB/s | 103 kB 00:00 2023-11-17T05:19:55.075 INFO:teuthology.orchestra.run.smithi003.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 962 kB/s | 249 kB 00:00 2023-11-17T05:19:55.175 INFO:teuthology.orchestra.run.smithi003.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 503 kB/s | 109 kB 00:00 2023-11-17T05:19:55.225 INFO:teuthology.orchestra.run.smithi003.stdout:(14/32): python3-cryptography-3.2.1-6.el8.x86_6 2.0 MB/s | 641 kB 00:00 2023-11-17T05:19:56.151 INFO:teuthology.orchestra.run.smithi003.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 41 kB/s | 44 kB 00:01 2023-11-17T05:19:56.198 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-11-17T05:19:56.422 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-11-17T05:19:56.693 INFO:teuthology.orchestra.run.smithi003.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 122 kB/s | 185 kB 00:01 2023-11-17T05:19:56.751 INFO:teuthology.orchestra.run.smithi003.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 113 kB/s | 173 kB 00:01 2023-11-17T05:19:56.855 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-11-17T05:19:56.934 INFO:teuthology.orchestra.run.smithi003.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 46 kB/s | 11 kB 00:00 2023-11-17T05:19:57.001 INFO:teuthology.orchestra.run.smithi003.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 70 kB/s | 18 kB 00:00 2023-11-17T05:19:57.105 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-11-17T05:19:57.135 INFO:teuthology.orchestra.run.smithi003.stdout:(20/32): python3-cherrypy-18.4.0-1.el8.noarch.r 391 kB/s | 384 kB 00:00 2023-11-17T05:19:57.202 INFO:teuthology.orchestra.run.smithi003.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 182 kB/s | 49 kB 00:00 2023-11-17T05:19:57.294 INFO:teuthology.orchestra.run.smithi003.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 201 kB/s | 59 kB 00:00 2023-11-17T05:19:57.329 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-11-17T05:19:57.444 INFO:teuthology.orchestra.run.smithi003.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 64 kB/s | 16 kB 00:00 2023-11-17T05:19:57.494 INFO:teuthology.orchestra.run.smithi003.stdout:(24/32): python3-pecan-1.3.2-9.el8.noarch.rpm 787 kB/s | 283 kB 00:00 2023-11-17T05:19:57.526 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-11-17T05:19:57.543 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-11-17T05:19:57.578 INFO:teuthology.orchestra.run.smithi003.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 66 kB/s | 19 kB 00:00 2023-11-17T05:19:57.703 INFO:teuthology.orchestra.run.smithi003.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 92 kB/s | 24 kB 00:00 2023-11-17T05:19:57.722 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2023-11-17T05:19:57.737 INFO:teuthology.orchestra.run.smithi003.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 121 kB/s | 29 kB 00:00 2023-11-17T05:19:57.837 INFO:teuthology.orchestra.run.smithi003.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 106 kB/s | 27 kB 00:00 2023-11-17T05:19:58.071 INFO:teuthology.orchestra.run.smithi003.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 753 kB/s | 251 kB 00:00 2023-11-17T05:19:58.118 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2023-11-17T05:19:58.118 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2023-11-17T05:19:58.118 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2023-11-17T05:19:58.118 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2023-11-17T05:19:58.118 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2023-11-17T05:19:58.118 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2023-11-17T05:19:58.118 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2023-11-17T05:19:58.118 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-11-17T05:19:58.119 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-11-17T05:19:58.119 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-11-17T05:19:58.119 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-11-17T05:19:58.119 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-11-17T05:19:58.119 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-11-17T05:19:58.119 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-11-17T05:19:58.119 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-11-17T05:19:58.119 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-11-17T05:19:58.120 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-11-17T05:19:58.120 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-17T05:19:58.120 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-11-17T05:19:58.120 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-11-17T05:19:58.120 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-11-17T05:19:58.121 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-11-17T05:19:58.121 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-11-17T05:19:58.121 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-11-17T05:19:58.121 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-11-17T05:19:58.121 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-11-17T05:19:58.121 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-11-17T05:19:58.121 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-11-17T05:19:58.121 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-11-17T05:19:58.122 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-11-17T05:19:58.122 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-11-17T05:19:58.122 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-11-17T05:19:58.154 INFO:teuthology.orchestra.run.smithi003.stdout:(30/32): python3-waitress-1.4.3-1.el8.noarch.rp 566 kB/s | 255 kB 00:00 2023-11-17T05:19:58.180 INFO:teuthology.orchestra.run.smithi003.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 249 kB/s | 85 kB 00:00 2023-11-17T05:19:58.321 INFO:teuthology.orchestra.run.smithi003.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 92 kB/s | 23 kB 00:00 2023-11-17T05:19:58.326 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-11-17T05:19:58.326 INFO:teuthology.orchestra.run.smithi003.stdout:Total 6.9 MB/s | 31 MB 00:04 2023-11-17T05:19:58.415 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-11-17T05:19:58.458 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-11-17T05:19:58.458 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-11-17T05:19:58.797 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-11-17T05:19:58.800 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-11-17T05:19:59.342 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-11-17T05:19:59.343 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:19:59.343 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-11-17T05:19:59.343 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-2:16.2.4-0.el8.x86_64 2023-11-17T05:19:59.343 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2023-11-17T05:19:59.343 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2023-11-17T05:19:59.343 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2023-11-17T05:19:59.343 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2023-11-17T05:19:59.343 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2023-11-17T05:19:59.343 INFO:teuthology.orchestra.run.smithi160.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-17T05:19:59.343 INFO:teuthology.orchestra.run.smithi160.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-17T05:19:59.343 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-11-17T05:19:59.343 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-17T05:19:59.343 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-17T05:19:59.344 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-11-17T05:19:59.344 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-17T05:19:59.344 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-17T05:19:59.344 INFO:teuthology.orchestra.run.smithi160.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-17T05:19:59.344 INFO:teuthology.orchestra.run.smithi160.stdout: python3-mako-1.0.6-14.el8.noarch 2023-11-17T05:19:59.344 INFO:teuthology.orchestra.run.smithi160.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-17T05:19:59.344 INFO:teuthology.orchestra.run.smithi160.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-17T05:19:59.344 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-17T05:19:59.344 INFO:teuthology.orchestra.run.smithi160.stdout: python3-portend-2.6-1.el8.noarch 2023-11-17T05:19:59.344 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-17T05:19:59.344 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-17T05:19:59.344 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pytz-2017.2-11.el8.noarch 2023-11-17T05:19:59.345 INFO:teuthology.orchestra.run.smithi160.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-17T05:19:59.345 INFO:teuthology.orchestra.run.smithi160.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-17T05:19:59.345 INFO:teuthology.orchestra.run.smithi160.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-17T05:19:59.345 INFO:teuthology.orchestra.run.smithi160.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-17T05:19:59.345 INFO:teuthology.orchestra.run.smithi160.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-17T05:19:59.345 INFO:teuthology.orchestra.run.smithi160.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-17T05:19:59.345 INFO:teuthology.orchestra.run.smithi160.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-17T05:19:59.345 INFO:teuthology.orchestra.run.smithi160.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-17T05:19:59.345 INFO:teuthology.orchestra.run.smithi160.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-17T05:19:59.345 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:19:59.345 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-11-17T05:19:59.575 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-base 2023-11-17T05:19:59.936 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-11-17T05:20:00.147 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-11-17T05:20:00.351 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-11-17T05:20:00.435 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:12 ago on Fri 17 Nov 2023 05:18:48 AM UTC. 2023-11-17T05:20:00.645 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-11-17T05:20:00.841 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-11-17T05:20:01.005 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-11-17T05:20:01.180 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-11-17T05:20:01.370 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-11-17T05:20:01.648 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-11-17T05:20:01.851 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-11-17T05:20:01.891 INFO:teuthology.orchestra.run.smithi160.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-17T05:20:02.097 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-11-17T05:20:02.106 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-11-17T05:20:02.108 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2023-11-17T05:20:02.108 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-11-17T05:20:02.223 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install cephadm 2023-11-17T05:20:02.347 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-11-17T05:20:02.593 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-11-17T05:20:02.993 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-11-17T05:20:03.079 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:15 ago on Fri 17 Nov 2023 05:18:48 AM UTC. 2023-11-17T05:20:03.273 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-11-17T05:20:03.434 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-11-17T05:20:03.658 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-11-17T05:20:03.994 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-17T05:21:24.407 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-11-17T05:21:24.407 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:21:24.407 INFO:teuthology.orchestra.run.smithi160.stdout: Package Architecture Version Repository Size 2023-11-17T05:21:24.407 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:21:24.407 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-11-17T05:21:24.407 INFO:teuthology.orchestra.run.smithi160.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2023-11-17T05:21:24.407 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:21:24.480 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-11-17T05:21:24.480 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:21:24.480 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2023-11-17T05:21:24.480 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:21:24.480 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 71 k 2023-11-17T05:21:24.480 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 290 k 2023-11-17T05:21:24.480 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-11-17T05:21:24.480 INFO:teuthology.orchestra.run.smithi160.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.5 MB/s | 71 kB 00:00 2023-11-17T05:21:24.480 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-11-17T05:21:24.480 INFO:teuthology.orchestra.run.smithi160.stdout:Total 1.5 MB/s | 71 kB 00:00 2023-11-17T05:21:24.481 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-11-17T05:21:24.481 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-11-17T05:21:24.481 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-11-17T05:21:24.481 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-11-17T05:21:24.481 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-11-17T05:21:24.481 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-11-17T05:21:24.481 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-17T05:21:24.481 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-17T05:21:24.481 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-17T05:21:24.481 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-17T05:21:24.481 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:21:24.481 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-11-17T05:21:24.482 INFO:teuthology.orchestra.run.smithi160.stdout: cephadm-2:16.2.4-0.el8.noarch 2023-11-17T05:21:24.482 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:21:24.482 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-11-17T05:21:24.482 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-immutable-object-cache 2023-11-17T05:21:24.563 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-11-17T05:21:24.563 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-11-17T05:21:24.563 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-11-17T05:21:24.564 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-11-17T05:21:24.564 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-11-17T05:21:24.564 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-11-17T05:21:24.564 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-11-17T05:21:24.564 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-11-17T05:21:24.564 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-11-17T05:21:24.564 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2023-11-17T05:21:24.564 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-11-17T05:21:24.564 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-11-17T05:21:24.564 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-11-17T05:21:24.564 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-11-17T05:21:24.564 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-11-17T05:21:24.564 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-11-17T05:21:24.565 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-11-17T05:21:24.565 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-11-17T05:21:24.565 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2023-11-17T05:21:24.565 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2023-11-17T05:21:24.565 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2023-11-17T05:21:24.590 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2023-11-17T05:21:24.590 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2023-11-17T05:21:24.590 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2023-11-17T05:21:24.590 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2023-11-17T05:21:24.590 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2023-11-17T05:21:24.590 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-11-17T05:21:24.590 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-11-17T05:21:24.591 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-11-17T05:21:24.591 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-11-17T05:21:24.591 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-11-17T05:21:24.591 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-11-17T05:21:24.591 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-11-17T05:21:24.591 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-11-17T05:21:24.591 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-11-17T05:21:24.591 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-11-17T05:21:24.591 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-17T05:21:24.591 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-11-17T05:21:24.591 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-11-17T05:21:24.591 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-11-17T05:21:24.592 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-11-17T05:21:24.592 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-11-17T05:21:24.592 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-11-17T05:21:24.592 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-11-17T05:21:24.592 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-11-17T05:21:24.592 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-11-17T05:21:24.592 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-11-17T05:21:24.592 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-11-17T05:21:24.592 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-11-17T05:21:24.592 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-11-17T05:21:24.592 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-11-17T05:21:24.592 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-11-17T05:21:24.592 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:21:24.593 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-11-17T05:21:24.593 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-2:16.2.4-0.el8.x86_64 2023-11-17T05:21:24.593 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2023-11-17T05:21:24.593 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2023-11-17T05:21:24.593 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2023-11-17T05:21:24.593 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2023-11-17T05:21:24.593 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2023-11-17T05:21:24.593 INFO:teuthology.orchestra.run.smithi003.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-17T05:21:24.593 INFO:teuthology.orchestra.run.smithi003.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-17T05:21:24.593 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-11-17T05:21:24.593 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-17T05:21:24.593 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-17T05:21:24.594 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-11-17T05:21:24.594 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-17T05:21:24.594 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-17T05:21:24.594 INFO:teuthology.orchestra.run.smithi003.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-17T05:21:24.594 INFO:teuthology.orchestra.run.smithi003.stdout: python3-mako-1.0.6-14.el8.noarch 2023-11-17T05:21:24.594 INFO:teuthology.orchestra.run.smithi003.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-17T05:21:24.594 INFO:teuthology.orchestra.run.smithi003.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-17T05:21:24.594 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-17T05:21:24.594 INFO:teuthology.orchestra.run.smithi003.stdout: python3-portend-2.6-1.el8.noarch 2023-11-17T05:21:24.594 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-17T05:21:24.594 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-17T05:21:24.594 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pytz-2017.2-11.el8.noarch 2023-11-17T05:21:24.595 INFO:teuthology.orchestra.run.smithi003.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-17T05:21:24.595 INFO:teuthology.orchestra.run.smithi003.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-17T05:21:24.595 INFO:teuthology.orchestra.run.smithi003.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-17T05:21:24.595 INFO:teuthology.orchestra.run.smithi003.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-17T05:21:24.673 INFO:teuthology.orchestra.run.smithi003.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-17T05:21:24.673 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-17T05:21:24.673 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-17T05:21:24.673 INFO:teuthology.orchestra.run.smithi003.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-17T05:21:24.673 INFO:teuthology.orchestra.run.smithi003.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-17T05:21:24.673 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:21:24.673 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-17T05:21:24.673 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-base 2023-11-17T05:21:25.543 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:02:37 ago on Fri 17 Nov 2023 05:18:48 AM UTC. 2023-11-17T05:21:25.674 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:33 ago on Fri 17 Nov 2023 05:18:52 AM UTC. 2023-11-17T05:21:27.212 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-11-17T05:21:27.213 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:21:27.213 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repo Size 2023-11-17T05:21:27.213 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:21:27.213 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-11-17T05:21:27.213 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2023-11-17T05:21:27.213 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:21:27.213 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-11-17T05:21:27.214 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:21:27.214 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2023-11-17T05:21:27.214 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:21:27.214 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 163 k 2023-11-17T05:21:27.214 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 587 k 2023-11-17T05:21:27.214 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-11-17T05:21:27.265 INFO:teuthology.orchestra.run.smithi160.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 3.1 MB/s | 163 kB 00:00 2023-11-17T05:21:27.266 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-11-17T05:21:27.266 INFO:teuthology.orchestra.run.smithi160.stdout:Total 3.1 MB/s | 163 kB 00:00 2023-11-17T05:21:27.266 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-11-17T05:21:27.276 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-11-17T05:21:27.276 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-11-17T05:21:27.324 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-11-17T05:21:27.327 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-11-17T05:21:27.382 INFO:teuthology.orchestra.run.smithi003.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-17T05:21:27.438 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-11-17T05:21:27.620 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-17T05:21:27.626 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-17T05:21:27.628 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2023-11-17T05:21:27.628 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-17T05:21:27.727 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install cephadm 2023-11-17T05:21:28.179 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-17T05:21:28.711 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:36 ago on Fri 17 Nov 2023 05:18:52 AM UTC. 2023-11-17T05:21:28.852 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-17T05:21:28.852 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:21:28.852 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-11-17T05:21:28.852 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2023-11-17T05:21:28.852 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:21:28.853 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-11-17T05:21:29.085 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-mgr 2023-11-17T05:21:29.954 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:02:41 ago on Fri 17 Nov 2023 05:18:48 AM UTC. 2023-11-17T05:21:30.561 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-17T05:21:30.561 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:21:30.562 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2023-11-17T05:21:30.562 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:21:30.562 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-11-17T05:21:30.562 INFO:teuthology.orchestra.run.smithi003.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2023-11-17T05:21:30.562 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:21:30.562 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-11-17T05:21:30.562 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:21:30.562 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-11-17T05:21:30.562 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:21:30.563 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 71 k 2023-11-17T05:21:30.563 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 290 k 2023-11-17T05:21:30.563 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-11-17T05:21:30.605 INFO:teuthology.orchestra.run.smithi003.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.7 MB/s | 71 kB 00:00 2023-11-17T05:21:30.606 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-11-17T05:21:30.606 INFO:teuthology.orchestra.run.smithi003.stdout:Total 1.6 MB/s | 71 kB 00:00 2023-11-17T05:21:30.606 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-11-17T05:21:30.614 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-11-17T05:21:30.614 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-11-17T05:21:30.616 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-11-17T05:21:30.620 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-11-17T05:21:30.661 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-11-17T05:21:31.158 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-17T05:21:31.371 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-17T05:21:31.397 INFO:teuthology.orchestra.run.smithi160.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-17T05:21:31.612 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-11-17T05:21:31.613 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2023-11-17T05:21:31.613 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-11-17T05:21:31.638 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-17T05:21:31.723 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-mgr-dashboard 2023-11-17T05:21:32.388 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-17T05:21:32.388 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:21:32.388 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-11-17T05:21:32.388 INFO:teuthology.orchestra.run.smithi003.stdout: cephadm-2:16.2.4-0.el8.noarch 2023-11-17T05:21:32.389 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:21:32.389 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-17T05:21:32.580 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:02:44 ago on Fri 17 Nov 2023 05:18:48 AM UTC. 2023-11-17T05:21:32.601 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-immutable-object-cache 2023-11-17T05:21:33.597 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:41 ago on Fri 17 Nov 2023 05:18:52 AM UTC. 2023-11-17T05:21:34.255 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-11-17T05:21:34.256 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:21:34.256 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2023-11-17T05:21:34.256 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:21:34.256 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-11-17T05:21:34.256 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2023-11-17T05:21:34.256 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2023-11-17T05:21:34.256 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2023-11-17T05:21:34.256 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2023-11-17T05:21:34.256 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-11-17T05:21:34.256 INFO:teuthology.orchestra.run.smithi160.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-17T05:21:34.257 INFO:teuthology.orchestra.run.smithi160.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-17T05:21:34.257 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:21:34.257 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-11-17T05:21:34.257 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:21:34.257 INFO:teuthology.orchestra.run.smithi160.stdout:Install 6 Packages 2023-11-17T05:21:34.257 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:21:34.257 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 3.0 M 2023-11-17T05:21:34.258 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 15 M 2023-11-17T05:21:34.258 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-11-17T05:21:34.663 INFO:teuthology.orchestra.run.smithi160.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 313 kB/s | 24 kB 00:00 2023-11-17T05:21:34.692 INFO:teuthology.orchestra.run.smithi160.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 84 kB/s | 8.9 kB 00:00 2023-11-17T05:21:34.759 INFO:teuthology.orchestra.run.smithi160.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 15 MB/s | 2.7 MB 00:00 2023-11-17T05:21:34.818 INFO:teuthology.orchestra.run.smithi160.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 269 kB/s | 34 kB 00:00 2023-11-17T05:21:34.843 INFO:teuthology.orchestra.run.smithi160.stdout:(5/6): python3-jwt-1.6.1-2.el8.noarch.rpm 237 kB/s | 43 kB 00:00 2023-11-17T05:21:34.873 INFO:teuthology.orchestra.run.smithi160.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.8 MB/s | 196 kB 00:00 2023-11-17T05:21:34.873 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-11-17T05:21:34.873 INFO:teuthology.orchestra.run.smithi160.stdout:Total 4.8 MB/s | 3.0 MB 00:00 2023-11-17T05:21:34.880 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-11-17T05:21:34.890 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-11-17T05:21:34.890 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-11-17T05:21:35.066 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-11-17T05:21:35.069 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-11-17T05:21:35.371 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-11-17T05:21:35.441 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-17T05:21:35.441 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:21:35.441 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2023-11-17T05:21:35.442 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:21:35.442 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-11-17T05:21:35.442 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2023-11-17T05:21:35.442 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:21:35.442 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-11-17T05:21:35.442 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:21:35.442 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-11-17T05:21:35.442 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:21:35.443 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 163 k 2023-11-17T05:21:35.443 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 587 k 2023-11-17T05:21:35.443 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-11-17T05:21:35.520 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 2.1 MB/s | 163 kB 00:00 2023-11-17T05:21:35.522 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-11-17T05:21:35.523 INFO:teuthology.orchestra.run.smithi003.stdout:Total 2.1 MB/s | 163 kB 00:00 2023-11-17T05:21:35.524 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-11-17T05:21:35.533 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-11-17T05:21:35.533 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-11-17T05:21:35.586 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-11-17T05:21:35.589 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-11-17T05:21:35.600 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-11-17T05:21:35.707 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-11-17T05:21:35.887 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-11-17T05:21:35.893 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-17T05:21:36.078 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-11-17T05:21:36.270 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2023-11-17T05:21:36.354 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-17T05:21:37.197 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-17T05:21:37.198 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:21:37.198 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-11-17T05:21:37.198 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2023-11-17T05:21:37.198 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:21:37.198 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-17T05:21:37.288 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2023-11-17T05:21:37.398 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr 2023-11-17T05:21:37.564 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-11-17T05:21:37.875 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-11-17T05:21:37.876 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2023-11-17T05:21:37.876 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2023-11-17T05:21:37.876 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2023-11-17T05:21:37.876 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-11-17T05:21:37.876 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-11-17T05:21:38.398 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:46 ago on Fri 17 Nov 2023 05:18:52 AM UTC. 2023-11-17T05:21:38.633 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-11-17T05:21:38.633 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:21:38.633 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-11-17T05:21:38.633 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2023-11-17T05:21:38.633 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2023-11-17T05:21:38.634 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2023-11-17T05:21:38.634 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-17T05:21:38.634 INFO:teuthology.orchestra.run.smithi160.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-17T05:21:38.634 INFO:teuthology.orchestra.run.smithi160.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-17T05:21:38.634 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:21:38.634 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-11-17T05:21:38.760 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-mgr-diskprediction-local 2023-11-17T05:21:39.616 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:02:51 ago on Fri 17 Nov 2023 05:18:48 AM UTC. 2023-11-17T05:21:40.003 INFO:teuthology.orchestra.run.smithi003.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-17T05:21:40.246 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-17T05:21:40.247 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2023-11-17T05:21:40.247 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-17T05:21:40.351 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-dashboard 2023-11-17T05:21:41.273 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-11-17T05:21:41.277 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================================ 2023-11-17T05:21:41.277 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2023-11-17T05:21:41.277 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================================ 2023-11-17T05:21:41.277 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-11-17T05:21:41.277 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2023-11-17T05:21:41.277 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2023-11-17T05:21:41.278 INFO:teuthology.orchestra.run.smithi160.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-11-17T05:21:41.278 INFO:teuthology.orchestra.run.smithi160.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-11-17T05:21:41.278 INFO:teuthology.orchestra.run.smithi160.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-11-17T05:21:41.278 INFO:teuthology.orchestra.run.smithi160.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-11-17T05:21:41.278 INFO:teuthology.orchestra.run.smithi160.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-11-17T05:21:41.278 INFO:teuthology.orchestra.run.smithi160.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-11-17T05:21:41.278 INFO:teuthology.orchestra.run.smithi160.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-11-17T05:21:41.278 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:21:41.278 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-11-17T05:21:41.278 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================================ 2023-11-17T05:21:41.278 INFO:teuthology.orchestra.run.smithi160.stdout:Install 8 Packages 2023-11-17T05:21:41.279 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:21:41.279 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 37 M 2023-11-17T05:21:41.280 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 210 M 2023-11-17T05:21:41.280 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-11-17T05:21:41.350 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:49 ago on Fri 17 Nov 2023 05:18:52 AM UTC. 2023-11-17T05:21:41.931 INFO:teuthology.orchestra.run.smithi160.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 40 MB/s | 8.6 MB 00:00 2023-11-17T05:21:42.725 INFO:teuthology.orchestra.run.smithi160.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 4.7 MB/s | 3.7 MB 00:00 2023-11-17T05:21:42.798 INFO:teuthology.orchestra.run.smithi160.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 4.5 MB/s | 4.9 MB 00:01 2023-11-17T05:21:42.856 INFO:teuthology.orchestra.run.smithi160.stdout:(4/8): openblas-0.3.15-4.el8.x86_64.rpm 4.2 MB/s | 4.7 MB 00:01 2023-11-17T05:21:42.882 INFO:teuthology.orchestra.run.smithi160.stdout:(5/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.4 MB/s | 225 kB 00:00 2023-11-17T05:21:43.099 INFO:teuthology.orchestra.run.smithi160.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 795 kB/s | 172 kB 00:00 2023-11-17T05:21:43.166 INFO:teuthology.orchestra.run.smithi160.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 2.0 MB/s | 645 kB 00:00 2023-11-17T05:21:43.214 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-17T05:21:43.215 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:21:43.215 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2023-11-17T05:21:43.215 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:21:43.216 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-11-17T05:21:43.216 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2023-11-17T05:21:43.216 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2023-11-17T05:21:43.216 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2023-11-17T05:21:43.216 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2023-11-17T05:21:43.216 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-11-17T05:21:43.216 INFO:teuthology.orchestra.run.smithi003.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-17T05:21:43.216 INFO:teuthology.orchestra.run.smithi003.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-17T05:21:43.216 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:21:43.216 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-11-17T05:21:43.216 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:21:43.216 INFO:teuthology.orchestra.run.smithi003.stdout:Install 6 Packages 2023-11-17T05:21:43.216 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:21:43.217 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 3.0 M 2023-11-17T05:21:43.217 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 15 M 2023-11-17T05:21:43.217 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-11-17T05:21:43.520 INFO:teuthology.orchestra.run.smithi003.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 491 kB/s | 24 kB 00:00 2023-11-17T05:21:43.545 INFO:teuthology.orchestra.run.smithi003.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 120 kB/s | 8.9 kB 00:00 2023-11-17T05:21:43.620 INFO:teuthology.orchestra.run.smithi003.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 18 MB/s | 2.7 MB 00:00 2023-11-17T05:21:43.729 INFO:teuthology.orchestra.run.smithi003.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 204 kB/s | 43 kB 00:00 2023-11-17T05:21:43.750 INFO:teuthology.orchestra.run.smithi160.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 14 MB/s | 14 MB 00:00 2023-11-17T05:21:43.750 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-11-17T05:21:43.750 INFO:teuthology.orchestra.run.smithi160.stdout:Total 15 MB/s | 37 MB 00:02 2023-11-17T05:21:43.870 INFO:teuthology.orchestra.run.smithi003.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 103 kB/s | 34 kB 00:00 2023-11-17T05:21:43.978 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-11-17T05:21:43.997 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-11-17T05:21:43.997 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-11-17T05:21:44.179 INFO:teuthology.orchestra.run.smithi003.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 352 kB/s | 196 kB 00:00 2023-11-17T05:21:44.181 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-11-17T05:21:44.181 INFO:teuthology.orchestra.run.smithi003.stdout:Total 3.1 MB/s | 3.0 MB 00:00 2023-11-17T05:21:44.195 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-11-17T05:21:44.206 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-11-17T05:21:44.206 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-11-17T05:21:44.329 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-11-17T05:21:44.332 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-11-17T05:21:44.383 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-11-17T05:21:44.386 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-11-17T05:21:44.666 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-11-17T05:21:44.894 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-11-17T05:21:44.969 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-11-17T05:21:45.130 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-11-17T05:21:45.133 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-17T05:21:45.254 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-17T05:21:45.335 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-11-17T05:21:45.434 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-17T05:21:45.507 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2023-11-17T05:21:46.028 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-17T05:21:46.267 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-11-17T05:21:46.600 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2023-11-17T05:21:46.846 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-11-17T05:21:46.882 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-11-17T05:21:47.041 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-17T05:21:47.175 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-11-17T05:21:47.175 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2023-11-17T05:21:47.175 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2023-11-17T05:21:47.175 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2023-11-17T05:21:47.175 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-11-17T05:21:47.175 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-11-17T05:21:47.731 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-17T05:21:47.923 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-11-17T05:21:47.924 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:21:47.924 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-11-17T05:21:47.924 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2023-11-17T05:21:47.924 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2023-11-17T05:21:47.924 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2023-11-17T05:21:47.924 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-17T05:21:47.924 INFO:teuthology.orchestra.run.smithi003.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-17T05:21:47.924 INFO:teuthology.orchestra.run.smithi003.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-17T05:21:47.924 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:21:47.924 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-17T05:21:48.083 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-diskprediction-local 2023-11-17T05:21:48.120 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-11-17T05:21:48.394 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-17T05:21:49.069 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:02:57 ago on Fri 17 Nov 2023 05:18:52 AM UTC. 2023-11-17T05:21:50.145 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-17T05:21:50.914 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-17T05:21:50.916 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================================ 2023-11-17T05:21:50.916 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2023-11-17T05:21:50.916 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================================ 2023-11-17T05:21:50.916 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-11-17T05:21:50.916 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2023-11-17T05:21:50.916 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2023-11-17T05:21:50.916 INFO:teuthology.orchestra.run.smithi003.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-11-17T05:21:50.916 INFO:teuthology.orchestra.run.smithi003.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-11-17T05:21:50.916 INFO:teuthology.orchestra.run.smithi003.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-11-17T05:21:50.917 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-11-17T05:21:50.917 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-11-17T05:21:50.917 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-11-17T05:21:50.917 INFO:teuthology.orchestra.run.smithi003.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-11-17T05:21:50.917 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:21:50.917 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-11-17T05:21:50.917 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================================ 2023-11-17T05:21:50.917 INFO:teuthology.orchestra.run.smithi003.stdout:Install 8 Packages 2023-11-17T05:21:50.917 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:21:50.918 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 37 M 2023-11-17T05:21:50.918 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 210 M 2023-11-17T05:21:50.918 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-11-17T05:21:51.334 INFO:teuthology.orchestra.run.smithi003.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 46 MB/s | 8.6 MB 00:00 2023-11-17T05:21:51.483 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-11-17T05:21:51.712 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-11-17T05:21:51.742 INFO:teuthology.orchestra.run.smithi003.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 8.0 MB/s | 4.7 MB 00:00 2023-11-17T05:21:51.809 INFO:teuthology.orchestra.run.smithi003.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 7.4 MB/s | 4.9 MB 00:00 2023-11-17T05:21:51.834 INFO:teuthology.orchestra.run.smithi003.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.4 MB/s | 225 kB 00:00 2023-11-17T05:21:51.902 INFO:teuthology.orchestra.run.smithi003.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 6.6 MB/s | 3.7 MB 00:00 2023-11-17T05:21:52.010 INFO:teuthology.orchestra.run.smithi003.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 1.6 MB/s | 172 kB 00:00 2023-11-17T05:21:52.035 INFO:teuthology.orchestra.run.smithi003.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 3.1 MB/s | 645 kB 00:00 2023-11-17T05:21:52.043 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-11-17T05:21:52.043 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2023-11-17T05:21:52.044 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-11-17T05:21:52.044 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-11-17T05:21:52.044 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-11-17T05:21:52.044 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-11-17T05:21:52.044 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-11-17T05:21:52.051 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-11-17T05:21:52.502 INFO:teuthology.orchestra.run.smithi003.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 20 MB/s | 14 MB 00:00 2023-11-17T05:21:52.503 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-11-17T05:21:52.503 INFO:teuthology.orchestra.run.smithi003.stdout:Total 23 MB/s | 37 MB 00:01 2023-11-17T05:21:52.738 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-11-17T05:21:52.739 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:21:52.739 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-11-17T05:21:52.739 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2023-11-17T05:21:52.739 INFO:teuthology.orchestra.run.smithi160.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-11-17T05:21:52.739 INFO:teuthology.orchestra.run.smithi160.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-11-17T05:21:52.739 INFO:teuthology.orchestra.run.smithi160.stdout: openblas-0.3.15-4.el8.x86_64 2023-11-17T05:21:52.739 INFO:teuthology.orchestra.run.smithi160.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-11-17T05:21:52.739 INFO:teuthology.orchestra.run.smithi160.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-17T05:21:52.739 INFO:teuthology.orchestra.run.smithi160.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-17T05:21:52.739 INFO:teuthology.orchestra.run.smithi160.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-11-17T05:21:52.739 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:21:52.739 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-11-17T05:21:52.757 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-11-17T05:21:52.779 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-11-17T05:21:52.779 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-11-17T05:21:52.900 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-mgr-rook 2023-11-17T05:21:53.132 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-11-17T05:21:53.135 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-11-17T05:21:53.762 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:03:05 ago on Fri 17 Nov 2023 05:18:48 AM UTC. 2023-11-17T05:21:53.762 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-11-17T05:21:53.915 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-17T05:21:54.058 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-17T05:21:54.210 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-17T05:21:54.866 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-17T05:21:55.023 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-11-17T05:21:55.441 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-11-17T05:21:55.442 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:21:55.442 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2023-11-17T05:21:55.442 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:21:55.442 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-11-17T05:21:55.442 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2023-11-17T05:21:55.442 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2023-11-17T05:21:55.442 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-17T05:21:55.443 INFO:teuthology.orchestra.run.smithi160.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-17T05:21:55.443 INFO:teuthology.orchestra.run.smithi160.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-17T05:21:55.443 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-11-17T05:21:55.443 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-11-17T05:21:55.443 INFO:teuthology.orchestra.run.smithi160.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-17T05:21:55.443 INFO:teuthology.orchestra.run.smithi160.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-11-17T05:21:55.443 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-11-17T05:21:55.443 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-11-17T05:21:55.443 INFO:teuthology.orchestra.run.smithi160.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-11-17T05:21:55.443 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-17T05:21:55.444 INFO:teuthology.orchestra.run.smithi160.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-17T05:21:55.444 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:21:55.444 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-11-17T05:21:55.444 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:21:55.444 INFO:teuthology.orchestra.run.smithi160.stdout:Install 13 Packages 2023-11-17T05:21:55.444 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:21:55.444 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 1.9 M 2023-11-17T05:21:55.444 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 24 M 2023-11-17T05:21:55.444 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-11-17T05:21:55.624 INFO:teuthology.orchestra.run.smithi160.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 607 kB/s | 27 kB 00:00 2023-11-17T05:21:55.710 INFO:teuthology.orchestra.run.smithi160.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 156 kB/s | 20 kB 00:00 2023-11-17T05:21:55.711 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-11-17T05:21:55.733 INFO:teuthology.orchestra.run.smithi160.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 174 kB/s | 27 kB 00:00 2023-11-17T05:21:55.817 INFO:teuthology.orchestra.run.smithi160.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 658 kB/s | 126 kB 00:00 2023-11-17T05:21:55.842 INFO:teuthology.orchestra.run.smithi160.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 822 kB/s | 110 kB 00:00 2023-11-17T05:21:55.856 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-17T05:21:55.867 INFO:teuthology.orchestra.run.smithi160.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 1.1 MB/s | 155 kB 00:00 2023-11-17T05:21:55.918 INFO:teuthology.orchestra.run.smithi160.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 424 kB/s | 43 kB 00:00 2023-11-17T05:21:56.301 INFO:teuthology.orchestra.run.smithi160.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 72 kB/s | 33 kB 00:00 2023-11-17T05:21:56.326 INFO:teuthology.orchestra.run.smithi160.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 35 kB/s | 16 kB 00:00 2023-11-17T05:21:56.444 INFO:teuthology.orchestra.run.smithi160.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 155 kB/s | 81 kB 00:00 2023-11-17T05:21:56.510 INFO:teuthology.orchestra.run.smithi160.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 350 kB/s | 64 kB 00:00 2023-11-17T05:21:56.552 INFO:teuthology.orchestra.run.smithi160.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 565 kB/s | 61 kB 00:00 2023-11-17T05:21:56.640 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-17T05:21:56.947 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-11-17T05:21:57.192 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-17T05:21:57.769 INFO:teuthology.orchestra.run.smithi160.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 813 kB/s | 1.2 MB 00:01 2023-11-17T05:21:57.772 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-11-17T05:21:57.772 INFO:teuthology.orchestra.run.smithi160.stdout:Total 840 kB/s | 1.9 MB 00:02 2023-11-17T05:21:57.831 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-11-17T05:21:57.848 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-11-17T05:21:57.848 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-11-17T05:21:57.936 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-11-17T05:21:57.940 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-11-17T05:21:58.457 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-11-17T05:21:59.095 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-11-17T05:21:59.180 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-17T05:21:59.269 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-11-17T05:21:59.518 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-11-17T05:21:59.772 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-11-17T05:22:00.014 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-11-17T05:22:00.230 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-11-17T05:22:00.547 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-11-17T05:22:00.581 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-11-17T05:22:00.779 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-11-17T05:22:01.133 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-11-17T05:22:01.513 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-11-17T05:22:01.513 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2023-11-17T05:22:01.513 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-11-17T05:22:01.513 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-11-17T05:22:01.513 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-11-17T05:22:01.514 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-11-17T05:22:01.514 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-11-17T05:22:01.514 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-11-17T05:22:01.826 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-11-17T05:22:02.138 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-11-17T05:22:02.275 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-11-17T05:22:02.275 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:22:02.275 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-11-17T05:22:02.275 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2023-11-17T05:22:02.275 INFO:teuthology.orchestra.run.smithi003.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-11-17T05:22:02.275 INFO:teuthology.orchestra.run.smithi003.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-11-17T05:22:02.275 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-0.3.15-4.el8.x86_64 2023-11-17T05:22:02.276 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-11-17T05:22:02.276 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-17T05:22:02.276 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-17T05:22:02.276 INFO:teuthology.orchestra.run.smithi003.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-11-17T05:22:02.276 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:22:02.276 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-17T05:22:02.366 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-11-17T05:22:02.389 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-rook 2023-11-17T05:22:02.555 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-11-17T05:22:02.764 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-11-17T05:22:03.051 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-11-17T05:22:03.051 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2023-11-17T05:22:03.051 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-11-17T05:22:03.051 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-11-17T05:22:03.051 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-11-17T05:22:03.051 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-11-17T05:22:03.051 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-11-17T05:22:03.051 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-11-17T05:22:03.051 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-11-17T05:22:03.052 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-11-17T05:22:03.052 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-11-17T05:22:03.052 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-11-17T05:22:03.052 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-11-17T05:22:03.379 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:03:11 ago on Fri 17 Nov 2023 05:18:52 AM UTC. 2023-11-17T05:22:03.810 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-11-17T05:22:03.811 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:22:03.811 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-11-17T05:22:03.811 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2023-11-17T05:22:03.811 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-17T05:22:03.811 INFO:teuthology.orchestra.run.smithi160.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-17T05:22:03.811 INFO:teuthology.orchestra.run.smithi160.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-17T05:22:03.811 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-17T05:22:03.811 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-17T05:22:03.811 INFO:teuthology.orchestra.run.smithi160.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-17T05:22:03.811 INFO:teuthology.orchestra.run.smithi160.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-17T05:22:03.811 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-17T05:22:03.811 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-17T05:22:03.811 INFO:teuthology.orchestra.run.smithi160.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-17T05:22:03.812 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-17T05:22:03.812 INFO:teuthology.orchestra.run.smithi160.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-17T05:22:03.812 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:22:03.812 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-11-17T05:22:03.934 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-mgr-cephadm 2023-11-17T05:22:04.791 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:03:16 ago on Fri 17 Nov 2023 05:18:48 AM UTC. 2023-11-17T05:22:05.240 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-17T05:22:05.241 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:22:05.242 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2023-11-17T05:22:05.242 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:22:05.242 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-11-17T05:22:05.242 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2023-11-17T05:22:05.242 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2023-11-17T05:22:05.242 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-17T05:22:05.242 INFO:teuthology.orchestra.run.smithi003.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-17T05:22:05.242 INFO:teuthology.orchestra.run.smithi003.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-17T05:22:05.242 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-11-17T05:22:05.242 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-11-17T05:22:05.242 INFO:teuthology.orchestra.run.smithi003.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-17T05:22:05.242 INFO:teuthology.orchestra.run.smithi003.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-11-17T05:22:05.243 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-11-17T05:22:05.243 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-11-17T05:22:05.243 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-11-17T05:22:05.243 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-17T05:22:05.243 INFO:teuthology.orchestra.run.smithi003.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-17T05:22:05.243 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:22:05.243 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-11-17T05:22:05.243 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:22:05.243 INFO:teuthology.orchestra.run.smithi003.stdout:Install 13 Packages 2023-11-17T05:22:05.243 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:22:05.244 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 1.9 M 2023-11-17T05:22:05.244 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 24 M 2023-11-17T05:22:05.244 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-11-17T05:22:05.740 INFO:teuthology.orchestra.run.smithi003.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 634 kB/s | 27 kB 00:00 2023-11-17T05:22:05.832 INFO:teuthology.orchestra.run.smithi003.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 149 kB/s | 20 kB 00:00 2023-11-17T05:22:05.857 INFO:teuthology.orchestra.run.smithi003.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 167 kB/s | 27 kB 00:00 2023-11-17T05:22:05.926 INFO:teuthology.orchestra.run.smithi003.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 688 kB/s | 126 kB 00:00 2023-11-17T05:22:05.950 INFO:teuthology.orchestra.run.smithi003.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 939 kB/s | 110 kB 00:00 2023-11-17T05:22:05.976 INFO:teuthology.orchestra.run.smithi003.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 1.3 MB/s | 155 kB 00:00 2023-11-17T05:22:06.042 INFO:teuthology.orchestra.run.smithi003.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 364 kB/s | 43 kB 00:00 2023-11-17T05:22:06.240 INFO:teuthology.orchestra.run.smithi003.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 65 kB/s | 16 kB 00:00 2023-11-17T05:22:06.316 INFO:teuthology.orchestra.run.smithi003.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 90 kB/s | 33 kB 00:00 2023-11-17T05:22:06.368 INFO:teuthology.orchestra.run.smithi003.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 249 kB/s | 81 kB 00:00 2023-11-17T05:22:06.457 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-11-17T05:22:06.458 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:22:06.458 INFO:teuthology.orchestra.run.smithi160.stdout: Package Arch Version Repository Size 2023-11-17T05:22:06.458 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:22:06.458 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-11-17T05:22:06.459 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2023-11-17T05:22:06.459 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2023-11-17T05:22:06.459 INFO:teuthology.orchestra.run.smithi160.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-11-17T05:22:06.459 INFO:teuthology.orchestra.run.smithi160.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-11-17T05:22:06.459 INFO:teuthology.orchestra.run.smithi160.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-11-17T05:22:06.459 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-11-17T05:22:06.460 INFO:teuthology.orchestra.run.smithi160.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-11-17T05:22:06.460 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:22:06.460 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-11-17T05:22:06.460 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:22:06.460 INFO:teuthology.orchestra.run.smithi160.stdout:Install 6 Packages 2023-11-17T05:22:06.460 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:22:06.461 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 5.8 M 2023-11-17T05:22:06.462 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 24 M 2023-11-17T05:22:06.462 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-11-17T05:22:06.468 INFO:teuthology.orchestra.run.smithi003.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 613 kB/s | 61 kB 00:00 2023-11-17T05:22:06.494 INFO:teuthology.orchestra.run.smithi003.stdout:(12/13): python3-rsa-4.9-2.el8.noarch.rpm 362 kB/s | 64 kB 00:00 2023-11-17T05:22:06.711 INFO:teuthology.orchestra.run.smithi003.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.4 MB/s | 1.2 MB 00:00 2023-11-17T05:22:06.711 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-11-17T05:22:06.712 INFO:teuthology.orchestra.run.smithi003.stdout:Total 1.3 MB/s | 1.9 MB 00:01 2023-11-17T05:22:06.734 INFO:teuthology.orchestra.run.smithi160.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.4 MB/s | 89 kB 00:00 2023-11-17T05:22:06.774 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-11-17T05:22:06.792 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-11-17T05:22:06.792 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-11-17T05:22:06.889 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-11-17T05:22:06.894 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-11-17T05:22:06.976 INFO:teuthology.orchestra.run.smithi160.stdout:(2/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.7 MB/s | 538 kB 00:00 2023-11-17T05:22:31.324 INFO:teuthology.orchestra.run.smithi160.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 52 kB/s | 21 kB 00:00 2023-11-17T05:22:31.324 INFO:teuthology.orchestra.run.smithi160.stdout:(4/6): python3-babel-2.5.1-7.el8.noarch.rpm 8.9 MB/s | 4.8 MB 00:00 2023-11-17T05:22:31.324 INFO:teuthology.orchestra.run.smithi160.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 217 kB/s | 49 kB 00:00 2023-11-17T05:22:31.324 INFO:teuthology.orchestra.run.smithi160.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 439 kB/s | 355 kB 00:00 2023-11-17T05:22:31.324 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-11-17T05:22:31.324 INFO:teuthology.orchestra.run.smithi160.stdout:Total 4.4 MB/s | 5.8 MB 00:01 2023-11-17T05:22:31.324 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-11-17T05:22:31.324 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-11-17T05:22:31.324 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-11-17T05:22:31.324 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-11-17T05:22:31.324 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-11-17T05:22:31.324 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-11-17T05:22:31.325 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-11-17T05:22:31.325 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-11-17T05:22:31.325 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-11-17T05:22:31.325 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-11-17T05:22:31.325 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-11-17T05:22:31.325 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-11-17T05:22:31.325 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-11-17T05:22:31.325 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2023-11-17T05:22:31.325 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-11-17T05:22:31.325 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-11-17T05:22:31.325 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-11-17T05:22:31.325 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-11-17T05:22:31.325 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-11-17T05:22:31.325 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:22:31.326 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-11-17T05:22:31.326 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2023-11-17T05:22:31.326 INFO:teuthology.orchestra.run.smithi160.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2023-11-17T05:22:31.326 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2023-11-17T05:22:31.326 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:22:31.330 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-11-17T05:22:31.396 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install ceph-fuse 2023-11-17T05:22:31.397 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-11-17T05:22:31.397 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-11-17T05:22:31.397 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-11-17T05:22:31.397 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-11-17T05:22:31.397 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-11-17T05:22:31.397 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-11-17T05:22:31.397 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-11-17T05:22:31.397 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-11-17T05:22:31.397 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-11-17T05:22:31.397 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-11-17T05:22:31.397 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-11-17T05:22:31.397 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-11-17T05:22:31.397 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-11-17T05:22:31.398 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-11-17T05:22:31.398 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-11-17T05:22:31.398 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2023-11-17T05:22:31.398 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-11-17T05:22:31.404 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-11-17T05:22:31.436 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-11-17T05:22:31.436 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-11-17T05:22:31.437 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-11-17T05:22:31.437 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-11-17T05:22:31.437 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-11-17T05:22:31.437 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-11-17T05:22:31.437 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-11-17T05:22:31.437 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-11-17T05:22:31.437 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-11-17T05:22:31.437 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-11-17T05:22:31.438 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:22:31.438 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-11-17T05:22:31.438 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2023-11-17T05:22:31.438 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-17T05:22:31.438 INFO:teuthology.orchestra.run.smithi003.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-17T05:22:31.438 INFO:teuthology.orchestra.run.smithi003.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-17T05:22:31.438 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-17T05:22:31.438 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-17T05:22:31.438 INFO:teuthology.orchestra.run.smithi003.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-17T05:22:31.438 INFO:teuthology.orchestra.run.smithi003.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-17T05:22:31.438 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-17T05:22:31.438 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-17T05:22:31.438 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-17T05:22:31.438 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-17T05:22:31.439 INFO:teuthology.orchestra.run.smithi003.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-17T05:22:31.439 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:22:31.439 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-17T05:22:31.439 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-cephadm 2023-11-17T05:22:32.315 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:03:44 ago on Fri 17 Nov 2023 05:18:48 AM UTC. 2023-11-17T05:22:32.734 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:03:40 ago on Fri 17 Nov 2023 05:18:52 AM UTC. 2023-11-17T05:22:33.991 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-11-17T05:22:33.992 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:22:33.992 INFO:teuthology.orchestra.run.smithi160.stdout: Package Architecture Version Repository Size 2023-11-17T05:22:33.992 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:22:33.992 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-11-17T05:22:33.993 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2023-11-17T05:22:33.993 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2023-11-17T05:22:33.993 INFO:teuthology.orchestra.run.smithi160.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-11-17T05:22:33.993 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:22:33.993 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-11-17T05:22:33.993 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:22:33.993 INFO:teuthology.orchestra.run.smithi160.stdout:Install 2 Packages 2023-11-17T05:22:33.993 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:22:33.993 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 968 k 2023-11-17T05:22:33.994 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 3.2 M 2023-11-17T05:22:33.994 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-11-17T05:22:34.099 INFO:teuthology.orchestra.run.smithi160.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 13 MB/s | 885 kB 00:00 2023-11-17T05:22:34.174 INFO:teuthology.orchestra.run.smithi160.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 593 kB/s | 83 kB 00:00 2023-11-17T05:22:34.175 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-11-17T05:22:34.175 INFO:teuthology.orchestra.run.smithi160.stdout:Total 5.2 MB/s | 968 kB 00:00 2023-11-17T05:22:34.185 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-11-17T05:22:34.206 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-11-17T05:22:34.206 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-11-17T05:22:34.292 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-11-17T05:22:34.295 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-11-17T05:22:34.310 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-17T05:22:34.311 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:22:34.312 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2023-11-17T05:22:34.312 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:22:34.312 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-11-17T05:22:34.312 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2023-11-17T05:22:34.312 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2023-11-17T05:22:34.312 INFO:teuthology.orchestra.run.smithi003.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-11-17T05:22:34.312 INFO:teuthology.orchestra.run.smithi003.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-11-17T05:22:34.312 INFO:teuthology.orchestra.run.smithi003.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-11-17T05:22:34.312 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-11-17T05:22:34.312 INFO:teuthology.orchestra.run.smithi003.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-11-17T05:22:34.312 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:22:34.312 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-11-17T05:22:34.313 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:22:34.313 INFO:teuthology.orchestra.run.smithi003.stdout:Install 6 Packages 2023-11-17T05:22:34.313 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:22:34.313 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 5.8 M 2023-11-17T05:22:34.313 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 24 M 2023-11-17T05:22:34.313 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-11-17T05:22:34.463 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-11-17T05:22:34.754 INFO:teuthology.orchestra.run.smithi003.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.6 MB/s | 89 kB 00:00 2023-11-17T05:22:34.759 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-11-17T05:22:34.839 INFO:teuthology.orchestra.run.smithi003.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 255 kB/s | 21 kB 00:00 2023-11-17T05:22:34.946 INFO:teuthology.orchestra.run.smithi003.stdout:(3/6): python3-execnet-1.7.1-1.el8.noarch.rpm 3.2 MB/s | 355 kB 00:00 2023-11-17T05:22:34.959 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-11-17T05:22:34.997 INFO:teuthology.orchestra.run.smithi003.stdout:(4/6): python3-remoto-1.2.1-1.el8.noarch.rpm 978 kB/s | 49 kB 00:00 2023-11-17T05:22:35.072 INFO:teuthology.orchestra.run.smithi003.stdout:(5/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.4 MB/s | 538 kB 00:00 2023-11-17T05:22:35.322 INFO:teuthology.orchestra.run.smithi003.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 7.7 MB/s | 4.8 MB 00:00 2023-11-17T05:22:35.323 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-11-17T05:22:35.323 INFO:teuthology.orchestra.run.smithi003.stdout:Total 5.8 MB/s | 5.8 MB 00:01 2023-11-17T05:22:35.347 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-11-17T05:22:35.347 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2023-11-17T05:22:35.381 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-11-17T05:22:35.392 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-11-17T05:22:35.392 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-11-17T05:22:35.518 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-11-17T05:22:35.521 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-11-17T05:22:35.800 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-11-17T05:22:36.039 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-11-17T05:22:36.039 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:22:36.039 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-11-17T05:22:36.039 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-11-17T05:22:36.039 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:22:36.039 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-11-17T05:22:36.078 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-11-17T05:22:36.283 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install librados-devel 2023-11-17T05:22:36.384 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-11-17T05:22:37.157 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:03:49 ago on Fri 17 Nov 2023 05:18:48 AM UTC. 2023-11-17T05:22:37.388 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-11-17T05:22:37.716 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-11-17T05:22:37.962 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-11-17T05:22:38.133 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-11-17T05:22:38.465 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-11-17T05:22:38.465 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2023-11-17T05:22:38.465 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-11-17T05:22:38.465 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-11-17T05:22:38.465 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-11-17T05:22:38.465 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-11-17T05:22:38.823 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-11-17T05:22:38.824 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:22:38.824 INFO:teuthology.orchestra.run.smithi160.stdout: Package Architecture Version Repository Size 2023-11-17T05:22:38.824 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:22:38.824 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-11-17T05:22:38.824 INFO:teuthology.orchestra.run.smithi160.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2023-11-17T05:22:38.824 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:22:38.824 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-11-17T05:22:38.824 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:22:38.824 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2023-11-17T05:22:38.824 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:22:38.825 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 131 k 2023-11-17T05:22:38.825 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 439 k 2023-11-17T05:22:38.825 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-11-17T05:22:38.871 INFO:teuthology.orchestra.run.smithi160.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.8 MB/s | 131 kB 00:00 2023-11-17T05:22:38.871 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-11-17T05:22:38.872 INFO:teuthology.orchestra.run.smithi160.stdout:Total 2.7 MB/s | 131 kB 00:00 2023-11-17T05:22:38.872 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-11-17T05:22:38.877 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-11-17T05:22:38.877 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-11-17T05:22:38.927 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-11-17T05:22:38.930 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-11-17T05:22:39.070 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-11-17T05:22:39.286 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-11-17T05:22:39.286 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:22:39.286 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-11-17T05:22:39.286 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2023-11-17T05:22:39.286 INFO:teuthology.orchestra.run.smithi003.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2023-11-17T05:22:39.286 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2023-11-17T05:22:39.286 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:22:39.286 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-17T05:22:39.317 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-17T05:22:39.416 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-fuse 2023-11-17T05:22:39.608 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-17T05:22:40.294 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-17T05:22:40.294 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:22:40.294 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-11-17T05:22:40.294 INFO:teuthology.orchestra.run.smithi160.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2023-11-17T05:22:40.295 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:22:40.295 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-11-17T05:22:40.406 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:03:48 ago on Fri 17 Nov 2023 05:18:52 AM UTC. 2023-11-17T05:22:40.577 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install libcephfs2 2023-11-17T05:22:41.448 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:03:53 ago on Fri 17 Nov 2023 05:18:48 AM UTC. 2023-11-17T05:22:42.274 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-17T05:22:42.275 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:22:42.275 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2023-11-17T05:22:42.275 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:22:42.275 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-11-17T05:22:42.275 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2023-11-17T05:22:42.275 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2023-11-17T05:22:42.275 INFO:teuthology.orchestra.run.smithi003.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-11-17T05:22:42.275 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:22:42.275 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-11-17T05:22:42.275 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:22:42.276 INFO:teuthology.orchestra.run.smithi003.stdout:Install 2 Packages 2023-11-17T05:22:42.276 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:22:42.276 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 968 k 2023-11-17T05:22:42.276 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 3.2 M 2023-11-17T05:22:42.276 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-11-17T05:22:42.373 INFO:teuthology.orchestra.run.smithi003.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 15 MB/s | 885 kB 00:00 2023-11-17T05:22:42.448 INFO:teuthology.orchestra.run.smithi003.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 621 kB/s | 83 kB 00:00 2023-11-17T05:22:42.448 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-11-17T05:22:42.449 INFO:teuthology.orchestra.run.smithi003.stdout:Total 5.5 MB/s | 968 kB 00:00 2023-11-17T05:22:42.462 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-11-17T05:22:42.486 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-11-17T05:22:42.486 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-11-17T05:22:42.588 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-11-17T05:22:42.591 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-11-17T05:22:42.773 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-11-17T05:22:42.901 INFO:teuthology.orchestra.run.smithi160.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-17T05:22:43.075 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-11-17T05:22:43.116 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-11-17T05:22:43.118 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2023-11-17T05:22:43.118 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-11-17T05:22:43.259 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install libcephfs-devel 2023-11-17T05:22:43.275 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-11-17T05:22:43.709 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-11-17T05:22:43.709 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2023-11-17T05:22:44.124 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:03:56 ago on Fri 17 Nov 2023 05:18:48 AM UTC. 2023-11-17T05:22:44.614 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-11-17T05:22:44.614 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:22:44.614 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-11-17T05:22:44.614 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-11-17T05:22:44.614 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:22:44.614 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-17T05:22:44.839 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install librados-devel 2023-11-17T05:22:45.783 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-11-17T05:22:45.785 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:22:45.785 INFO:teuthology.orchestra.run.smithi160.stdout: Package Architecture Version Repository Size 2023-11-17T05:22:45.785 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:22:45.785 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-11-17T05:22:45.785 INFO:teuthology.orchestra.run.smithi160.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2023-11-17T05:22:45.785 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:22:45.785 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-11-17T05:22:45.785 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:22:45.785 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2023-11-17T05:22:45.785 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:22:45.785 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 24 k 2023-11-17T05:22:45.786 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 94 k 2023-11-17T05:22:45.786 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-11-17T05:22:45.843 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:03:53 ago on Fri 17 Nov 2023 05:18:52 AM UTC. 2023-11-17T05:22:45.846 INFO:teuthology.orchestra.run.smithi160.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 420 kB/s | 24 kB 00:00 2023-11-17T05:22:45.846 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-11-17T05:22:45.846 INFO:teuthology.orchestra.run.smithi160.stdout:Total 411 kB/s | 24 kB 00:00 2023-11-17T05:22:45.846 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-11-17T05:22:45.849 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-11-17T05:22:45.849 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-11-17T05:22:45.850 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-11-17T05:22:45.853 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-11-17T05:22:45.905 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-11-17T05:22:46.081 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-17T05:22:46.372 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-17T05:22:47.036 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-17T05:22:47.036 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:22:47.036 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-11-17T05:22:47.036 INFO:teuthology.orchestra.run.smithi160.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2023-11-17T05:22:47.036 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:22:47.036 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-11-17T05:22:47.164 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install librados2 2023-11-17T05:22:47.692 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-17T05:22:47.693 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:22:47.693 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2023-11-17T05:22:47.693 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:22:47.693 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-11-17T05:22:47.693 INFO:teuthology.orchestra.run.smithi003.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2023-11-17T05:22:47.693 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:22:47.694 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-11-17T05:22:47.694 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:22:47.694 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-11-17T05:22:47.694 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:22:47.694 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 131 k 2023-11-17T05:22:47.694 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 439 k 2023-11-17T05:22:47.695 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-11-17T05:22:47.737 INFO:teuthology.orchestra.run.smithi003.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 3.0 MB/s | 131 kB 00:00 2023-11-17T05:22:47.738 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-11-17T05:22:47.738 INFO:teuthology.orchestra.run.smithi003.stdout:Total 2.9 MB/s | 131 kB 00:00 2023-11-17T05:22:47.739 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-11-17T05:22:47.745 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-11-17T05:22:47.745 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-11-17T05:22:47.802 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-11-17T05:22:47.805 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-11-17T05:22:47.925 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-11-17T05:22:48.029 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:04:00 ago on Fri 17 Nov 2023 05:18:48 AM UTC. 2023-11-17T05:22:48.148 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-17T05:22:48.478 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-17T05:22:49.230 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-17T05:22:49.230 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:22:49.230 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-11-17T05:22:49.231 INFO:teuthology.orchestra.run.smithi003.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2023-11-17T05:22:49.231 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:22:49.231 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-17T05:22:49.464 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install libcephfs2 2023-11-17T05:22:49.492 INFO:teuthology.orchestra.run.smithi160.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-17T05:22:49.708 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-11-17T05:22:49.709 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2023-11-17T05:22:49.709 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-11-17T05:22:49.822 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install librbd1 2023-11-17T05:22:50.461 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:03:58 ago on Fri 17 Nov 2023 05:18:52 AM UTC. 2023-11-17T05:22:50.687 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:04:02 ago on Fri 17 Nov 2023 05:18:48 AM UTC. 2023-11-17T05:22:52.105 INFO:teuthology.orchestra.run.smithi003.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-17T05:22:52.120 INFO:teuthology.orchestra.run.smithi160.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-17T05:22:52.335 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-11-17T05:22:52.336 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2023-11-17T05:22:52.336 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-11-17T05:22:52.347 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-17T05:22:52.348 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2023-11-17T05:22:52.348 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-17T05:22:52.442 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install python3-rados 2023-11-17T05:22:52.463 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install libcephfs-devel 2023-11-17T05:22:53.303 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:04:05 ago on Fri 17 Nov 2023 05:18:48 AM UTC. 2023-11-17T05:22:53.488 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:04:01 ago on Fri 17 Nov 2023 05:18:52 AM UTC. 2023-11-17T05:22:54.756 INFO:teuthology.orchestra.run.smithi160.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-17T05:22:54.971 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-11-17T05:22:54.972 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2023-11-17T05:22:54.972 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-11-17T05:22:55.105 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install python3-rgw 2023-11-17T05:22:55.316 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-17T05:22:55.317 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:22:55.317 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2023-11-17T05:22:55.317 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:22:55.317 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-11-17T05:22:55.317 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2023-11-17T05:22:55.318 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:22:55.318 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-11-17T05:22:55.318 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:22:55.318 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-11-17T05:22:55.318 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:22:55.318 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 24 k 2023-11-17T05:22:55.319 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 94 k 2023-11-17T05:22:55.319 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-11-17T05:22:55.364 INFO:teuthology.orchestra.run.smithi003.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 530 kB/s | 24 kB 00:00 2023-11-17T05:22:55.365 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-11-17T05:22:55.365 INFO:teuthology.orchestra.run.smithi003.stdout:Total 513 kB/s | 24 kB 00:00 2023-11-17T05:22:55.366 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-11-17T05:22:55.370 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-11-17T05:22:55.370 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-11-17T05:22:55.373 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-11-17T05:22:55.376 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-11-17T05:22:55.459 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-11-17T05:22:55.630 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-17T05:22:55.938 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-17T05:22:55.962 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:04:07 ago on Fri 17 Nov 2023 05:18:48 AM UTC. 2023-11-17T05:22:56.674 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-17T05:22:56.674 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:22:56.674 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-11-17T05:22:56.675 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2023-11-17T05:22:56.675 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:22:56.675 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-17T05:22:56.809 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install librados2 2023-11-17T05:22:57.409 INFO:teuthology.orchestra.run.smithi160.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-17T05:22:57.624 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-11-17T05:22:57.625 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2023-11-17T05:22:57.625 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-11-17T05:22:57.732 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install python3-cephfs 2023-11-17T05:22:57.805 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:04:05 ago on Fri 17 Nov 2023 05:18:52 AM UTC. 2023-11-17T05:22:58.591 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:04:10 ago on Fri 17 Nov 2023 05:18:48 AM UTC. 2023-11-17T05:22:59.406 INFO:teuthology.orchestra.run.smithi003.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-17T05:22:59.647 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-17T05:22:59.649 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2023-11-17T05:22:59.649 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-17T05:22:59.755 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install librbd1 2023-11-17T05:23:00.033 INFO:teuthology.orchestra.run.smithi160.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-17T05:23:00.249 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-11-17T05:23:00.250 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2023-11-17T05:23:00.250 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-11-17T05:23:00.353 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install python3-rbd 2023-11-17T05:23:00.750 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:04:08 ago on Fri 17 Nov 2023 05:18:52 AM UTC. 2023-11-17T05:23:01.218 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:04:13 ago on Fri 17 Nov 2023 05:18:48 AM UTC. 2023-11-17T05:23:02.375 INFO:teuthology.orchestra.run.smithi003.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-17T05:23:02.617 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-17T05:23:02.618 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2023-11-17T05:23:02.618 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-17T05:23:02.669 INFO:teuthology.orchestra.run.smithi160.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-17T05:23:02.744 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-rados 2023-11-17T05:23:02.904 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-11-17T05:23:02.905 INFO:teuthology.orchestra.run.smithi160.stdout:Nothing to do. 2023-11-17T05:23:02.905 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-11-17T05:23:03.012 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install rbd-fuse 2023-11-17T05:23:03.849 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:04:11 ago on Fri 17 Nov 2023 05:18:52 AM UTC. 2023-11-17T05:23:03.872 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:04:15 ago on Fri 17 Nov 2023 05:18:48 AM UTC. 2023-11-17T05:23:05.461 INFO:teuthology.orchestra.run.smithi003.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-17T05:23:05.539 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-11-17T05:23:05.540 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:23:05.540 INFO:teuthology.orchestra.run.smithi160.stdout: Package Architecture Version Repository Size 2023-11-17T05:23:05.540 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:23:05.540 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-11-17T05:23:05.540 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2023-11-17T05:23:05.540 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:23:05.540 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-11-17T05:23:05.540 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:23:05.540 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2023-11-17T05:23:05.540 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:23:05.541 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 89 k 2023-11-17T05:23:05.541 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 298 k 2023-11-17T05:23:05.541 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-11-17T05:23:05.585 INFO:teuthology.orchestra.run.smithi160.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 2.0 MB/s | 89 kB 00:00 2023-11-17T05:23:05.586 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-11-17T05:23:05.587 INFO:teuthology.orchestra.run.smithi160.stdout:Total 1.9 MB/s | 89 kB 00:00 2023-11-17T05:23:05.587 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-11-17T05:23:05.596 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-11-17T05:23:05.596 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-11-17T05:23:05.646 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-11-17T05:23:05.649 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-11-17T05:23:05.709 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-17T05:23:05.710 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2023-11-17T05:23:05.710 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-17T05:23:05.795 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-11-17T05:23:13.915 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-rgw 2023-11-17T05:23:13.916 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-17T05:23:13.916 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-17T05:23:13.916 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-17T05:23:13.916 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:23:13.916 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-11-17T05:23:13.916 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2023-11-17T05:23:13.917 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:23:13.917 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-11-17T05:23:13.917 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install rbd-mirror 2023-11-17T05:23:14.794 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:04:26 ago on Fri 17 Nov 2023 05:18:48 AM UTC. 2023-11-17T05:23:14.919 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:04:22 ago on Fri 17 Nov 2023 05:18:52 AM UTC. 2023-11-17T05:23:16.450 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-11-17T05:23:16.450 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:23:16.450 INFO:teuthology.orchestra.run.smithi160.stdout: Package Architecture Version Repository Size 2023-11-17T05:23:16.450 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:23:16.450 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-11-17T05:23:16.450 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-11-17T05:23:16.450 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:23:16.451 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-11-17T05:23:16.451 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:23:16.451 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2023-11-17T05:23:16.451 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:23:16.451 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 3.7 M 2023-11-17T05:23:16.451 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 18 M 2023-11-17T05:23:16.451 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-11-17T05:23:16.529 INFO:teuthology.orchestra.run.smithi003.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-17T05:23:16.546 INFO:teuthology.orchestra.run.smithi160.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 39 MB/s | 3.7 MB 00:00 2023-11-17T05:23:16.546 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-11-17T05:23:16.546 INFO:teuthology.orchestra.run.smithi160.stdout:Total 39 MB/s | 3.7 MB 00:00 2023-11-17T05:23:16.547 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-11-17T05:23:16.558 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-11-17T05:23:16.558 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-11-17T05:23:16.621 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-11-17T05:23:16.624 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-11-17T05:23:16.775 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-17T05:23:16.776 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2023-11-17T05:23:16.777 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-17T05:23:16.879 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-cephfs 2023-11-17T05:23:17.120 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-11-17T05:23:17.353 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-17T05:23:17.752 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-17T05:23:17.872 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:04:25 ago on Fri 17 Nov 2023 05:18:52 AM UTC. 2023-11-17T05:23:18.411 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-17T05:23:18.411 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:23:18.411 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-11-17T05:23:18.411 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2023-11-17T05:23:18.411 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:23:18.412 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-11-17T05:23:18.649 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum -y install rbd-nbd 2023-11-17T05:23:19.479 INFO:teuthology.orchestra.run.smithi003.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-17T05:23:19.521 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:04:31 ago on Fri 17 Nov 2023 05:18:48 AM UTC. 2023-11-17T05:23:19.722 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-17T05:23:19.723 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2023-11-17T05:23:19.723 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-17T05:23:19.834 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-rbd 2023-11-17T05:23:20.839 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:04:28 ago on Fri 17 Nov 2023 05:18:52 AM UTC. 2023-11-17T05:23:21.167 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2023-11-17T05:23:21.167 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:23:21.167 INFO:teuthology.orchestra.run.smithi160.stdout: Package Architecture Version Repository Size 2023-11-17T05:23:21.167 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:23:21.167 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2023-11-17T05:23:21.167 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2023-11-17T05:23:21.168 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:23:21.169 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2023-11-17T05:23:21.169 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2023-11-17T05:23:21.169 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2023-11-17T05:23:21.169 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:23:21.170 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 173 k 2023-11-17T05:23:21.171 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 649 k 2023-11-17T05:23:21.171 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2023-11-17T05:23:21.225 INFO:teuthology.orchestra.run.smithi160.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.1 MB/s | 173 kB 00:00 2023-11-17T05:23:21.225 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2023-11-17T05:23:21.225 INFO:teuthology.orchestra.run.smithi160.stdout:Total 3.0 MB/s | 173 kB 00:00 2023-11-17T05:23:21.226 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2023-11-17T05:23:21.236 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2023-11-17T05:23:21.236 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2023-11-17T05:23:21.286 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2023-11-17T05:23:21.289 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2023-11-17T05:23:21.412 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2023-11-17T05:23:21.609 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-17T05:23:21.916 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-17T05:23:22.442 INFO:teuthology.orchestra.run.smithi003.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-17T05:23:22.590 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-17T05:23:22.591 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:23:22.591 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2023-11-17T05:23:22.591 INFO:teuthology.orchestra.run.smithi160.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2023-11-17T05:23:22.591 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:23:22.591 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2023-11-17T05:23:22.683 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-17T05:23:22.684 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2023-11-17T05:23:22.684 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-17T05:23:22.789 DEBUG:teuthology.parallel:result is None 2023-11-17T05:23:22.791 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install rbd-fuse 2023-11-17T05:23:23.803 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:04:31 ago on Fri 17 Nov 2023 05:18:52 AM UTC. 2023-11-17T05:23:25.668 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-17T05:23:25.668 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:23:25.668 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2023-11-17T05:23:25.668 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:23:25.668 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-11-17T05:23:25.668 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2023-11-17T05:23:25.668 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:23:25.668 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-11-17T05:23:25.669 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:23:25.669 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-11-17T05:23:25.669 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:23:25.669 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 89 k 2023-11-17T05:23:25.669 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 298 k 2023-11-17T05:23:25.669 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-11-17T05:23:25.725 INFO:teuthology.orchestra.run.smithi003.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.6 MB/s | 89 kB 00:00 2023-11-17T05:23:25.726 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-11-17T05:23:25.726 INFO:teuthology.orchestra.run.smithi003.stdout:Total 1.5 MB/s | 89 kB 00:00 2023-11-17T05:23:25.726 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-11-17T05:23:25.738 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-11-17T05:23:25.739 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-11-17T05:23:25.795 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-11-17T05:23:25.799 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-11-17T05:23:25.921 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-11-17T05:23:26.100 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-17T05:23:26.433 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-17T05:23:27.201 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-17T05:23:27.201 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:23:27.201 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-11-17T05:23:27.201 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2023-11-17T05:23:27.201 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:23:27.201 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-17T05:23:27.424 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install rbd-mirror 2023-11-17T05:23:28.460 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:04:36 ago on Fri 17 Nov 2023 05:18:52 AM UTC. 2023-11-17T05:23:30.304 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-17T05:23:30.305 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:23:30.306 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2023-11-17T05:23:30.306 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:23:30.306 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-11-17T05:23:30.306 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-11-17T05:23:30.306 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:23:30.307 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-11-17T05:23:30.307 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:23:30.307 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-11-17T05:23:30.307 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:23:30.307 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 3.7 M 2023-11-17T05:23:30.307 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 18 M 2023-11-17T05:23:30.308 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-11-17T05:23:30.411 INFO:teuthology.orchestra.run.smithi003.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 36 MB/s | 3.7 MB 00:00 2023-11-17T05:23:30.411 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-11-17T05:23:30.412 INFO:teuthology.orchestra.run.smithi003.stdout:Total 35 MB/s | 3.7 MB 00:00 2023-11-17T05:23:30.412 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-11-17T05:23:30.426 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-11-17T05:23:30.426 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-11-17T05:23:30.498 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-11-17T05:23:30.501 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-11-17T05:23:31.009 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-11-17T05:23:31.242 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-17T05:23:31.723 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-17T05:23:34.633 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-17T05:23:34.633 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:23:34.633 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-11-17T05:23:34.633 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2023-11-17T05:23:34.634 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:23:34.634 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-17T05:23:34.635 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install rbd-nbd 2023-11-17T05:23:35.634 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:04:43 ago on Fri 17 Nov 2023 05:18:52 AM UTC. 2023-11-17T05:24:01.002 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2023-11-17T05:24:01.002 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:24:01.002 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2023-11-17T05:24:01.002 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:24:01.002 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2023-11-17T05:24:01.002 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2023-11-17T05:24:01.003 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:24:01.003 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2023-11-17T05:24:01.003 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2023-11-17T05:24:01.003 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2023-11-17T05:24:01.003 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:24:01.003 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 173 k 2023-11-17T05:24:01.003 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 649 k 2023-11-17T05:24:01.003 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2023-11-17T05:24:01.003 INFO:teuthology.orchestra.run.smithi003.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.6 MB/s | 173 kB 00:00 2023-11-17T05:24:01.003 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2023-11-17T05:24:01.003 INFO:teuthology.orchestra.run.smithi003.stdout:Total 3.5 MB/s | 173 kB 00:00 2023-11-17T05:24:01.003 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2023-11-17T05:24:01.004 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2023-11-17T05:24:01.004 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2023-11-17T05:24:01.004 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2023-11-17T05:24:01.004 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2023-11-17T05:24:01.004 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2023-11-17T05:24:01.004 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-17T05:24:01.004 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-17T05:24:01.004 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-17T05:24:01.004 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:24:01.004 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2023-11-17T05:24:01.004 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2023-11-17T05:24:01.004 INFO:teuthology.orchestra.run.smithi003.stdout: 2023-11-17T05:24:01.004 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2023-11-17T05:24:01.005 DEBUG:teuthology.parallel:result is None 2023-11-17T05:24:01.005 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-17T05:24:01.199 INFO:teuthology.packaging:ref: None 2023-11-17T05:24:01.199 INFO:teuthology.packaging:tag: v16.2.4 2023-11-17T05:24:01.199 INFO:teuthology.packaging:branch: None 2023-11-17T05:24:01.199 INFO:teuthology.packaging:sha1: ed3c8e926b9de93026c8a7c1d86be5d6821b52ac 2023-11-17T05:24:01.420 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-17T05:24:01.421 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-17T05:24:01.646 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-17T05:24:01.676 INFO:teuthology.orchestra.run.smithi003.stdout:16.2.4-0.el8 2023-11-17T05:24:01.677 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2023-11-17T05:24:01.677 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2023-11-17T05:24:01.677 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-17T05:24:01.677 INFO:teuthology.packaging:ref: None 2023-11-17T05:24:01.678 INFO:teuthology.packaging:tag: v16.2.4 2023-11-17T05:24:01.678 INFO:teuthology.packaging:branch: None 2023-11-17T05:24:01.678 INFO:teuthology.packaging:sha1: ed3c8e926b9de93026c8a7c1d86be5d6821b52ac 2023-11-17T05:24:01.880 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-17T05:24:01.881 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-17T05:24:02.103 DEBUG:teuthology.orchestra.run.smithi160:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-17T05:24:02.131 INFO:teuthology.orchestra.run.smithi160.stdout:16.2.4-0.el8 2023-11-17T05:24:02.131 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2023-11-17T05:24:02.131 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2023-11-17T05:24:02.131 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-11-17T05:24:02.132 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-11-17T05:24:02.132 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-17T05:24:02.168 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-11-17T05:24:02.168 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-17T05:24:02.203 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-11-17T05:24:02.203 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-11-17T05:24:02.203 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/daemon-helper 2023-11-17T05:24:02.242 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-17T05:24:02.317 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-11-17T05:24:02.318 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/usr/bin/daemon-helper 2023-11-17T05:24:02.354 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-17T05:24:02.426 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-11-17T05:24:02.427 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-11-17T05:24:02.427 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-17T05:24:02.461 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-17T05:24:02.535 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-11-17T05:24:02.536 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-17T05:24:02.567 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-17T05:24:02.638 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-11-17T05:24:02.639 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-11-17T05:24:02.639 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/stdin-killer 2023-11-17T05:24:02.673 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-17T05:24:02.747 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-11-17T05:24:02.747 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/usr/bin/stdin-killer 2023-11-17T05:24:02.778 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-17T05:24:02.850 INFO:teuthology.run_tasks:Running task print... 2023-11-17T05:24:02.863 INFO:teuthology.task.print:**** done install task... 2023-11-17T05:24:02.863 INFO:teuthology.run_tasks:Running task cephadm... 2023-11-17T05:24:02.971 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': 'ed3c8e926b9de93026c8a7c1d86be5d6821b52ac'} 2023-11-17T05:24:02.972 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2023-11-17T05:24:02.972 INFO:tasks.cephadm:Cluster fsid is 851feb1e-8509-11ee-8dbe-212e2dc638e7 2023-11-17T05:24:02.972 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-11-17T05:24:02.972 INFO:tasks.cephadm:No mon roles; fabricating mons 2023-11-17T05:24:02.973 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi003': '172.21.15.3', 'mon.smithi160': '172.21.15.160'} 2023-11-17T05:24:02.973 INFO:tasks.cephadm:Normalizing hostnames... 2023-11-17T05:24:02.973 DEBUG:teuthology.orchestra.run.smithi003:> sudo hostname $(hostname -s) 2023-11-17T05:24:03.009 DEBUG:teuthology.orchestra.run.smithi160:> sudo hostname $(hostname -s) 2023-11-17T05:24:03.040 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2023-11-17T05:24:03.040 DEBUG:teuthology.orchestra.run.smithi003:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-11-17T05:24:03.194 INFO:teuthology.orchestra.run.smithi003.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Nov 17 05:24 /home/ubuntu/cephtest/cephadm 2023-11-17T05:24:03.194 DEBUG:teuthology.orchestra.run.smithi160:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-11-17T05:24:03.332 INFO:teuthology.orchestra.run.smithi160.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Nov 17 05:24 /home/ubuntu/cephtest/cephadm 2023-11-17T05:24:03.332 DEBUG:teuthology.orchestra.run.smithi003:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-11-17T05:24:03.357 DEBUG:teuthology.orchestra.run.smithi160:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-11-17T05:24:03.387 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /etc/ceph 2023-11-17T05:24:03.431 DEBUG:teuthology.orchestra.run.smithi160:> sudo mkdir -p /etc/ceph 2023-11-17T05:24:03.460 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 777 /etc/ceph 2023-11-17T05:24:03.506 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod 777 /etc/ceph 2023-11-17T05:24:03.535 INFO:tasks.cephadm:Writing seed config... 2023-11-17T05:24:03.536 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-11-17T05:24:03.536 INFO:tasks.cephadm: override: [client] debug client = 20 2023-11-17T05:24:03.536 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-11-17T05:24:03.537 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-11-17T05:24:03.537 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-11-17T05:24:03.537 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2023-11-17T05:24:03.537 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-11-17T05:24:03.537 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-11-17T05:24:03.537 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-11-17T05:24:03.537 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-11-17T05:24:03.537 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-11-17T05:24:03.537 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-11-17T05:24:03.537 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-11-17T05:24:03.537 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-11-17T05:24:03.538 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-11-17T05:24:03.538 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-11-17T05:24:03.538 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-11-17T05:24:03.538 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-11-17T05:24:03.538 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-11-17T05:24:03.538 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-11-17T05:24:03.538 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-11-17T05:24:03.538 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2023-11-17T05:24:03.538 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2023-11-17T05:24:03.539 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2023-11-17T05:24:03.539 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-11-17T05:24:03.539 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-11-17T05:24:03.539 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2023-11-17T05:24:03.539 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2023-11-17T05:24:03.541 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-11-17T05:24:03.541 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-11-17T05:24:03.541 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2023-11-17T05:24:03.541 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-11-17T05:24:03.542 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-11-17T05:24:03.542 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-11-17T05:24:03.542 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-11-17T05:24:03.543 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-11-17T05:24:03.543 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-11-17T05:24:03.544 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-11-17T05:24:03.544 DEBUG:teuthology.orchestra.run.smithi003:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-11-17T05:24:03.569 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 851feb1e-8509-11ee-8dbe-212e2dc638e7 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-11-17T05:24:03.570 DEBUG:teuthology.orchestra.run.smithi003:mon.smithi003> sudo journalctl -f -n 0 -u ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7@mon.smithi003.service 2023-11-17T05:24:03.613 INFO:tasks.cephadm:Bootstrapping... 2023-11-17T05:24:03.613 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.3 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-11-17T05:24:03.647 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:-- Logs begin at Fri 2023-11-17 05:07:29 UTC. -- 2023-11-17T05:24:03.946 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-11-17T05:24:04.669 INFO:teuthology.orchestra.run.smithi003.stderr:/bin/podman: 4.6.1 2023-11-17T05:24:04.679 INFO:teuthology.orchestra.run.smithi003.stderr:Verifying podman|docker is present... 2023-11-17T05:24:04.679 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-11-17T05:24:04.705 INFO:teuthology.orchestra.run.smithi003.stderr:/bin/podman: 4.6.1 2023-11-17T05:24:04.715 INFO:teuthology.orchestra.run.smithi003.stderr:Verifying lvm2 is present... 2023-11-17T05:24:04.715 INFO:teuthology.orchestra.run.smithi003.stderr:Verifying time synchronization is in place... 2023-11-17T05:24:04.715 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl is-enabled chrony.service 2023-11-17T05:24:04.723 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-11-17T05:24:04.724 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl is-active chrony.service 2023-11-17T05:24:04.730 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: inactive 2023-11-17T05:24:04.731 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl is-enabled chronyd.service 2023-11-17T05:24:04.737 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: enabled 2023-11-17T05:24:04.738 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl is-active chronyd.service 2023-11-17T05:24:04.744 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: active 2023-11-17T05:24:04.745 INFO:teuthology.orchestra.run.smithi003.stderr:Unit chronyd.service is enabled and running 2023-11-17T05:24:04.745 INFO:teuthology.orchestra.run.smithi003.stderr:Repeating the final host check... 2023-11-17T05:24:04.745 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-11-17T05:24:04.767 INFO:teuthology.orchestra.run.smithi003.stderr:/bin/podman: 4.6.1 2023-11-17T05:24:04.774 INFO:teuthology.orchestra.run.smithi003.stderr:podman|docker (/bin/podman) is present 2023-11-17T05:24:04.774 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl is present 2023-11-17T05:24:04.774 INFO:teuthology.orchestra.run.smithi003.stderr:lvcreate is present 2023-11-17T05:24:04.774 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl is-enabled chrony.service 2023-11-17T05:24:04.781 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-11-17T05:24:04.781 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl is-active chrony.service 2023-11-17T05:24:04.787 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: inactive 2023-11-17T05:24:04.788 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl is-enabled chronyd.service 2023-11-17T05:24:04.794 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: enabled 2023-11-17T05:24:04.795 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl is-active chronyd.service 2023-11-17T05:24:04.802 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: active 2023-11-17T05:24:04.802 INFO:teuthology.orchestra.run.smithi003.stderr:Unit chronyd.service is enabled and running 2023-11-17T05:24:04.802 INFO:teuthology.orchestra.run.smithi003.stderr:Host looks OK 2023-11-17T05:24:04.803 INFO:teuthology.orchestra.run.smithi003.stderr:Cluster fsid: 851feb1e-8509-11ee-8dbe-212e2dc638e7 2023-11-17T05:24:04.803 INFO:teuthology.orchestra.run.smithi003.stderr:Acquiring lock 140593299061224 on /run/cephadm/851feb1e-8509-11ee-8dbe-212e2dc638e7.lock 2023-11-17T05:24:04.803 INFO:teuthology.orchestra.run.smithi003.stderr:Lock 140593299061224 acquired on /run/cephadm/851feb1e-8509-11ee-8dbe-212e2dc638e7.lock 2023-11-17T05:24:04.803 INFO:teuthology.orchestra.run.smithi003.stderr:Verifying IP 172.21.15.3 port 3300 ... 2023-11-17T05:24:04.803 INFO:teuthology.orchestra.run.smithi003.stderr:Verifying IP 172.21.15.3 port 6789 ... 2023-11-17T05:24:04.803 INFO:teuthology.orchestra.run.smithi003.stderr:Base mon IP is 172.21.15.3, final addrv is [v2:172.21.15.3:3300,v1:172.21.15.3:6789] 2023-11-17T05:24:04.804 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /sbin/ip route ls 2023-11-17T05:24:04.806 INFO:teuthology.orchestra.run.smithi003.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.3 metric 100 2023-11-17T05:24:04.806 INFO:teuthology.orchestra.run.smithi003.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.3 metric 100 2023-11-17T05:24:04.807 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /sbin/ip -6 route ls 2023-11-17T05:24:04.809 INFO:teuthology.orchestra.run.smithi003.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2023-11-17T05:24:04.809 INFO:teuthology.orchestra.run.smithi003.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2023-11-17T05:24:04.809 INFO:teuthology.orchestra.run.smithi003.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1758sec hoplimit 64 pref medium 2023-11-17T05:24:04.810 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /sbin/ip -6 addr ls 2023-11-17T05:24:04.812 INFO:teuthology.orchestra.run.smithi003.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-11-17T05:24:04.812 INFO:teuthology.orchestra.run.smithi003.stderr:/sbin/ip: inet6 ::1/128 scope host 2023-11-17T05:24:04.813 INFO:teuthology.orchestra.run.smithi003.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2023-11-17T05:24:04.813 INFO:teuthology.orchestra.run.smithi003.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2023-11-17T05:24:04.813 INFO:teuthology.orchestra.run.smithi003.stderr:/sbin/ip: inet6 fe80::ec4:7aff:febd:15dc/64 scope link 2023-11-17T05:24:04.813 INFO:teuthology.orchestra.run.smithi003.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2023-11-17T05:24:04.814 INFO:teuthology.orchestra.run.smithi003.stderr:Mon IP 172.21.15.3 is in CIDR network 172.21.0.0/20 2023-11-17T05:24:04.814 INFO:teuthology.orchestra.run.smithi003.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-11-17T05:24:04.815 INFO:teuthology.orchestra.run.smithi003.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2023-11-17T05:24:04.815 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2023-11-17T05:24:04.913 INFO:teuthology.orchestra.run.smithi003.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2023-11-17T05:24:11.391 INFO:teuthology.orchestra.run.smithi003.stderr:/bin/podman: Getting image source signatures 2023-11-17T05:24:11.405 INFO:teuthology.orchestra.run.smithi003.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2023-11-17T05:24:11.406 INFO:teuthology.orchestra.run.smithi003.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2023-11-17T05:24:23.977 INFO:teuthology.orchestra.run.smithi003.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2023-11-17T05:24:25.017 INFO:teuthology.orchestra.run.smithi003.stderr:/bin/podman: Writing manifest to image destination 2023-11-17T05:24:25.017 INFO:teuthology.orchestra.run.smithi003.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2023-11-17T05:24:25.017 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2023-11-17T05:24:25.025 INFO:teuthology.orchestra.run.smithi003.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2023-11-17T05:24:25.663 INFO:teuthology.orchestra.run.smithi003.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2023-11-17T05:24:25.663 INFO:teuthology.orchestra.run.smithi003.stderr:Extracting ceph user uid/gid from container image... 2023-11-17T05:24:25.663 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2023-11-17T05:24:25.957 INFO:teuthology.orchestra.run.smithi003.stderr:stat: 167 167 2023-11-17T05:24:26.285 INFO:teuthology.orchestra.run.smithi003.stderr:Creating initial keys... 2023-11-17T05:24:26.285 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-11-17T05:24:26.581 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-authtool: AQAK+VZl0sMjIhAAHPOQKgndePQexhrvKOJkXg== 2023-11-17T05:24:26.912 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-11-17T05:24:27.231 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-authtool: AQAL+VZls1xIDRAAuVxBR6r4G+9uQflwqLjjMQ== 2023-11-17T05:24:27.581 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-11-17T05:24:27.905 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-authtool: AQAL+VZlRuHFMxAAo+rTI2BDZ7s2m8ZRHZfHcw== 2023-11-17T05:24:28.244 INFO:teuthology.orchestra.run.smithi003.stderr:Creating initial monmap... 2023-11-17T05:24:28.245 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmpandbaxr2:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 --addv smithi003 [v2:172.21.15.3:3300,v1:172.21.15.3:6789] /tmp/monmap 2023-11-17T05:24:28.548 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2023-11-17T05:24:28.548 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 851feb1e-8509-11ee-8dbe-212e2dc638e7 2023-11-17T05:24:28.548 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-11-17T05:24:28.902 INFO:teuthology.orchestra.run.smithi003.stderr:monmaptool for smithi003 [v2:172.21.15.3:3300,v1:172.21.15.3:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-11-17T05:24:28.902 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/monmaptool: set fsid to 851feb1e-8509-11ee-8dbe-212e2dc638e7 2023-11-17T05:24:28.902 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-11-17T05:24:28.903 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-11-17T05:24:28.903 INFO:teuthology.orchestra.run.smithi003.stderr:Creating mon... 2023-11-17T05:24:28.903 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/mon.smithi003:/var/lib/ceph/mon/ceph-smithi003:z -v /tmp/ceph-tmpl0c8prbq:/tmp/keyring:z -v /tmp/ceph-tmpandbaxr2:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi003 --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2023-11-17T05:24:29.183 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.178+0000 7fc8b660d700 0 set uid:gid to 167:167 (ceph:ceph) 2023-11-17T05:24:29.183 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.179+0000 7fc8b660d700 1 imported monmap: 2023-11-17T05:24:29.183 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: epoch 0 2023-11-17T05:24:29.184 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 2023-11-17T05:24:29.184 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: last_changed 2023-11-17T05:24:28.545719+0000 2023-11-17T05:24:29.184 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: created 2023-11-17T05:24:28.545719+0000 2023-11-17T05:24:29.184 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2023-11-17T05:24:29.184 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: election_strategy: 1 2023-11-17T05:24:29.184 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.smithi003 2023-11-17T05:24:29.184 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: 2023-11-17T05:24:29.184 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.179+0000 7fc8b660d700 0 /usr/bin/ceph-mon: set fsid to 851feb1e-8509-11ee-8dbe-212e2dc638e7 2023-11-17T05:24:29.184 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: RocksDB version: 6.8.1 2023-11-17T05:24:29.184 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: 2023-11-17T05:24:29.185 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-11-17T05:24:29.185 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Compile date May 13 2021 2023-11-17T05:24:29.185 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: DB SUMMARY 2023-11-17T05:24:29.185 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: 2023-11-17T05:24:29.185 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi003/store.db dir, Total Num: 0, files: 2023-11-17T05:24:29.185 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: 2023-11-17T05:24:29.221 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi003/store.db: 2023-11-17T05:24:29.221 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: 2023-11-17T05:24:29.222 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.error_if_exists: 0 2023-11-17T05:24:29.222 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.create_if_missing: 1 2023-11-17T05:24:29.222 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.paranoid_checks: 1 2023-11-17T05:24:29.222 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.env: 0x561f4f11d1c0 2023-11-17T05:24:29.222 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.fs: Posix File System 2023-11-17T05:24:29.222 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.info_log: 0x561f4fffcec0 2023-11-17T05:24:29.222 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.max_file_opening_threads: 16 2023-11-17T05:24:29.222 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.statistics: (nil) 2023-11-17T05:24:29.222 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.use_fsync: 0 2023-11-17T05:24:29.222 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.max_log_file_size: 0 2023-11-17T05:24:29.222 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-17T05:24:29.223 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.log_file_time_to_roll: 0 2023-11-17T05:24:29.223 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.keep_log_file_num: 1000 2023-11-17T05:24:29.223 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.recycle_log_file_num: 0 2023-11-17T05:24:29.223 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.allow_fallocate: 1 2023-11-17T05:24:29.223 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.allow_mmap_reads: 0 2023-11-17T05:24:29.223 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.allow_mmap_writes: 0 2023-11-17T05:24:29.223 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.use_direct_reads: 0 2023-11-17T05:24:29.223 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-17T05:24:29.223 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.create_missing_column_families: 0 2023-11-17T05:24:29.223 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.db_log_dir: 2023-11-17T05:24:29.223 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi003/store.db 2023-11-17T05:24:29.223 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.table_cache_numshardbits: 6 2023-11-17T05:24:29.223 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.max_subcompactions: 1 2023-11-17T05:24:29.224 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.max_background_flushes: -1 2023-11-17T05:24:29.224 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-11-17T05:24:29.224 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-11-17T05:24:29.224 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-17T05:24:29.224 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-17T05:24:29.224 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-11-17T05:24:29.224 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.advise_random_on_open: 1 2023-11-17T05:24:29.224 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.db_write_buffer_size: 0 2023-11-17T05:24:29.224 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.write_buffer_manager: 0x561f50003530 2023-11-17T05:24:29.224 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-17T05:24:29.224 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-11-17T05:24:29.224 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-17T05:24:29.225 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.use_adaptive_mutex: 0 2023-11-17T05:24:29.225 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.rate_limiter: (nil) 2023-11-17T05:24:29.225 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-17T05:24:29.225 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.wal_recovery_mode: 2 2023-11-17T05:24:29.225 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.enable_thread_tracking: 0 2023-11-17T05:24:29.225 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.enable_pipelined_write: 0 2023-11-17T05:24:29.225 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.unordered_write: 0 2023-11-17T05:24:29.225 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-17T05:24:29.225 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-17T05:24:29.225 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-17T05:24:29.225 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-17T05:24:29.225 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.row_cache: None 2023-11-17T05:24:29.225 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.wal_filter: None 2023-11-17T05:24:29.226 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-17T05:24:29.226 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.allow_ingest_behind: 0 2023-11-17T05:24:29.226 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.preserve_deletes: 0 2023-11-17T05:24:29.226 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.two_write_queues: 0 2023-11-17T05:24:29.226 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.manual_wal_flush: 0 2023-11-17T05:24:29.226 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.atomic_flush: 0 2023-11-17T05:24:29.226 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-17T05:24:29.226 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.persist_stats_to_disk: 0 2023-11-17T05:24:29.226 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-11-17T05:24:29.226 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.log_readahead_size: 0 2023-11-17T05:24:29.226 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-11-17T05:24:29.226 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.max_background_jobs: 2 2023-11-17T05:24:29.227 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.max_background_compactions: -1 2023-11-17T05:24:29.227 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-17T05:24:29.227 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-17T05:24:29.228 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.delayed_write_rate : 16777216 2023-11-17T05:24:29.228 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.max_total_wal_size: 0 2023-11-17T05:24:29.228 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-17T05:24:29.229 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.stats_dump_period_sec: 600 2023-11-17T05:24:29.229 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.stats_persist_period_sec: 600 2023-11-17T05:24:29.229 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-17T05:24:29.229 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.max_open_files: -1 2023-11-17T05:24:29.229 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.bytes_per_sync: 0 2023-11-17T05:24:29.229 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-11-17T05:24:29.229 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-11-17T05:24:29.229 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Options.compaction_readahead_size: 0 2023-11-17T05:24:29.229 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Compression algorithms supported: 2023-11-17T05:24:29.229 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-17T05:24:29.229 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: kZSTD supported: 0 2023-11-17T05:24:29.229 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: kXpressCompression supported: 0 2023-11-17T05:24:29.230 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: kLZ4HCCompression supported: 1 2023-11-17T05:24:29.230 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: kLZ4Compression supported: 1 2023-11-17T05:24:29.230 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: kBZip2Compression supported: 0 2023-11-17T05:24:29.230 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: kZlibCompression supported: 1 2023-11-17T05:24:29.230 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: kSnappyCompression supported: 1 2023-11-17T05:24:29.230 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-11-17T05:24:29.230 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.180+0000 7fc8b660d700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2023-11-17T05:24:29.230 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: 2023-11-17T05:24:29.230 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi003/store.db/MANIFEST-000001 2023-11-17T05:24:29.230 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: 2023-11-17T05:24:29.230 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-11-17T05:24:29.230 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: 2023-11-17T05:24:29.231 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-17T05:24:29.231 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.merge_operator: 2023-11-17T05:24:29.231 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.compaction_filter: None 2023-11-17T05:24:29.231 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.compaction_filter_factory: None 2023-11-17T05:24:29.231 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-11-17T05:24:29.231 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.table_factory: BlockBasedTable 2023-11-17T05:24:29.231 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x561f4ff14120) 2023-11-17T05:24:29.231 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2023-11-17T05:24:29.231 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2023-11-17T05:24:29.231 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2023-11-17T05:24:29.231 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2023-11-17T05:24:29.231 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: index_type: 0 2023-11-17T05:24:29.231 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2023-11-17T05:24:29.232 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: index_shortening: 1 2023-11-17T05:24:29.232 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2023-11-17T05:24:29.232 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2023-11-17T05:24:29.232 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: checksum: 1 2023-11-17T05:24:29.232 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2023-11-17T05:24:29.232 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: block_cache: 0x561f4ff4ad10 2023-11-17T05:24:29.232 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2023-11-17T05:24:29.232 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: block_cache_options: 2023-11-17T05:24:29.232 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: capacity : 536870912 2023-11-17T05:24:29.232 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2023-11-17T05:24:29.232 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2023-11-17T05:24:29.232 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2023-11-17T05:24:29.232 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2023-11-17T05:24:29.233 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2023-11-17T05:24:29.233 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: block_size: 4096 2023-11-17T05:24:29.233 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2023-11-17T05:24:29.233 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2023-11-17T05:24:29.233 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2023-11-17T05:24:29.233 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2023-11-17T05:24:29.233 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: partition_filters: 0 2023-11-17T05:24:29.233 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2023-11-17T05:24:29.233 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2023-11-17T05:24:29.233 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2023-11-17T05:24:29.233 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: verify_compression: 0 2023-11-17T05:24:29.233 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2023-11-17T05:24:29.233 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: format_version: 2 2023-11-17T05:24:29.233 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2023-11-17T05:24:29.234 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: block_align: 0 2023-11-17T05:24:29.234 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: 2023-11-17T05:24:29.234 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.write_buffer_size: 33554432 2023-11-17T05:24:29.234 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.max_write_buffer_number: 2 2023-11-17T05:24:29.234 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.compression: NoCompression 2023-11-17T05:24:29.234 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.bottommost_compression: Disabled 2023-11-17T05:24:29.234 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.prefix_extractor: nullptr 2023-11-17T05:24:29.234 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-17T05:24:29.234 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.num_levels: 7 2023-11-17T05:24:29.234 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-17T05:24:29.234 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-17T05:24:29.234 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-17T05:24:29.234 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-17T05:24:29.235 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-17T05:24:29.235 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-17T05:24:29.235 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-17T05:24:29.235 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-17T05:24:29.235 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-17T05:24:29.235 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.compression_opts.window_bits: -14 2023-11-17T05:24:29.235 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.compression_opts.level: 32767 2023-11-17T05:24:29.235 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.compression_opts.strategy: 0 2023-11-17T05:24:29.235 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-17T05:24:29.235 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-17T05:24:29.235 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.compression_opts.enabled: false 2023-11-17T05:24:29.235 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-17T05:24:29.235 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-17T05:24:29.236 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-17T05:24:29.236 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.target_file_size_base: 67108864 2023-11-17T05:24:29.236 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.target_file_size_multiplier: 1 2023-11-17T05:24:29.236 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-17T05:24:29.236 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-17T05:24:29.236 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-17T05:24:29.236 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-17T05:24:29.236 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-17T05:24:29.236 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-17T05:24:29.236 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-17T05:24:29.236 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-17T05:24:29.236 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-17T05:24:29.236 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-17T05:24:29.237 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-17T05:24:29.237 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-17T05:24:29.237 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.arena_block_size: 4194304 2023-11-17T05:24:29.237 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-17T05:24:29.237 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-17T05:24:29.237 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-11-17T05:24:29.237 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.disable_auto_compactions: 0 2023-11-17T05:24:29.237 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-17T05:24:29.237 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-17T05:24:29.237 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-17T05:24:29.237 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-17T05:24:29.237 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-17T05:24:29.237 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-17T05:24:29.237 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-17T05:24:29.238 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-17T05:24:29.238 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-17T05:24:29.238 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-17T05:24:29.238 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.table_properties_collectors: 2023-11-17T05:24:29.238 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.inplace_update_support: 0 2023-11-17T05:24:29.238 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-11-17T05:24:29.238 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-17T05:24:29.238 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-17T05:24:29.238 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.memtable_huge_page_size: 0 2023-11-17T05:24:29.238 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.bloom_locality: 0 2023-11-17T05:24:29.238 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.max_successive_merges: 0 2023-11-17T05:24:29.238 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-11-17T05:24:29.239 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.paranoid_file_checks: 0 2023-11-17T05:24:29.239 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.force_consistency_checks: 0 2023-11-17T05:24:29.239 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.report_bg_io_stats: 0 2023-11-17T05:24:29.239 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.ttl: 2592000 2023-11-17T05:24:29.239 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.181+0000 7fc8b660d700 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-11-17T05:24:29.239 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.182+0000 7fc8b660d700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi003/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-11-17T05:24:29.239 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: 2023-11-17T05:24:29.239 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.182+0000 7fc8b660d700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2023-11-17T05:24:29.239 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: 2023-11-17T05:24:29.239 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.182+0000 7fc8b660d700 4 rocksdb: DB pointer 0x561f5000f800 2023-11-17T05:24:29.239 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.183+0000 7fc89f4a6700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-11-17T05:24:29.239 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.183+0000 7fc89f4a6700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-11-17T05:24:29.239 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2023-11-17T05:24:29.240 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-11-17T05:24:29.240 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-11-17T05:24:29.240 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-17T05:24:29.240 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-17T05:24:29.240 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-11-17T05:24:29.240 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-11-17T05:24:29.240 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-17T05:24:29.240 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: 2023-11-17T05:24:29.240 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-17T05:24:29.240 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-17T05:24:29.240 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-17T05:24:29.240 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-17T05:24:29.240 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-17T05:24:29.241 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: 2023-11-17T05:24:29.241 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-17T05:24:29.241 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-17T05:24:29.241 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-17T05:24:29.241 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-11-17T05:24:29.241 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-11-17T05:24:29.241 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-17T05:24:29.241 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-11-17T05:24:29.241 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-11-17T05:24:29.241 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-11-17T05:24:29.241 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-17T05:24:29.241 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-17T05:24:29.241 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-17T05:24:29.241 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: 2023-11-17T05:24:29.242 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-11-17T05:24:29.242 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: 2023-11-17T05:24:29.242 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-17T05:24:29.242 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-17T05:24:29.242 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-17T05:24:29.242 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-17T05:24:29.242 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-17T05:24:29.242 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: 2023-11-17T05:24:29.242 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-17T05:24:29.242 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-17T05:24:29.242 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-17T05:24:29.242 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-11-17T05:24:29.242 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-11-17T05:24:29.243 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-17T05:24:29.243 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-11-17T05:24:29.243 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-11-17T05:24:29.243 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-11-17T05:24:29.243 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-17T05:24:29.243 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-17T05:24:29.243 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-17T05:24:29.243 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: 2023-11-17T05:24:29.243 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-11-17T05:24:29.243 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: 2023-11-17T05:24:29.243 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.184+0000 7fc8b660d700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2023-11-17T05:24:29.243 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.184+0000 7fc8b660d700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2023-11-17T05:24:29.243 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph-mon: debug 2023-11-17T05:24:29.184+0000 7fc8b660d700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi003 for mon.smithi003 2023-11-17T05:24:29.548 INFO:teuthology.orchestra.run.smithi003.stderr:create mon.smithi003 on 2023-11-17T05:24:29.549 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7 2023-11-17T05:24:29.555 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl disable ceph.target 2023-11-17T05:24:29.565 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-11-17T05:24:29.659 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl enable ceph.target 2023-11-17T05:24:29.667 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-11-17T05:24:29.758 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl start ceph.target 2023-11-17T05:24:29.766 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl enable ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7.target 2023-11-17T05:24:29.772 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7.target → /etc/systemd/system/ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7.target. 2023-11-17T05:24:29.772 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7.target → /etc/systemd/system/ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7.target. 2023-11-17T05:24:29.861 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl start ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7.target 2023-11-17T05:24:29.869 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl daemon-reload 2023-11-17T05:24:29.969 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl stop ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7@mon.smithi003 2023-11-17T05:24:30.009 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl reset-failed ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7@mon.smithi003 2023-11-17T05:24:30.013 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: Failed to reset failed state of unit ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7@mon.smithi003.service: Unit ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7@mon.smithi003.service not loaded. 2023-11-17T05:24:30.013 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl enable ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7@mon.smithi003 2023-11-17T05:24:30.020 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: Created symlink /etc/systemd/system/ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7.target.wants/ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7@mon.smithi003.service → /etc/systemd/system/ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7@.service. 2023-11-17T05:24:30.118 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl start ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7@mon.smithi003 2023-11-17T05:24:30.477 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:30 smithi003 systemd[1]: Starting Ceph mon.smithi003 for 851feb1e-8509-11ee-8dbe-212e2dc638e7... 2023-11-17T05:24:30.483 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl is-enabled firewalld.service 2023-11-17T05:24:30.502 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: disabled 2023-11-17T05:24:30.502 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl is-active firewalld.service 2023-11-17T05:24:30.502 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: inactive 2023-11-17T05:24:30.502 INFO:teuthology.orchestra.run.smithi003.stderr:firewalld.service is not enabled 2023-11-17T05:24:30.502 INFO:teuthology.orchestra.run.smithi003.stderr:Not possible to enable service . firewalld.service is not available 2023-11-17T05:24:30.502 INFO:teuthology.orchestra.run.smithi003.stderr:Not possible to open ports <[]>. firewalld.service is not available 2023-11-17T05:24:30.502 INFO:teuthology.orchestra.run.smithi003.stderr:Waiting for mon to start... 2023-11-17T05:24:30.502 INFO:teuthology.orchestra.run.smithi003.stderr:Waiting for mon... 2023-11-17T05:24:30.502 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/mon.smithi003:/var/lib/ceph/mon/ceph-smithi003:z -v /tmp/ceph-tmptcuja8tf:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpk7o1jie2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2023-11-17T05:24:30.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:30 smithi003 bash[111387]: ad2106b1dfb4512041927c65cda74570e1879b15c96849ae12f9cc6143f89f05 2023-11-17T05:24:30.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:30 smithi003 systemd[1]: Started Ceph mon.smithi003 for 851feb1e-8509-11ee-8dbe-212e2dc638e7. 2023-11-17T05:24:30.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:30 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111401]: cluster 2023-11-17T05:24:30.510443+0000 mon.smithi003 (mon.0) 0 : [INF] mkfs 851feb1e-8509-11ee-8dbe-212e2dc638e7 2023-11-17T05:24:30.771 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:30 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111401]: cluster 2023-11-17T05:24:30.492195+0000 mon.smithi003 (mon.0) 1 : cluster [INF] mon.smithi003 is new leader, mons smithi003 in quorum (ranks 0) 2023-11-17T05:24:30.905 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.904+0000 7f35be2e1700 1 Processor -- start 2023-11-17T05:24:30.905 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.904+0000 7f35be2e1700 1 -- start start 2023-11-17T05:24:30.905 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.904+0000 7f35be2e1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35b806fcd0 0x7f35b80fe7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-17T05:24:30.905 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.904+0000 7f35be2e1700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b80fed20 con 0x7f35b806fcd0 2023-11-17T05:24:30.908 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.905+0000 7f35b7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35b806fcd0 0x7f35b80fe7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:24:30.908 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.905+0000 7f35b7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35b806fcd0 0x7f35b80fe7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60626/0 (socket says 172.21.15.3:60626) 2023-11-17T05:24:30.908 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.905+0000 7f35b7fff700 1 -- 172.21.15.3:0/2445689026 learned_addr learned my addr 172.21.15.3:0/2445689026 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:24:30.909 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.905+0000 7f35b7fff700 1 -- 172.21.15.3:0/2445689026 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b80fee60 con 0x7f35b806fcd0 2023-11-17T05:24:30.909 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.905+0000 7f35b7fff700 1 --2- 172.21.15.3:0/2445689026 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35b806fcd0 0x7f35b80fe7a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f35ac00b5f0 tx=0x7f35ac006ee0).ready entity=mon.0 client_cookie=3101b1ef088b71a7 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:24:30.909 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.905+0000 7f35b6ffd700 1 -- 172.21.15.3:0/2445689026 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35ac00d600 con 0x7f35b806fcd0 2023-11-17T05:24:30.909 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.905+0000 7f35b6ffd700 1 -- 172.21.15.3:0/2445689026 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35ac00d760 con 0x7f35b806fcd0 2023-11-17T05:24:30.909 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.905+0000 7f35b6ffd700 1 -- 172.21.15.3:0/2445689026 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35ac00da30 con 0x7f35b806fcd0 2023-11-17T05:24:30.909 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.906+0000 7f35be2e1700 1 -- 172.21.15.3:0/2445689026 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35b806fcd0 msgr2=0x7f35b80fe7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:24:30.909 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.906+0000 7f35be2e1700 1 --2- 172.21.15.3:0/2445689026 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35b806fcd0 0x7f35b80fe7a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f35ac00b5f0 tx=0x7f35ac006ee0).stop 2023-11-17T05:24:30.909 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.906+0000 7f35be2e1700 1 -- 172.21.15.3:0/2445689026 shutdown_connections 2023-11-17T05:24:30.909 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.906+0000 7f35be2e1700 1 --2- 172.21.15.3:0/2445689026 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35b806fcd0 0x7f35b80fe7a0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:24:30.909 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.906+0000 7f35be2e1700 1 -- 172.21.15.3:0/2445689026 >> 172.21.15.3:0/2445689026 conn(0x7f35b80f6280 msgr2=0x7f35b80f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:24:30.910 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.906+0000 7f35be2e1700 1 -- 172.21.15.3:0/2445689026 shutdown_connections 2023-11-17T05:24:30.910 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.906+0000 7f35be2e1700 1 -- 172.21.15.3:0/2445689026 wait complete. 2023-11-17T05:24:30.910 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.907+0000 7f35be2e1700 1 Processor -- start 2023-11-17T05:24:30.910 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.907+0000 7f35be2e1700 1 -- start start 2023-11-17T05:24:30.910 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.907+0000 7f35be2e1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35b806fcd0 0x7f35b810c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:24:30.910 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.907+0000 7f35be2e1700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b810c8d0 con 0x7f35b806fcd0 2023-11-17T05:24:30.910 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.907+0000 7f35b7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35b806fcd0 0x7f35b810c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:24:30.910 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.907+0000 7f35b7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35b806fcd0 0x7f35b810c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60638/0 (socket says 172.21.15.3:60638) 2023-11-17T05:24:30.910 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.907+0000 7f35b7fff700 1 -- 172.21.15.3:0/4130757325 learned_addr learned my addr 172.21.15.3:0/4130757325 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:24:30.911 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.908+0000 7f35b7fff700 1 -- 172.21.15.3:0/4130757325 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35ac00b200 con 0x7f35b806fcd0 2023-11-17T05:24:30.911 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.908+0000 7f35b7fff700 1 --2- 172.21.15.3:0/4130757325 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35b806fcd0 0x7f35b810c3e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f35ac006c70 tx=0x7f35ac016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:24:30.911 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.908+0000 7f35b4ff9700 1 -- 172.21.15.3:0/4130757325 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35ac011b40 con 0x7f35b806fcd0 2023-11-17T05:24:30.911 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.908+0000 7f35b4ff9700 1 -- 172.21.15.3:0/4130757325 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35ac011ca0 con 0x7f35b806fcd0 2023-11-17T05:24:30.911 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.908+0000 7f35be2e1700 1 -- 172.21.15.3:0/4130757325 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35b810cad0 con 0x7f35b806fcd0 2023-11-17T05:24:30.911 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.908+0000 7f35b4ff9700 1 -- 172.21.15.3:0/4130757325 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35ac01d430 con 0x7f35b806fcd0 2023-11-17T05:24:30.912 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.908+0000 7f35be2e1700 1 -- 172.21.15.3:0/4130757325 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35b810cee0 con 0x7f35b806fcd0 2023-11-17T05:24:30.912 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.908+0000 7f35b4ff9700 1 -- 172.21.15.3:0/4130757325 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f35ac00f070 con 0x7f35b806fcd0 2023-11-17T05:24:30.912 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.908+0000 7f35b4ff9700 1 -- 172.21.15.3:0/4130757325 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f35ac027460 con 0x7f35b806fcd0 2023-11-17T05:24:30.912 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.909+0000 7f35be2e1700 1 -- 172.21.15.3:0/4130757325 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35b8045ad0 con 0x7f35b806fcd0 2023-11-17T05:24:30.912 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.911+0000 7f35b4ff9700 1 -- 172.21.15.3:0/4130757325 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f35ac016070 con 0x7f35b806fcd0 2023-11-17T05:24:30.985 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.984+0000 7f35be2e1700 1 -- 172.21.15.3:0/4130757325 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f35b8106250 con 0x7f35b806fcd0 2023-11-17T05:24:30.985 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.984+0000 7f35b4ff9700 1 -- 172.21.15.3:0/4130757325 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+324 (secure 0 0 0) 0x7f35ac035160 con 0x7f35b806fcd0 2023-11-17T05:24:30.986 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: cluster: 2023-11-17T05:24:30.986 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: id: 851feb1e-8509-11ee-8dbe-212e2dc638e7 2023-11-17T05:24:30.986 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: health: HEALTH_OK 2023-11-17T05:24:30.986 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.986 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: services: 2023-11-17T05:24:30.986 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi003 (age 0.47585s) 2023-11-17T05:24:30.986 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: mgr: no daemons active 2023-11-17T05:24:30.986 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2023-11-17T05:24:30.986 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.986 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: data: 2023-11-17T05:24:30.986 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2023-11-17T05:24:30.986 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2023-11-17T05:24:30.988 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2023-11-17T05:24:30.988 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: pgs: 2023-11-17T05:24:30.988 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.988 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.985+0000 7f35be2e1700 1 -- 172.21.15.3:0/4130757325 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35b806fcd0 msgr2=0x7f35b810c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:24:30.988 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.985+0000 7f35be2e1700 1 --2- 172.21.15.3:0/4130757325 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35b806fcd0 0x7f35b810c3e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f35ac006c70 tx=0x7f35ac016000).stop 2023-11-17T05:24:30.988 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.985+0000 7f35be2e1700 1 -- 172.21.15.3:0/4130757325 shutdown_connections 2023-11-17T05:24:30.988 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.985+0000 7f35be2e1700 1 --2- 172.21.15.3:0/4130757325 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35b806fcd0 0x7f35b810c3e0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:24:30.988 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.985+0000 7f35be2e1700 1 -- 172.21.15.3:0/4130757325 >> 172.21.15.3:0/4130757325 conn(0x7f35b80f6280 msgr2=0x7f35b80f7fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:24:30.988 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.985+0000 7f35be2e1700 1 -- 172.21.15.3:0/4130757325 shutdown_connections 2023-11-17T05:24:30.988 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:30.985+0000 7f35be2e1700 1 -- 172.21.15.3:0/4130757325 wait complete. 2023-11-17T05:24:31.334 INFO:teuthology.orchestra.run.smithi003.stderr:mon is available 2023-11-17T05:24:31.334 INFO:teuthology.orchestra.run.smithi003.stderr:Assimilating anything we can from ceph.conf... 2023-11-17T05:24:31.334 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptcuja8tf:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpk7o1jie2:/etc/ceph/ceph.conf:z -v /var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/mon.smithi003:/var/lib/ceph/mon/ceph-smithi003:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi003/config 2023-11-17T05:24:31.745 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.744+0000 7fb5f669c700 1 Processor -- start 2023-11-17T05:24:31.745 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.744+0000 7fb5f669c700 1 -- start start 2023-11-17T05:24:31.746 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.744+0000 7fb5f669c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5f006fcd0 0x7fb5f00fe7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-17T05:24:31.746 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.744+0000 7fb5f669c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f00fed20 con 0x7fb5f006fcd0 2023-11-17T05:24:31.746 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.745+0000 7fb5effff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5f006fcd0 0x7fb5f00fe7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:24:31.746 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.745+0000 7fb5effff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5f006fcd0 0x7fb5f00fe7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60642/0 (socket says 172.21.15.3:60642) 2023-11-17T05:24:31.746 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.745+0000 7fb5effff700 1 -- 172.21.15.3:0/927901436 learned_addr learned my addr 172.21.15.3:0/927901436 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:24:31.746 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.745+0000 7fb5effff700 1 -- 172.21.15.3:0/927901436 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5f00fee60 con 0x7fb5f006fcd0 2023-11-17T05:24:31.746 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.745+0000 7fb5effff700 1 --2- 172.21.15.3:0/927901436 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5f006fcd0 0x7fb5f00fe7a0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fb5d800b5f0 tx=0x7fb5d8006ee0).ready entity=mon.0 client_cookie=6eea4bdaf48cb852 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:24:31.746 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.745+0000 7fb5eeffd700 1 -- 172.21.15.3:0/927901436 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5d800d600 con 0x7fb5f006fcd0 2023-11-17T05:24:31.747 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.745+0000 7fb5eeffd700 1 -- 172.21.15.3:0/927901436 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5d800d760 con 0x7fb5f006fcd0 2023-11-17T05:24:31.747 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.745+0000 7fb5eeffd700 1 -- 172.21.15.3:0/927901436 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5d800da30 con 0x7fb5f006fcd0 2023-11-17T05:24:31.748 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.746+0000 7fb5f669c700 1 -- 172.21.15.3:0/927901436 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5f006fcd0 msgr2=0x7fb5f00fe7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:24:31.748 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.746+0000 7fb5f669c700 1 --2- 172.21.15.3:0/927901436 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5f006fcd0 0x7fb5f00fe7a0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fb5d800b5f0 tx=0x7fb5d8006ee0).stop 2023-11-17T05:24:31.748 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.746+0000 7fb5f669c700 1 -- 172.21.15.3:0/927901436 shutdown_connections 2023-11-17T05:24:31.748 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.746+0000 7fb5f669c700 1 --2- 172.21.15.3:0/927901436 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5f006fcd0 0x7fb5f00fe7a0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:24:31.748 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.746+0000 7fb5f669c700 1 -- 172.21.15.3:0/927901436 >> 172.21.15.3:0/927901436 conn(0x7fb5f00f6280 msgr2=0x7fb5f00f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:24:31.748 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.746+0000 7fb5f669c700 1 -- 172.21.15.3:0/927901436 shutdown_connections 2023-11-17T05:24:31.749 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.746+0000 7fb5f669c700 1 -- 172.21.15.3:0/927901436 wait complete. 2023-11-17T05:24:31.749 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.746+0000 7fb5f669c700 1 Processor -- start 2023-11-17T05:24:31.749 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.747+0000 7fb5f669c700 1 -- start start 2023-11-17T05:24:31.749 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.747+0000 7fb5f669c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5f006fcd0 0x7fb5f00fc220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:24:31.749 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.747+0000 7fb5f669c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f00fa6a0 con 0x7fb5f006fcd0 2023-11-17T05:24:31.749 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.747+0000 7fb5effff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5f006fcd0 0x7fb5f00fc220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:24:31.749 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.747+0000 7fb5effff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5f006fcd0 0x7fb5f00fc220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60654/0 (socket says 172.21.15.3:60654) 2023-11-17T05:24:31.749 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.747+0000 7fb5effff700 1 -- 172.21.15.3:0/2582038699 learned_addr learned my addr 172.21.15.3:0/2582038699 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:24:31.750 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.747+0000 7fb5effff700 1 -- 172.21.15.3:0/2582038699 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5d800b200 con 0x7fb5f006fcd0 2023-11-17T05:24:31.750 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.747+0000 7fb5effff700 1 --2- 172.21.15.3:0/2582038699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5f006fcd0 0x7fb5f00fc220 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fb5d8006c70 tx=0x7fb5d8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:24:31.750 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.748+0000 7fb5ecff9700 1 -- 172.21.15.3:0/2582038699 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5d8011b40 con 0x7fb5f006fcd0 2023-11-17T05:24:31.750 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.748+0000 7fb5ecff9700 1 -- 172.21.15.3:0/2582038699 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5d8011ca0 con 0x7fb5f006fcd0 2023-11-17T05:24:31.750 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.748+0000 7fb5f669c700 1 -- 172.21.15.3:0/2582038699 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5f00fa840 con 0x7fb5f006fcd0 2023-11-17T05:24:31.750 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.748+0000 7fb5ecff9700 1 -- 172.21.15.3:0/2582038699 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5d801d430 con 0x7fb5f006fcd0 2023-11-17T05:24:31.750 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.748+0000 7fb5f669c700 1 -- 172.21.15.3:0/2582038699 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5f00fac50 con 0x7fb5f006fcd0 2023-11-17T05:24:31.750 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.748+0000 7fb5ecff9700 1 -- 172.21.15.3:0/2582038699 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fb5d800f070 con 0x7fb5f006fcd0 2023-11-17T05:24:31.750 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.748+0000 7fb5ecff9700 1 -- 172.21.15.3:0/2582038699 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fb5d8027460 con 0x7fb5f006fcd0 2023-11-17T05:24:31.750 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.749+0000 7fb5f669c700 1 -- 172.21.15.3:0/2582038699 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5f0059100 con 0x7fb5f006fcd0 2023-11-17T05:24:31.752 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.751+0000 7fb5ecff9700 1 -- 172.21.15.3:0/2582038699 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fb5d8016070 con 0x7fb5f006fcd0 2023-11-17T05:24:31.774 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:31 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111401]: cluster 2023-11-17T05:24:30.509017+0000 mon.smithi003 (mon.0) 2 : cluster [INF] mon.smithi003 is new leader, mons smithi003 in quorum (ranks 0) 2023-11-17T05:24:31.774 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:31 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111401]: cluster 2023-11-17T05:24:30.509279+0000 mon.smithi003 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi003=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0]} 2023-11-17T05:24:31.774 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:31 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111401]: cluster 2023-11-17T05:24:30.512809+0000 mon.smithi003 (mon.0) 4 : cluster [DBG] fsmap 2023-11-17T05:24:31.774 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:31 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111401]: cluster 2023-11-17T05:24:30.515084+0000 mon.smithi003 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-11-17T05:24:31.774 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:31 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111401]: cluster 2023-11-17T05:24:30.515535+0000 mon.smithi003 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2023-11-17T05:24:31.774 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:31 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111401]: audit 2023-11-17T05:24:30.984833+0000 mon.smithi003 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.3:0/4130757325' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-11-17T05:24:31.823 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.821+0000 7fb5f669c700 1 -- 172.21.15.3:0/2582038699 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fb5f00ff360 con 0x7fb5f006fcd0 2023-11-17T05:24:31.826 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.825+0000 7fb5ecff9700 1 -- 172.21.15.3:0/2582038699 <== mon.0 v2:172.21.15.3:3300/0 7 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb5d801d620 con 0x7fb5f006fcd0 2023-11-17T05:24:31.826 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.825+0000 7fb5ecff9700 1 -- 172.21.15.3:0/2582038699 <== mon.0 v2:172.21.15.3:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+462 (secure 0 0 0) 0x7fb5f00ff360 con 0x7fb5f006fcd0 2023-11-17T05:24:31.828 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.829 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: [global] 2023-11-17T05:24:31.829 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: fsid = 851feb1e-8509-11ee-8dbe-212e2dc638e7 2023-11-17T05:24:31.829 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.3:3300,v1:172.21.15.3:6789] 2023-11-17T05:24:31.829 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2023-11-17T05:24:31.829 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2023-11-17T05:24:31.829 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2023-11-17T05:24:31.829 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2023-11-17T05:24:31.829 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.829 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: [mgr] 2023-11-17T05:24:31.829 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2023-11-17T05:24:31.829 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.829 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: [osd] 2023-11-17T05:24:31.829 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2023-11-17T05:24:31.830 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2023-11-17T05:24:31.830 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2023-11-17T05:24:31.830 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2023-11-17T05:24:31.830 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2023-11-17T05:24:31.830 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2023-11-17T05:24:31.830 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.826+0000 7fb5f669c700 1 -- 172.21.15.3:0/2582038699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5f006fcd0 msgr2=0x7fb5f00fc220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:24:31.830 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.826+0000 7fb5f669c700 1 --2- 172.21.15.3:0/2582038699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5f006fcd0 0x7fb5f00fc220 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fb5d8006c70 tx=0x7fb5d8016000).stop 2023-11-17T05:24:31.830 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.826+0000 7fb5f669c700 1 -- 172.21.15.3:0/2582038699 shutdown_connections 2023-11-17T05:24:31.830 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.826+0000 7fb5f669c700 1 --2- 172.21.15.3:0/2582038699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5f006fcd0 0x7fb5f00fc220 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:24:31.830 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.826+0000 7fb5f669c700 1 -- 172.21.15.3:0/2582038699 >> 172.21.15.3:0/2582038699 conn(0x7fb5f00f6280 msgr2=0x7fb5f00f8040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:24:31.830 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.827+0000 7fb5f669c700 1 -- 172.21.15.3:0/2582038699 shutdown_connections 2023-11-17T05:24:31.830 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:31.827+0000 7fb5f669c700 1 -- 172.21.15.3:0/2582038699 wait complete. 2023-11-17T05:24:32.175 INFO:teuthology.orchestra.run.smithi003.stderr:Generating new minimal ceph.conf... 2023-11-17T05:24:32.176 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptcuja8tf:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpk7o1jie2:/etc/ceph/ceph.conf:z -v /var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/mon.smithi003:/var/lib/ceph/mon/ceph-smithi003:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi003/config 2023-11-17T05:24:32.563 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.561+0000 7f2023c45700 1 Processor -- start 2023-11-17T05:24:32.563 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.562+0000 7f2023c45700 1 -- start start 2023-11-17T05:24:32.563 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.562+0000 7f2023c45700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f201c0faae0 0x7f201c0faf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-17T05:24:32.563 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.562+0000 7f2023c45700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f201c0fb480 con 0x7f201c0faae0 2023-11-17T05:24:32.563 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.562+0000 7f20219e1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f201c0faae0 0x7f201c0faf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:24:32.563 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.562+0000 7f20219e1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f201c0faae0 0x7f201c0faf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60658/0 (socket says 172.21.15.3:60658) 2023-11-17T05:24:32.563 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.562+0000 7f20219e1700 1 -- 172.21.15.3:0/1444438836 learned_addr learned my addr 172.21.15.3:0/1444438836 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:24:32.565 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.562+0000 7f20219e1700 1 -- 172.21.15.3:0/1444438836 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f201c0fb5c0 con 0x7f201c0faae0 2023-11-17T05:24:32.565 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.563+0000 7f20219e1700 1 --2- 172.21.15.3:0/1444438836 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f201c0faae0 0x7f201c0faf00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f201800b870 tx=0x7f2018006f70).ready entity=mon.0 client_cookie=35737d968ca70b21 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:24:32.565 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.563+0000 7f20209df700 1 -- 172.21.15.3:0/1444438836 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f201800da20 con 0x7f201c0faae0 2023-11-17T05:24:32.565 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.563+0000 7f20209df700 1 -- 172.21.15.3:0/1444438836 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f2018010470 con 0x7f201c0faae0 2023-11-17T05:24:32.565 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.563+0000 7f20209df700 1 -- 172.21.15.3:0/1444438836 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f201800f5b0 con 0x7f201c0faae0 2023-11-17T05:24:32.565 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.563+0000 7f2023c45700 1 -- 172.21.15.3:0/1444438836 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f201c0faae0 msgr2=0x7f201c0faf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:24:32.565 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.563+0000 7f2023c45700 1 --2- 172.21.15.3:0/1444438836 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f201c0faae0 0x7f201c0faf00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f201800b870 tx=0x7f2018006f70).stop 2023-11-17T05:24:32.565 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.563+0000 7f2023c45700 1 -- 172.21.15.3:0/1444438836 shutdown_connections 2023-11-17T05:24:32.565 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.563+0000 7f2023c45700 1 --2- 172.21.15.3:0/1444438836 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f201c0faae0 0x7f201c0faf00 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:24:32.565 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.563+0000 7f2023c45700 1 -- 172.21.15.3:0/1444438836 >> 172.21.15.3:0/1444438836 conn(0x7f201c0f6280 msgr2=0x7f201c0f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:24:32.566 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.564+0000 7f2023c45700 1 -- 172.21.15.3:0/1444438836 shutdown_connections 2023-11-17T05:24:32.566 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.564+0000 7f2023c45700 1 -- 172.21.15.3:0/1444438836 wait complete. 2023-11-17T05:24:32.567 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.565+0000 7f2023c45700 1 Processor -- start 2023-11-17T05:24:32.567 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.565+0000 7f2023c45700 1 -- start start 2023-11-17T05:24:32.567 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.565+0000 7f2023c45700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f201c0faae0 0x7f201c108750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:24:32.567 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.565+0000 7f2023c45700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f201c108c40 con 0x7f201c0faae0 2023-11-17T05:24:32.568 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.565+0000 7f20219e1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f201c0faae0 0x7f201c108750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:24:32.568 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.565+0000 7f20219e1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f201c0faae0 0x7f201c108750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60670/0 (socket says 172.21.15.3:60670) 2023-11-17T05:24:32.568 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.565+0000 7f20219e1700 1 -- 172.21.15.3:0/1193116402 learned_addr learned my addr 172.21.15.3:0/1193116402 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:24:32.568 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.566+0000 7f20219e1700 1 -- 172.21.15.3:0/1193116402 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f201800b480 con 0x7f201c0faae0 2023-11-17T05:24:32.568 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.566+0000 7f20219e1700 1 --2- 172.21.15.3:0/1193116402 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f201c0faae0 0x7f201c108750 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f2018006b80 tx=0x7f2018006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:24:32.568 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.566+0000 7f200a7fc700 1 -- 172.21.15.3:0/1193116402 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f201800f3d0 con 0x7f201c0faae0 2023-11-17T05:24:32.568 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.566+0000 7f200a7fc700 1 -- 172.21.15.3:0/1193116402 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f201800fef0 con 0x7f201c0faae0 2023-11-17T05:24:32.568 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.566+0000 7f200a7fc700 1 -- 172.21.15.3:0/1193116402 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2018019c70 con 0x7f201c0faae0 2023-11-17T05:24:32.568 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.566+0000 7f2023c45700 1 -- 172.21.15.3:0/1193116402 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f201c108e40 con 0x7f201c0faae0 2023-11-17T05:24:32.568 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.566+0000 7f2023c45700 1 -- 172.21.15.3:0/1193116402 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f201c109250 con 0x7f201c0faae0 2023-11-17T05:24:32.568 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.566+0000 7f200a7fc700 1 -- 172.21.15.3:0/1193116402 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f2018016030 con 0x7f201c0faae0 2023-11-17T05:24:32.568 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.566+0000 7f200a7fc700 1 -- 172.21.15.3:0/1193116402 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f2018024790 con 0x7f201c0faae0 2023-11-17T05:24:32.569 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.567+0000 7f2023c45700 1 -- 172.21.15.3:0/1193116402 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f201c045ad0 con 0x7f201c0faae0 2023-11-17T05:24:32.570 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.570+0000 7f200a7fc700 1 -- 172.21.15.3:0/1193116402 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f2018014070 con 0x7f201c0faae0 2023-11-17T05:24:32.640 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.640+0000 7f2023c45700 1 -- 172.21.15.3:0/1193116402 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f201c0fc8c0 con 0x7f201c0faae0 2023-11-17T05:24:32.641 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.640+0000 7f200a7fc700 1 -- 172.21.15.3:0/1193116402 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+173 (secure 0 0 0) 0x7f201800de10 con 0x7f201c0faae0 2023-11-17T05:24:32.642 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.641+0000 7f2023c45700 1 -- 172.21.15.3:0/1193116402 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f201c0faae0 msgr2=0x7f201c108750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:24:32.642 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.641+0000 7f2023c45700 1 --2- 172.21.15.3:0/1193116402 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f201c0faae0 0x7f201c108750 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f2018006b80 tx=0x7f2018006f70).stop 2023-11-17T05:24:32.642 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.641+0000 7f2023c45700 1 -- 172.21.15.3:0/1193116402 shutdown_connections 2023-11-17T05:24:32.642 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.641+0000 7f2023c45700 1 --2- 172.21.15.3:0/1193116402 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f201c0faae0 0x7f201c108750 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:24:32.642 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.641+0000 7f2023c45700 1 -- 172.21.15.3:0/1193116402 >> 172.21.15.3:0/1193116402 conn(0x7f201c0f6280 msgr2=0x7f201c0ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:24:32.642 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.641+0000 7f2023c45700 1 -- 172.21.15.3:0/1193116402 shutdown_connections 2023-11-17T05:24:32.642 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:32.641+0000 7f2023c45700 1 -- 172.21.15.3:0/1193116402 wait complete. 2023-11-17T05:24:32.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:32 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111401]: audit 2023-11-17T05:24:31.822668+0000 mon.smithi003 (mon.0) 8 : audit [INF] from='client.? 172.21.15.3:0/2582038699' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-11-17T05:24:32.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:32 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111401]: audit 2023-11-17T05:24:31.826263+0000 mon.smithi003 (mon.0) 9 : audit [INF] from='client.? 172.21.15.3:0/2582038699' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-11-17T05:24:33.026 INFO:teuthology.orchestra.run.smithi003.stderr:Restarting the monitor... 2023-11-17T05:24:33.026 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl restart ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7@mon.smithi003 2023-11-17T05:24:33.112 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 systemd[1]: Stopping Ceph mon.smithi003 for 851feb1e-8509-11ee-8dbe-212e2dc638e7... 2023-11-17T05:24:33.415 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111401]: debug 2023-11-17T05:24:33.111+0000 7f122e9c0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi003 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-11-17T05:24:33.415 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111401]: debug 2023-11-17T05:24:33.111+0000 7f122e9c0700 -1 mon.smithi003@0(leader) e1 *** Got Signal Terminated *** 2023-11-17T05:24:33.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 podman[111723]: ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003 2023-11-17T05:24:33.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 systemd[1]: ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7@mon.smithi003.service: Succeeded. 2023-11-17T05:24:33.771 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 systemd[1]: Stopped Ceph mon.smithi003 for 851feb1e-8509-11ee-8dbe-212e2dc638e7. 2023-11-17T05:24:33.771 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 systemd[1]: Starting Ceph mon.smithi003 for 851feb1e-8509-11ee-8dbe-212e2dc638e7... 2023-11-17T05:24:33.876 INFO:teuthology.orchestra.run.smithi003.stderr:Setting mon public_network to 172.21.0.0/20 2023-11-17T05:24:33.877 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptcuja8tf:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpk7o1jie2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2023-11-17T05:24:34.260 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.257+0000 7f431df0f700 1 Processor -- start 2023-11-17T05:24:34.260 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.257+0000 7f431df0f700 1 -- start start 2023-11-17T05:24:34.260 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.257+0000 7f431df0f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f431806fbd0 0x7f4318100ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-17T05:24:34.260 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.257+0000 7f431df0f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43180fc4a0 con 0x7f431806fbd0 2023-11-17T05:24:34.260 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.258+0000 7f431cf0d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f431806fbd0 0x7f4318100ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:24:34.260 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.258+0000 7f431cf0d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f431806fbd0 0x7f4318100ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60680/0 (socket says 172.21.15.3:60680) 2023-11-17T05:24:34.260 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.258+0000 7f431cf0d700 1 -- 172.21.15.3:0/1004056124 learned_addr learned my addr 172.21.15.3:0/1004056124 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:24:34.260 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.258+0000 7f431cf0d700 1 -- 172.21.15.3:0/1004056124 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4318100fa0 con 0x7f431806fbd0 2023-11-17T05:24:34.261 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.258+0000 7f431cf0d700 1 --2- 172.21.15.3:0/1004056124 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f431806fbd0 0x7f4318100ab0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f430c00b5f0 tx=0x7f430c006ee0).ready entity=mon.0 client_cookie=d9b0e5e1f2c8891d server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:24:34.261 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.259+0000 7f43177fe700 1 -- 172.21.15.3:0/1004056124 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f430c00d600 con 0x7f431806fbd0 2023-11-17T05:24:34.261 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.259+0000 7f43177fe700 1 -- 172.21.15.3:0/1004056124 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f430c00dbd0 con 0x7f431806fbd0 2023-11-17T05:24:34.261 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.259+0000 7f43177fe700 1 -- 172.21.15.3:0/1004056124 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f430c011c00 con 0x7f431806fbd0 2023-11-17T05:24:34.261 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.259+0000 7f431df0f700 1 -- 172.21.15.3:0/1004056124 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f431806fbd0 msgr2=0x7f4318100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:24:34.263 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.259+0000 7f431df0f700 1 --2- 172.21.15.3:0/1004056124 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f431806fbd0 0x7f4318100ab0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f430c00b5f0 tx=0x7f430c006ee0).stop 2023-11-17T05:24:34.263 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.259+0000 7f431df0f700 1 -- 172.21.15.3:0/1004056124 shutdown_connections 2023-11-17T05:24:34.264 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.259+0000 7f431df0f700 1 --2- 172.21.15.3:0/1004056124 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f431806fbd0 0x7f4318100ab0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:24:34.264 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.259+0000 7f431df0f700 1 -- 172.21.15.3:0/1004056124 >> 172.21.15.3:0/1004056124 conn(0x7f43180f6130 msgr2=0x7f43180f8550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:24:34.264 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.260+0000 7f431df0f700 1 -- 172.21.15.3:0/1004056124 shutdown_connections 2023-11-17T05:24:34.264 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.260+0000 7f431df0f700 1 -- 172.21.15.3:0/1004056124 wait complete. 2023-11-17T05:24:34.264 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.260+0000 7f431df0f700 1 Processor -- start 2023-11-17T05:24:34.264 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.260+0000 7f431df0f700 1 -- start start 2023-11-17T05:24:34.264 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.261+0000 7f431df0f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f431806fbd0 0x7f431810c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:24:34.264 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.261+0000 7f431df0f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f431810ce70 con 0x7f431806fbd0 2023-11-17T05:24:34.264 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.261+0000 7f431cf0d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f431806fbd0 0x7f431810c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:24:34.264 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.261+0000 7f431cf0d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f431806fbd0 0x7f431810c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60690/0 (socket says 172.21.15.3:60690) 2023-11-17T05:24:34.264 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.261+0000 7f431cf0d700 1 -- 172.21.15.3:0/1118544803 learned_addr learned my addr 172.21.15.3:0/1118544803 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:24:34.264 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.261+0000 7f431cf0d700 1 -- 172.21.15.3:0/1118544803 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f430c00b200 con 0x7f431806fbd0 2023-11-17T05:24:34.264 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.261+0000 7f431cf0d700 1 --2- 172.21.15.3:0/1118544803 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f431806fbd0 0x7f431810c980 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f430c011820 tx=0x7f430c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:24:34.265 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.261+0000 7f43157fa700 1 -- 172.21.15.3:0/1118544803 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f430c01b410 con 0x7f431806fbd0 2023-11-17T05:24:34.265 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.261+0000 7f43157fa700 1 -- 172.21.15.3:0/1118544803 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f430c01b9e0 con 0x7f431806fbd0 2023-11-17T05:24:34.265 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.261+0000 7f431df0f700 1 -- 172.21.15.3:0/1118544803 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f431810d070 con 0x7f431806fbd0 2023-11-17T05:24:34.265 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.261+0000 7f43157fa700 1 -- 172.21.15.3:0/1118544803 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f430c01a8b0 con 0x7f431806fbd0 2023-11-17T05:24:34.265 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.261+0000 7f431df0f700 1 -- 172.21.15.3:0/1118544803 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f431810d480 con 0x7f431806fbd0 2023-11-17T05:24:34.265 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.262+0000 7f43157fa700 1 -- 172.21.15.3:0/1118544803 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f430c00f070 con 0x7f431806fbd0 2023-11-17T05:24:34.265 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.262+0000 7f43157fa700 1 -- 172.21.15.3:0/1118544803 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f430c025bc0 con 0x7f431806fbd0 2023-11-17T05:24:34.265 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.263+0000 7f431df0f700 1 -- 172.21.15.3:0/1118544803 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4318045ad0 con 0x7f431806fbd0 2023-11-17T05:24:34.265 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.265+0000 7f43157fa700 1 -- 172.21.15.3:0/1118544803 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f430c016070 con 0x7f431806fbd0 2023-11-17T05:24:34.271 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 bash[111795]: da01184d6c1412db9c7915d1cf8570ec30db24569dfd56aa88c2f3ec3bb1396b 2023-11-17T05:24:34.271 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.872+0000 7f3930b01700 0 set uid:gid to 167:167 (ceph:ceph) 2023-11-17T05:24:34.271 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.872+0000 7f3930b01700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2023-11-17T05:24:34.272 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.872+0000 7f3930b01700 0 pidfile_write: ignore empty --pid-file 2023-11-17T05:24:34.272 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 systemd[1]: Started Ceph mon.smithi003 for 851feb1e-8509-11ee-8dbe-212e2dc638e7. 2023-11-17T05:24:34.272 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.877+0000 7f3930b01700 0 load: jerasure load: lrc load: isa 2023-11-17T05:24:34.272 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: RocksDB version: 6.8.1 2023-11-17T05:24:34.272 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: 2023-11-17T05:24:34.272 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-11-17T05:24:34.272 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Compile date May 13 2021 2023-11-17T05:24:34.272 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: DB SUMMARY 2023-11-17T05:24:34.273 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: 2023-11-17T05:24:34.273 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: CURRENT file: CURRENT 2023-11-17T05:24:34.273 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: 2023-11-17T05:24:34.273 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: IDENTITY file: IDENTITY 2023-11-17T05:24:34.273 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: 2023-11-17T05:24:34.274 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2023-11-17T05:24:34.276 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: 2023-11-17T05:24:34.276 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi003/store.db dir, Total Num: 1, files: 000004.sst 2023-11-17T05:24:34.276 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: 2023-11-17T05:24:34.276 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi003/store.db: 000006.log size: 92436 ; 2023-11-17T05:24:34.276 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: 2023-11-17T05:24:34.276 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.error_if_exists: 0 2023-11-17T05:24:34.276 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.create_if_missing: 0 2023-11-17T05:24:34.276 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.paranoid_checks: 1 2023-11-17T05:24:34.276 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.env: 0x5655246c81c0 2023-11-17T05:24:34.276 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.fs: Posix File System 2023-11-17T05:24:34.276 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.info_log: 0x565526d77960 2023-11-17T05:24:34.277 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.max_file_opening_threads: 16 2023-11-17T05:24:34.277 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.statistics: (nil) 2023-11-17T05:24:34.277 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.use_fsync: 0 2023-11-17T05:24:34.277 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.max_log_file_size: 0 2023-11-17T05:24:34.277 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-17T05:24:34.277 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.log_file_time_to_roll: 0 2023-11-17T05:24:34.277 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.keep_log_file_num: 1000 2023-11-17T05:24:34.277 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.recycle_log_file_num: 0 2023-11-17T05:24:34.277 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.allow_fallocate: 1 2023-11-17T05:24:34.277 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.allow_mmap_reads: 0 2023-11-17T05:24:34.277 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.allow_mmap_writes: 0 2023-11-17T05:24:34.277 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.use_direct_reads: 0 2023-11-17T05:24:34.277 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-17T05:24:34.278 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.create_missing_column_families: 0 2023-11-17T05:24:34.278 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.db_log_dir: 2023-11-17T05:24:34.278 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi003/store.db 2023-11-17T05:24:34.278 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.table_cache_numshardbits: 6 2023-11-17T05:24:34.278 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.max_subcompactions: 1 2023-11-17T05:24:34.278 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.max_background_flushes: -1 2023-11-17T05:24:34.278 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-11-17T05:24:34.278 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-11-17T05:24:34.278 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-17T05:24:34.278 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-17T05:24:34.278 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-11-17T05:24:34.283 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.advise_random_on_open: 1 2023-11-17T05:24:34.283 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.db_write_buffer_size: 0 2023-11-17T05:24:34.283 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.write_buffer_manager: 0x5655279ec030 2023-11-17T05:24:34.283 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-17T05:24:34.283 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-11-17T05:24:34.283 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-17T05:24:34.283 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.use_adaptive_mutex: 0 2023-11-17T05:24:34.284 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.rate_limiter: (nil) 2023-11-17T05:24:34.284 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-17T05:24:34.284 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.wal_recovery_mode: 2 2023-11-17T05:24:34.284 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.enable_thread_tracking: 0 2023-11-17T05:24:34.284 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.enable_pipelined_write: 0 2023-11-17T05:24:34.284 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.unordered_write: 0 2023-11-17T05:24:34.284 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-17T05:24:34.284 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-17T05:24:34.284 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-17T05:24:34.284 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-17T05:24:34.285 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.row_cache: None 2023-11-17T05:24:34.285 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.wal_filter: None 2023-11-17T05:24:34.285 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-17T05:24:34.285 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.allow_ingest_behind: 0 2023-11-17T05:24:34.285 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.preserve_deletes: 0 2023-11-17T05:24:34.285 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.two_write_queues: 0 2023-11-17T05:24:34.285 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.manual_wal_flush: 0 2023-11-17T05:24:34.285 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.atomic_flush: 0 2023-11-17T05:24:34.285 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-17T05:24:34.285 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.persist_stats_to_disk: 0 2023-11-17T05:24:34.286 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-11-17T05:24:34.286 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.log_readahead_size: 0 2023-11-17T05:24:34.286 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-11-17T05:24:34.286 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.max_background_jobs: 2 2023-11-17T05:24:34.286 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.max_background_compactions: -1 2023-11-17T05:24:34.286 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-17T05:24:34.286 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-17T05:24:34.286 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.delayed_write_rate : 16777216 2023-11-17T05:24:34.286 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.max_total_wal_size: 0 2023-11-17T05:24:34.286 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-17T05:24:34.286 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.stats_dump_period_sec: 600 2023-11-17T05:24:34.287 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.stats_persist_period_sec: 600 2023-11-17T05:24:34.287 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-17T05:24:34.288 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.max_open_files: -1 2023-11-17T05:24:34.288 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.bytes_per_sync: 0 2023-11-17T05:24:34.288 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-11-17T05:24:34.288 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-11-17T05:24:34.288 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Options.compaction_readahead_size: 0 2023-11-17T05:24:34.288 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Compression algorithms supported: 2023-11-17T05:24:34.288 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-17T05:24:34.289 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: kZSTD supported: 0 2023-11-17T05:24:34.290 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: kXpressCompression supported: 0 2023-11-17T05:24:34.290 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: kLZ4HCCompression supported: 1 2023-11-17T05:24:34.290 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: kLZ4Compression supported: 1 2023-11-17T05:24:34.290 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: kBZip2Compression supported: 0 2023-11-17T05:24:34.291 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: kZlibCompression supported: 1 2023-11-17T05:24:34.291 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: kSnappyCompression supported: 1 2023-11-17T05:24:34.292 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.878+0000 7f3930b01700 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-11-17T05:24:34.292 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi003/store.db/MANIFEST-000005 2023-11-17T05:24:34.292 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: 2023-11-17T05:24:34.293 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-11-17T05:24:34.293 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: 2023-11-17T05:24:34.293 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-17T05:24:34.293 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.merge_operator: 2023-11-17T05:24:34.293 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.compaction_filter: None 2023-11-17T05:24:34.293 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.compaction_filter_factory: None 2023-11-17T05:24:34.293 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-11-17T05:24:34.293 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.table_factory: BlockBasedTable 2023-11-17T05:24:34.293 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x565526c8e128) 2023-11-17T05:24:34.293 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cache_index_and_filter_blocks: 1 2023-11-17T05:24:34.293 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cache_index_and_filter_blocks_with_high_priority: 0 2023-11-17T05:24:34.293 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: pin_l0_filter_and_index_blocks_in_cache: 0 2023-11-17T05:24:34.294 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: pin_top_level_index_and_filter: 1 2023-11-17T05:24:34.294 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: index_type: 0 2023-11-17T05:24:34.294 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: data_block_index_type: 0 2023-11-17T05:24:34.294 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: index_shortening: 1 2023-11-17T05:24:34.294 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: data_block_hash_table_util_ratio: 0.750000 2023-11-17T05:24:34.294 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: hash_index_allow_collision: 1 2023-11-17T05:24:34.295 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: checksum: 1 2023-11-17T05:24:34.295 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: no_block_cache: 0 2023-11-17T05:24:34.295 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: block_cache: 0x565526cc4f10 2023-11-17T05:24:34.295 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: block_cache_name: BinnedLRUCache 2023-11-17T05:24:34.295 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: block_cache_options: 2023-11-17T05:24:34.295 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: capacity : 536870912 2023-11-17T05:24:34.295 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: num_shard_bits : 4 2023-11-17T05:24:34.295 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: strict_capacity_limit : 0 2023-11-17T05:24:34.295 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: high_pri_pool_ratio: 0.000 2023-11-17T05:24:34.295 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: block_cache_compressed: (nil) 2023-11-17T05:24:34.295 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: persistent_cache: (nil) 2023-11-17T05:24:34.295 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: block_size: 4096 2023-11-17T05:24:34.296 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: block_size_deviation: 10 2023-11-17T05:24:34.296 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: block_restart_interval: 16 2023-11-17T05:24:34.296 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: index_block_restart_interval: 1 2023-11-17T05:24:34.296 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: metadata_block_size: 4096 2023-11-17T05:24:34.296 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: partition_filters: 0 2023-11-17T05:24:34.296 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: use_delta_encoding: 1 2023-11-17T05:24:34.296 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: filter_policy: rocksdb.BuiltinBloomFilter 2023-11-17T05:24:34.296 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: whole_key_filtering: 1 2023-11-17T05:24:34.296 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: verify_compression: 0 2023-11-17T05:24:34.296 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: read_amp_bytes_per_bit: 0 2023-11-17T05:24:34.296 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: format_version: 2 2023-11-17T05:24:34.296 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: enable_index_compression: 1 2023-11-17T05:24:34.297 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: block_align: 0 2023-11-17T05:24:34.297 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: 2023-11-17T05:24:34.297 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.write_buffer_size: 33554432 2023-11-17T05:24:34.297 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.max_write_buffer_number: 2 2023-11-17T05:24:34.297 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.compression: NoCompression 2023-11-17T05:24:34.297 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.bottommost_compression: Disabled 2023-11-17T05:24:34.297 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.prefix_extractor: nullptr 2023-11-17T05:24:34.297 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-17T05:24:34.297 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.num_levels: 7 2023-11-17T05:24:34.297 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-17T05:24:34.297 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-17T05:24:34.298 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-17T05:24:34.298 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-17T05:24:34.298 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-17T05:24:34.298 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-17T05:24:34.298 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-17T05:24:34.298 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-17T05:24:34.298 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-17T05:24:34.298 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.compression_opts.window_bits: -14 2023-11-17T05:24:34.298 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.compression_opts.level: 32767 2023-11-17T05:24:34.298 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.compression_opts.strategy: 0 2023-11-17T05:24:34.299 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-17T05:24:34.299 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-17T05:24:34.299 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.compression_opts.enabled: false 2023-11-17T05:24:34.299 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-17T05:24:34.299 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-17T05:24:34.299 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-17T05:24:34.299 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.target_file_size_base: 67108864 2023-11-17T05:24:34.299 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.target_file_size_multiplier: 1 2023-11-17T05:24:34.299 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-17T05:24:34.299 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-17T05:24:34.299 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-17T05:24:34.299 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-17T05:24:34.300 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-17T05:24:34.300 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-17T05:24:34.300 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-17T05:24:34.300 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-17T05:24:34.300 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-17T05:24:34.300 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-17T05:24:34.300 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-17T05:24:34.300 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-17T05:24:34.300 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.arena_block_size: 4194304 2023-11-17T05:24:34.300 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-17T05:24:34.300 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-17T05:24:34.300 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-11-17T05:24:34.300 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.disable_auto_compactions: 0 2023-11-17T05:24:34.301 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-17T05:24:34.301 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-17T05:24:34.301 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-17T05:24:34.301 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-17T05:24:34.301 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-17T05:24:34.301 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-17T05:24:34.301 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-17T05:24:34.301 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-17T05:24:34.301 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-17T05:24:34.301 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-17T05:24:34.301 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.table_properties_collectors: 2023-11-17T05:24:34.302 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.inplace_update_support: 0 2023-11-17T05:24:34.302 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-11-17T05:24:34.302 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-17T05:24:34.302 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-17T05:24:34.302 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.memtable_huge_page_size: 0 2023-11-17T05:24:34.302 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.bloom_locality: 0 2023-11-17T05:24:34.302 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.max_successive_merges: 0 2023-11-17T05:24:34.302 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-11-17T05:24:34.302 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.paranoid_file_checks: 0 2023-11-17T05:24:34.302 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.force_consistency_checks: 0 2023-11-17T05:24:34.303 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.report_bg_io_stats: 0 2023-11-17T05:24:34.303 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.ttl: 2592000 2023-11-17T05:24:34.303 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.879+0000 7f3930b01700 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-11-17T05:24:34.303 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.880+0000 7f3930b01700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi003/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-11-17T05:24:34.303 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: 2023-11-17T05:24:34.303 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.880+0000 7f3930b01700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2023-11-17T05:24:34.303 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: 2023-11-17T05:24:34.303 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.880+0000 7f3930b01700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700198673880804, "job": 1, "event": "recovery_started", "log_files": [6]} 2023-11-17T05:24:34.303 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.880+0000 7f3930b01700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2023-11-17T05:24:34.303 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.881+0000 7f3930b01700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2023-11-17T05:24:34.303 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.881+0000 7f3930b01700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700198673882351, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 88236, "table_properties": {"data_size": 86363, "index_size": 350, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12269, "raw_average_key_size": 49, "raw_value_size": 79577, "raw_average_value_size": 322, "num_data_blocks": 11, "num_entries": 247, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1700198673, "oldest_key_time": 3, "file_creation_time": 0}} 2023-11-17T05:24:34.303 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.881+0000 7f3930b01700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2023-11-17T05:24:34.304 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: 2023-11-17T05:24:34.304 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.883+0000 7f3930b01700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700198673883645, "job": 1, "event": "recovery_finished"} 2023-11-17T05:24:34.304 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.884+0000 7f3930b01700 4 rocksdb: DB pointer 0x565526d89800 2023-11-17T05:24:34.304 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.884+0000 7f3917919700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-11-17T05:24:34.304 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.884+0000 7f3917919700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-11-17T05:24:34.304 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: ** DB Stats ** 2023-11-17T05:24:34.304 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: Uptime(secs): 0.0 total, 0.0 interval 2023-11-17T05:24:34.304 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-11-17T05:24:34.304 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-17T05:24:34.304 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-17T05:24:34.304 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-11-17T05:24:34.304 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-11-17T05:24:34.304 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-17T05:24:34.305 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: 2023-11-17T05:24:34.305 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: ** Compaction Stats [default] ** 2023-11-17T05:24:34.305 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-17T05:24:34.305 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-17T05:24:34.305 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: L0 2/0 87.86 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 81.9 0.00 0.00 1 0.001 0 0 2023-11-17T05:24:34.305 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: Sum 2/0 87.86 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 81.9 0.00 0.00 1 0.001 0 0 2023-11-17T05:24:34.305 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 81.9 0.00 0.00 1 0.001 0 0 2023-11-17T05:24:34.305 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: 2023-11-17T05:24:34.305 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: ** Compaction Stats [default] ** 2023-11-17T05:24:34.305 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-17T05:24:34.305 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-17T05:24:34.305 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 81.9 0.00 0.00 1 0.001 0 0 2023-11-17T05:24:34.305 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: Uptime(secs): 0.0 total, 0.0 interval 2023-11-17T05:24:34.306 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: Flush(GB): cumulative 0.000, interval 0.000 2023-11-17T05:24:34.306 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-17T05:24:34.306 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: AddFile(Total Files): cumulative 0, interval 0 2023-11-17T05:24:34.306 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: AddFile(L0 Files): cumulative 0, interval 0 2023-11-17T05:24:34.306 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: AddFile(Keys): cumulative 0, interval 0 2023-11-17T05:24:34.306 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: Cumulative compaction: 0.00 GB write, 15.35 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-17T05:24:34.306 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: Interval compaction: 0.00 GB write, 15.35 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-17T05:24:34.306 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-17T05:24:34.306 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: 2023-11-17T05:24:34.306 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: ** File Read Latency Histogram By Level [default] ** 2023-11-17T05:24:34.306 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: 2023-11-17T05:24:34.306 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: ** Compaction Stats [default] ** 2023-11-17T05:24:34.307 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-17T05:24:34.307 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-17T05:24:34.307 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: L0 2/0 87.86 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 81.9 0.00 0.00 1 0.001 0 0 2023-11-17T05:24:34.307 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: Sum 2/0 87.86 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 81.9 0.00 0.00 1 0.001 0 0 2023-11-17T05:24:34.307 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-17T05:24:34.307 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: 2023-11-17T05:24:34.307 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: ** Compaction Stats [default] ** 2023-11-17T05:24:34.307 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-17T05:24:34.307 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-17T05:24:34.307 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 81.9 0.00 0.00 1 0.001 0 0 2023-11-17T05:24:34.307 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: Uptime(secs): 0.0 total, 0.0 interval 2023-11-17T05:24:34.307 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: Flush(GB): cumulative 0.000, interval 0.000 2023-11-17T05:24:34.307 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-17T05:24:34.308 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: AddFile(Total Files): cumulative 0, interval 0 2023-11-17T05:24:34.308 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: AddFile(L0 Files): cumulative 0, interval 0 2023-11-17T05:24:34.308 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: AddFile(Keys): cumulative 0, interval 0 2023-11-17T05:24:34.308 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: Cumulative compaction: 0.00 GB write, 15.25 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-17T05:24:34.308 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-17T05:24:34.308 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-17T05:24:34.308 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: 2023-11-17T05:24:34.308 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: ** File Read Latency Histogram By Level [default] ** 2023-11-17T05:24:34.309 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: 2023-11-17T05:24:34.309 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.885+0000 7f3930b01700 0 starting mon.smithi003 rank 0 at public addrs [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] at bind addrs [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi003 fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 2023-11-17T05:24:34.309 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.887+0000 7f3930b01700 1 mon.smithi003@-1(???) e1 preinit fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 2023-11-17T05:24:34.309 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.887+0000 7f3930b01700 0 mon.smithi003@-1(???).mds e1 new map 2023-11-17T05:24:34.309 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.887+0000 7f3930b01700 0 mon.smithi003@-1(???).mds e1 print_map 2023-11-17T05:24:34.309 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: e1 2023-11-17T05:24:34.309 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: enable_multiple, ever_enabled_multiple: 1,1 2023-11-17T05:24:34.309 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-11-17T05:24:34.309 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: legacy client fscid: -1 2023-11-17T05:24:34.309 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: 2023-11-17T05:24:34.309 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: No filesystems configured 2023-11-17T05:24:34.310 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: 2023-11-17T05:24:34.310 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.888+0000 7f3930b01700 0 mon.smithi003@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-11-17T05:24:34.310 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.888+0000 7f3930b01700 0 mon.smithi003@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-17T05:24:34.310 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.888+0000 7f3930b01700 0 mon.smithi003@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-17T05:24:34.310 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.888+0000 7f3930b01700 0 mon.smithi003@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-17T05:24:34.310 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:24:31.822668+0000 mon.smithi003 (mon.0) 8 : audit [INF] from='client.? 172.21.15.3:0/2582038699' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-11-17T05:24:34.310 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:24:31.826263+0000 mon.smithi003 (mon.0) 9 : audit [INF] from='client.? 172.21.15.3:0/2582038699' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-11-17T05:24:34.310 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.888+0000 7f3930b01700 1 mon.smithi003@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-11-17T05:24:34.310 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.890+0000 7f3930b01700 20 expand_channel_meta expand map: {default=false} 2023-11-17T05:24:34.310 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.890+0000 7f3930b01700 20 expand_channel_meta from 'false' to 'false' 2023-11-17T05:24:34.310 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.890+0000 7f3930b01700 20 expand_channel_meta expanded map: {default=false} 2023-11-17T05:24:34.310 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.890+0000 7f3930b01700 20 expand_channel_meta expand map: {default=info} 2023-11-17T05:24:34.311 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.890+0000 7f3930b01700 20 expand_channel_meta from 'info' to 'info' 2023-11-17T05:24:34.311 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.890+0000 7f3930b01700 20 expand_channel_meta expanded map: {default=info} 2023-11-17T05:24:34.311 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.890+0000 7f3930b01700 20 expand_channel_meta expand map: {default=daemon} 2023-11-17T05:24:34.311 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.890+0000 7f3930b01700 20 expand_channel_meta from 'daemon' to 'daemon' 2023-11-17T05:24:34.311 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.890+0000 7f3930b01700 20 expand_channel_meta expanded map: {default=daemon} 2023-11-17T05:24:34.311 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.890+0000 7f3930b01700 20 expand_channel_meta expand map: {default=debug} 2023-11-17T05:24:34.311 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.890+0000 7f3930b01700 20 expand_channel_meta from 'debug' to 'debug' 2023-11-17T05:24:34.311 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:24:33.890+0000 7f3930b01700 20 expand_channel_meta expanded map: {default=debug} 2023-11-17T05:24:34.311 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:24:33.907312+0000 mon.smithi003 (mon.0) 1 : cluster [INF] mon.smithi003 is new leader, mons smithi003 in quorum (ranks 0) 2023-11-17T05:24:34.311 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:24:33.907420+0000 mon.smithi003 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi003=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0]} 2023-11-17T05:24:34.311 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:24:33.907475+0000 mon.smithi003 (mon.0) 3 : cluster [DBG] fsmap 2023-11-17T05:24:34.311 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:24:33.907524+0000 mon.smithi003 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-11-17T05:24:34.312 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:24:33.908002+0000 mon.smithi003 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2023-11-17T05:24:34.346 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.343+0000 7f431df0f700 1 -- 172.21.15.3:0/1118544803 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f43181068c0 con 0x7f431806fbd0 2023-11-17T05:24:34.350 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.348+0000 7f43157fa700 1 -- 172.21.15.3:0/1118544803 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f430c025d80 con 0x7f431806fbd0 2023-11-17T05:24:34.350 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.349+0000 7f431df0f700 1 -- 172.21.15.3:0/1118544803 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f431806fbd0 msgr2=0x7f431810c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:24:34.350 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.349+0000 7f431df0f700 1 --2- 172.21.15.3:0/1118544803 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f431806fbd0 0x7f431810c980 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f430c011820 tx=0x7f430c006ee0).stop 2023-11-17T05:24:34.350 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.349+0000 7f431df0f700 1 -- 172.21.15.3:0/1118544803 shutdown_connections 2023-11-17T05:24:34.350 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.349+0000 7f431df0f700 1 --2- 172.21.15.3:0/1118544803 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f431806fbd0 0x7f431810c980 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:24:34.350 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.349+0000 7f431df0f700 1 -- 172.21.15.3:0/1118544803 >> 172.21.15.3:0/1118544803 conn(0x7f43180f6130 msgr2=0x7f43180f7c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:24:34.351 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.349+0000 7f431df0f700 1 -- 172.21.15.3:0/1118544803 shutdown_connections 2023-11-17T05:24:34.351 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:34.349+0000 7f431df0f700 1 -- 172.21.15.3:0/1118544803 wait complete. 2023-11-17T05:24:34.767 INFO:teuthology.orchestra.run.smithi003.stderr:Wrote config to /etc/ceph/ceph.conf 2023-11-17T05:24:34.767 INFO:teuthology.orchestra.run.smithi003.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-11-17T05:24:34.767 INFO:teuthology.orchestra.run.smithi003.stderr:Creating mgr... 2023-11-17T05:24:34.768 INFO:teuthology.orchestra.run.smithi003.stderr:Verifying port 9283 ... 2023-11-17T05:24:34.771 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl daemon-reload 2023-11-17T05:24:34.883 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl stop ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7@mgr.smithi003.eddcuc 2023-11-17T05:24:34.911 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl reset-failed ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7@mgr.smithi003.eddcuc 2023-11-17T05:24:34.917 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: Failed to reset failed state of unit ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7@mgr.smithi003.eddcuc.service: Unit ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7@mgr.smithi003.eddcuc.service not loaded. 2023-11-17T05:24:34.917 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl enable ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7@mgr.smithi003.eddcuc 2023-11-17T05:24:34.924 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: Created symlink /etc/systemd/system/ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7.target.wants/ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7@mgr.smithi003.eddcuc.service → /etc/systemd/system/ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7@.service. 2023-11-17T05:24:35.020 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl start ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7@mgr.smithi003.eddcuc 2023-11-17T05:24:35.527 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl is-enabled firewalld.service 2023-11-17T05:24:35.538 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: disabled 2023-11-17T05:24:35.539 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl is-active firewalld.service 2023-11-17T05:24:35.548 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: inactive 2023-11-17T05:24:35.549 INFO:teuthology.orchestra.run.smithi003.stderr:firewalld.service is not enabled 2023-11-17T05:24:35.549 INFO:teuthology.orchestra.run.smithi003.stderr:Not possible to enable service . firewalld.service is not available 2023-11-17T05:24:35.549 INFO:teuthology.orchestra.run.smithi003.stderr:Not possible to open ports <[]>. firewalld.service is not available 2023-11-17T05:24:35.549 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl is-enabled firewalld.service 2023-11-17T05:24:35.558 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: disabled 2023-11-17T05:24:35.558 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl is-active firewalld.service 2023-11-17T05:24:35.567 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: inactive 2023-11-17T05:24:35.568 INFO:teuthology.orchestra.run.smithi003.stderr:firewalld.service is not enabled 2023-11-17T05:24:35.568 INFO:teuthology.orchestra.run.smithi003.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2023-11-17T05:24:35.568 INFO:teuthology.orchestra.run.smithi003.stderr:Waiting for mgr to start... 2023-11-17T05:24:35.569 INFO:teuthology.orchestra.run.smithi003.stderr:Waiting for mgr... 2023-11-17T05:24:35.569 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptcuja8tf:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpk7o1jie2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-11-17T05:24:35.683 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:35 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:24:34.348775+0000 mon.smithi003 (mon.0) 6 : audit [INF] from='client.? 172.21.15.3:0/1118544803' entity='client.admin' 2023-11-17T05:24:35.959 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:35.958+0000 7f48ab793700 1 Processor -- start 2023-11-17T05:24:35.960 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:35.958+0000 7f48ab793700 1 -- start start 2023-11-17T05:24:35.960 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:35.958+0000 7f48ab793700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48a406fc90 0x7f48a4100be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-17T05:24:35.960 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:35.958+0000 7f48ab793700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a40fc660 con 0x7f48a406fc90 2023-11-17T05:24:35.960 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:35.959+0000 7f48a952f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48a406fc90 0x7f48a4100be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:24:35.961 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:35.959+0000 7f48a952f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48a406fc90 0x7f48a4100be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60716/0 (socket says 172.21.15.3:60716) 2023-11-17T05:24:35.962 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:35.959+0000 7f48a952f700 1 -- 172.21.15.3:0/1721821589 learned_addr learned my addr 172.21.15.3:0/1721821589 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:24:35.962 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:35.959+0000 7f48a952f700 1 -- 172.21.15.3:0/1721821589 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a41010d0 con 0x7f48a406fc90 2023-11-17T05:24:35.962 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:35.959+0000 7f48a952f700 1 --2- 172.21.15.3:0/1721821589 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48a406fc90 0x7f48a4100be0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f48a000b5f0 tx=0x7f48a0006ee0).ready entity=mon.0 client_cookie=a7bae5699f6e063c server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:24:35.962 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:35.959+0000 7f4893fff700 1 -- 172.21.15.3:0/1721821589 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48a000d600 con 0x7f48a406fc90 2023-11-17T05:24:35.962 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:35.959+0000 7f4893fff700 1 -- 172.21.15.3:0/1721821589 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f48a000dbd0 con 0x7f48a406fc90 2023-11-17T05:24:35.962 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:35.960+0000 7f4893fff700 1 -- 172.21.15.3:0/1721821589 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48a0011c00 con 0x7f48a406fc90 2023-11-17T05:24:35.962 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:35.960+0000 7f48ab793700 1 -- 172.21.15.3:0/1721821589 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48a406fc90 msgr2=0x7f48a4100be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:24:35.963 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:35.960+0000 7f48ab793700 1 --2- 172.21.15.3:0/1721821589 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48a406fc90 0x7f48a4100be0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f48a000b5f0 tx=0x7f48a0006ee0).stop 2023-11-17T05:24:35.963 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:35.960+0000 7f48ab793700 1 -- 172.21.15.3:0/1721821589 shutdown_connections 2023-11-17T05:24:35.963 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:35.960+0000 7f48ab793700 1 --2- 172.21.15.3:0/1721821589 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48a406fc90 0x7f48a4100be0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:24:35.963 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:35.960+0000 7f48ab793700 1 -- 172.21.15.3:0/1721821589 >> 172.21.15.3:0/1721821589 conn(0x7f48a40f6280 msgr2=0x7f48a40f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:24:35.963 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:35.960+0000 7f48ab793700 1 -- 172.21.15.3:0/1721821589 shutdown_connections 2023-11-17T05:24:35.964 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:35.960+0000 7f48ab793700 1 -- 172.21.15.3:0/1721821589 wait complete. 2023-11-17T05:24:35.965 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:35.961+0000 7f48ab793700 1 Processor -- start 2023-11-17T05:24:35.965 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:35.961+0000 7f48ab793700 1 -- start start 2023-11-17T05:24:35.965 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:35.961+0000 7f48ab793700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48a406fc90 0x7f48a410cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:24:35.965 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:35.961+0000 7f48ab793700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a410d140 con 0x7f48a406fc90 2023-11-17T05:24:35.965 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:35.961+0000 7f48a952f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48a406fc90 0x7f48a410cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:24:35.965 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:35.961+0000 7f48a952f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48a406fc90 0x7f48a410cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60724/0 (socket says 172.21.15.3:60724) 2023-11-17T05:24:35.965 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:35.961+0000 7f48a952f700 1 -- 172.21.15.3:0/2592121300 learned_addr learned my addr 172.21.15.3:0/2592121300 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:24:35.965 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:35.962+0000 7f48a952f700 1 -- 172.21.15.3:0/2592121300 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a000b200 con 0x7f48a406fc90 2023-11-17T05:24:35.965 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:35.962+0000 7f48a952f700 1 --2- 172.21.15.3:0/2592121300 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48a406fc90 0x7f48a410cc50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f48a0000c00 tx=0x7f48a0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:24:35.965 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:35.962+0000 7f4891ffb700 1 -- 172.21.15.3:0/2592121300 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48a0011690 con 0x7f48a406fc90 2023-11-17T05:24:35.965 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:35.962+0000 7f48ab793700 1 -- 172.21.15.3:0/2592121300 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48a410d340 con 0x7f48a406fc90 2023-11-17T05:24:35.965 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:35.962+0000 7f4891ffb700 1 -- 172.21.15.3:0/2592121300 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f48a000d9f0 con 0x7f48a406fc90 2023-11-17T05:24:35.966 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:35.962+0000 7f4891ffb700 1 -- 172.21.15.3:0/2592121300 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48a001b710 con 0x7f48a406fc90 2023-11-17T05:24:35.966 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:35.962+0000 7f48ab793700 1 -- 172.21.15.3:0/2592121300 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48a410d750 con 0x7f48a406fc90 2023-11-17T05:24:35.966 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:35.963+0000 7f4891ffb700 1 -- 172.21.15.3:0/2592121300 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f48a000f070 con 0x7f48a406fc90 2023-11-17T05:24:35.966 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:35.963+0000 7f4891ffb700 1 -- 172.21.15.3:0/2592121300 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f48a0025a80 con 0x7f48a406fc90 2023-11-17T05:24:35.966 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:35.963+0000 7f48ab793700 1 -- 172.21.15.3:0/2592121300 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48a4045ad0 con 0x7f48a406fc90 2023-11-17T05:24:35.966 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:35.966+0000 7f4891ffb700 1 -- 172.21.15.3:0/2592121300 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f48a0016070 con 0x7f48a406fc90 2023-11-17T05:24:36.042 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:36.039+0000 7f48ab793700 1 -- 172.21.15.3:0/2592121300 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f48a4106a20 con 0x7f48a406fc90 2023-11-17T05:24:36.042 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:36.039+0000 7f4891ffb700 1 -- 172.21.15.3:0/2592121300 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f48a0025d10 con 0x7f48a406fc90 2023-11-17T05:24:36.043 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:36.043 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: { 2023-11-17T05:24:36.043 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "fsid": "851feb1e-8509-11ee-8dbe-212e2dc638e7", 2023-11-17T05:24:36.043 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "health": { 2023-11-17T05:24:36.043 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-17T05:24:36.043 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "checks": {}, 2023-11-17T05:24:36.043 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "mutes": [] 2023-11-17T05:24:36.043 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-17T05:24:36.043 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-17T05:24:36.043 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "quorum": [ 2023-11-17T05:24:36.043 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 0 2023-11-17T05:24:36.043 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: ], 2023-11-17T05:24:36.043 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-17T05:24:36.044 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "smithi003" 2023-11-17T05:24:36.044 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: ], 2023-11-17T05:24:36.044 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "quorum_age": 2, 2023-11-17T05:24:36.044 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "monmap": { 2023-11-17T05:24:36.044 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-17T05:24:36.044 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-17T05:24:36.044 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-17T05:24:36.044 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-17T05:24:36.048 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "osdmap": { 2023-11-17T05:24:36.048 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-17T05:24:36.048 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-17T05:24:36.048 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-17T05:24:36.048 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-17T05:24:36.048 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-17T05:24:36.048 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-17T05:24:36.048 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-17T05:24:36.048 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-17T05:24:36.048 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "pgmap": { 2023-11-17T05:24:36.048 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-17T05:24:36.049 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-17T05:24:36.049 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-17T05:24:36.049 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-17T05:24:36.049 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-17T05:24:36.049 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-17T05:24:36.049 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-17T05:24:36.049 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-17T05:24:36.049 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-17T05:24:36.049 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "fsmap": { 2023-11-17T05:24:36.049 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-17T05:24:36.049 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-17T05:24:36.050 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-17T05:24:36.050 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-17T05:24:36.050 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-17T05:24:36.050 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "available": false, 2023-11-17T05:24:36.050 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-17T05:24:36.050 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "modules": [ 2023-11-17T05:24:36.050 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "iostat", 2023-11-17T05:24:36.050 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "restful" 2023-11-17T05:24:36.050 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: ], 2023-11-17T05:24:36.050 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "services": {} 2023-11-17T05:24:36.050 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-17T05:24:36.050 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "servicemap": { 2023-11-17T05:24:36.050 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-17T05:24:36.051 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "modified": "2023-11-17T05:24:30.510925+0000", 2023-11-17T05:24:36.051 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "services": {} 2023-11-17T05:24:36.051 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-17T05:24:36.051 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-17T05:24:36.051 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: } 2023-11-17T05:24:36.051 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:36.040+0000 7f48ab793700 1 -- 172.21.15.3:0/2592121300 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48a406fc90 msgr2=0x7f48a410cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:24:36.051 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:36.040+0000 7f48ab793700 1 --2- 172.21.15.3:0/2592121300 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48a406fc90 0x7f48a410cc50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f48a0000c00 tx=0x7f48a0006ee0).stop 2023-11-17T05:24:36.051 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:36.040+0000 7f48ab793700 1 -- 172.21.15.3:0/2592121300 shutdown_connections 2023-11-17T05:24:36.052 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:36.040+0000 7f48ab793700 1 --2- 172.21.15.3:0/2592121300 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f48a406fc90 0x7f48a410cc50 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:24:36.052 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:36.040+0000 7f48ab793700 1 -- 172.21.15.3:0/2592121300 >> 172.21.15.3:0/2592121300 conn(0x7f48a40f6280 msgr2=0x7f48a40f7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:24:36.052 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:36.040+0000 7f48ab793700 1 -- 172.21.15.3:0/2592121300 shutdown_connections 2023-11-17T05:24:36.052 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:36.040+0000 7f48ab793700 1 -- 172.21.15.3:0/2592121300 wait complete. 2023-11-17T05:24:36.415 INFO:teuthology.orchestra.run.smithi003.stderr:mgr not available, waiting (1/15)... 2023-11-17T05:24:36.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:36 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:24:36.040078+0000 mon.smithi003 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.3:0/2592121300' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-17T05:24:41.460 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptcuja8tf:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpk7o1jie2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-11-17T05:24:41.460 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.815+0000 7f66c3b08700 1 Processor -- start 2023-11-17T05:24:41.460 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.815+0000 7f66c3b08700 1 -- start start 2023-11-17T05:24:41.460 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.815+0000 7f66c3b08700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66bc06fc90 0x7f66bc0fe7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-17T05:24:41.460 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.815+0000 7f66c3b08700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66bc0fed60 con 0x7f66bc06fc90 2023-11-17T05:24:41.460 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.816+0000 7f66c18a4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66bc06fc90 0x7f66bc0fe7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:24:41.460 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.816+0000 7f66c18a4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66bc06fc90 0x7f66bc0fe7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54520/0 (socket says 172.21.15.3:54520) 2023-11-17T05:24:41.460 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.816+0000 7f66c18a4700 1 -- 172.21.15.3:0/2069543322 learned_addr learned my addr 172.21.15.3:0/2069543322 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:24:41.460 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.816+0000 7f66c18a4700 1 -- 172.21.15.3:0/2069543322 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66bc0feea0 con 0x7f66bc06fc90 2023-11-17T05:24:41.460 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.816+0000 7f66c18a4700 1 --2- 172.21.15.3:0/2069543322 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66bc06fc90 0x7f66bc0fe7e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f66b800b5f0 tx=0x7f66b8006ee0).ready entity=mon.0 client_cookie=2c9fa44497353a5f server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:24:41.461 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.816+0000 7f66c08a2700 1 -- 172.21.15.3:0/2069543322 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66b800d600 con 0x7f66bc06fc90 2023-11-17T05:24:41.461 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.816+0000 7f66c08a2700 1 -- 172.21.15.3:0/2069543322 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f66b800dbd0 con 0x7f66bc06fc90 2023-11-17T05:24:41.461 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.816+0000 7f66c08a2700 1 -- 172.21.15.3:0/2069543322 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66b8011c00 con 0x7f66bc06fc90 2023-11-17T05:24:41.461 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.817+0000 7f66c3b08700 1 -- 172.21.15.3:0/2069543322 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66bc06fc90 msgr2=0x7f66bc0fe7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:24:41.461 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.817+0000 7f66c3b08700 1 --2- 172.21.15.3:0/2069543322 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66bc06fc90 0x7f66bc0fe7e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f66b800b5f0 tx=0x7f66b8006ee0).stop 2023-11-17T05:24:41.461 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.817+0000 7f66c3b08700 1 -- 172.21.15.3:0/2069543322 shutdown_connections 2023-11-17T05:24:41.461 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.817+0000 7f66c3b08700 1 --2- 172.21.15.3:0/2069543322 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66bc06fc90 0x7f66bc0fe7e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:24:41.461 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.817+0000 7f66c3b08700 1 -- 172.21.15.3:0/2069543322 >> 172.21.15.3:0/2069543322 conn(0x7f66bc0f6280 msgr2=0x7f66bc0f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:24:41.461 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.817+0000 7f66c3b08700 1 -- 172.21.15.3:0/2069543322 shutdown_connections 2023-11-17T05:24:41.461 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.817+0000 7f66c3b08700 1 -- 172.21.15.3:0/2069543322 wait complete. 2023-11-17T05:24:41.461 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.818+0000 7f66c3b08700 1 Processor -- start 2023-11-17T05:24:41.461 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.818+0000 7f66c3b08700 1 -- start start 2023-11-17T05:24:41.461 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.819+0000 7f66c3b08700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66bc06fc90 0x7f66bc10a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:24:41.461 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.819+0000 7f66c3b08700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66bc10ae90 con 0x7f66bc06fc90 2023-11-17T05:24:41.462 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.819+0000 7f66c18a4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66bc06fc90 0x7f66bc10a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:24:41.462 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.819+0000 7f66c18a4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66bc06fc90 0x7f66bc10a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54522/0 (socket says 172.21.15.3:54522) 2023-11-17T05:24:41.462 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.819+0000 7f66c18a4700 1 -- 172.21.15.3:0/2512856409 learned_addr learned my addr 172.21.15.3:0/2512856409 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:24:41.462 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.819+0000 7f66c18a4700 1 -- 172.21.15.3:0/2512856409 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66b800b200 con 0x7f66bc06fc90 2023-11-17T05:24:41.462 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.819+0000 7f66c18a4700 1 --2- 172.21.15.3:0/2512856409 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66bc06fc90 0x7f66bc10a9a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f66b800e040 tx=0x7f66b8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:24:41.462 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.819+0000 7f66aa7fc700 1 -- 172.21.15.3:0/2512856409 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66b801b410 con 0x7f66bc06fc90 2023-11-17T05:24:41.462 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.819+0000 7f66c3b08700 1 -- 172.21.15.3:0/2512856409 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66bc10b090 con 0x7f66bc06fc90 2023-11-17T05:24:41.462 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.819+0000 7f66aa7fc700 1 -- 172.21.15.3:0/2512856409 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f66b801b9e0 con 0x7f66bc06fc90 2023-11-17T05:24:41.462 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.819+0000 7f66aa7fc700 1 -- 172.21.15.3:0/2512856409 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66b801a770 con 0x7f66bc06fc90 2023-11-17T05:24:41.462 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.819+0000 7f66c3b08700 1 -- 172.21.15.3:0/2512856409 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66bc10b4a0 con 0x7f66bc06fc90 2023-11-17T05:24:41.462 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.820+0000 7f66aa7fc700 1 -- 172.21.15.3:0/2512856409 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f66b800f070 con 0x7f66bc06fc90 2023-11-17T05:24:41.462 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.820+0000 7f66aa7fc700 1 -- 172.21.15.3:0/2512856409 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f66b801ae60 con 0x7f66bc06fc90 2023-11-17T05:24:41.463 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.821+0000 7f66c3b08700 1 -- 172.21.15.3:0/2512856409 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66bc045ad0 con 0x7f66bc06fc90 2023-11-17T05:24:41.463 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.825+0000 7f66aa7fc700 1 -- 172.21.15.3:0/2512856409 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f66b8016070 con 0x7f66bc06fc90 2023-11-17T05:24:41.463 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.899+0000 7f66c3b08700 1 -- 172.21.15.3:0/2512856409 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f66bc0ffbd0 con 0x7f66bc06fc90 2023-11-17T05:24:41.463 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.900+0000 7f66aa7fc700 1 -- 172.21.15.3:0/2512856409 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f66b802e4e0 con 0x7f66bc06fc90 2023-11-17T05:24:41.463 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.463 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: { 2023-11-17T05:24:41.463 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "fsid": "851feb1e-8509-11ee-8dbe-212e2dc638e7", 2023-11-17T05:24:41.463 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "health": { 2023-11-17T05:24:41.463 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-17T05:24:41.463 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "checks": {}, 2023-11-17T05:24:41.463 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "mutes": [] 2023-11-17T05:24:41.463 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-17T05:24:41.463 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-17T05:24:41.463 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "quorum": [ 2023-11-17T05:24:41.464 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 0 2023-11-17T05:24:41.464 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: ], 2023-11-17T05:24:41.464 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-17T05:24:41.464 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "smithi003" 2023-11-17T05:24:41.464 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: ], 2023-11-17T05:24:41.464 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "quorum_age": 4, 2023-11-17T05:24:41.464 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "monmap": { 2023-11-17T05:24:41.464 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-17T05:24:41.464 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-17T05:24:41.464 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-17T05:24:41.464 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-17T05:24:41.464 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "osdmap": { 2023-11-17T05:24:41.464 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-17T05:24:41.465 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-17T05:24:41.465 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-17T05:24:41.465 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-17T05:24:41.465 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-17T05:24:41.465 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-17T05:24:41.465 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-17T05:24:41.465 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-17T05:24:41.465 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "pgmap": { 2023-11-17T05:24:41.465 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-17T05:24:41.465 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-17T05:24:41.465 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-17T05:24:41.465 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-17T05:24:41.465 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-17T05:24:41.465 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-17T05:24:41.466 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-17T05:24:41.466 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-17T05:24:41.466 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-17T05:24:41.466 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "fsmap": { 2023-11-17T05:24:41.466 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-17T05:24:41.466 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-17T05:24:41.466 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-17T05:24:41.466 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-17T05:24:41.466 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-17T05:24:41.466 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "available": false, 2023-11-17T05:24:41.466 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-17T05:24:41.466 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "modules": [ 2023-11-17T05:24:41.466 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "iostat", 2023-11-17T05:24:41.467 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "restful" 2023-11-17T05:24:41.467 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: ], 2023-11-17T05:24:41.467 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "services": {} 2023-11-17T05:24:41.467 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-17T05:24:41.467 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "servicemap": { 2023-11-17T05:24:41.467 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-17T05:24:41.467 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "modified": "2023-11-17T05:24:30.510925+0000", 2023-11-17T05:24:41.467 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "services": {} 2023-11-17T05:24:41.467 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-17T05:24:41.467 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-17T05:24:41.467 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: } 2023-11-17T05:24:41.467 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.901+0000 7f66c3b08700 1 -- 172.21.15.3:0/2512856409 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66bc06fc90 msgr2=0x7f66bc10a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:24:41.467 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.901+0000 7f66c3b08700 1 --2- 172.21.15.3:0/2512856409 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66bc06fc90 0x7f66bc10a9a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f66b800e040 tx=0x7f66b8006ee0).stop 2023-11-17T05:24:41.467 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.901+0000 7f66c3b08700 1 -- 172.21.15.3:0/2512856409 shutdown_connections 2023-11-17T05:24:41.468 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.901+0000 7f66c3b08700 1 --2- 172.21.15.3:0/2512856409 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f66bc06fc90 0x7f66bc10a9a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:24:41.468 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.901+0000 7f66c3b08700 1 -- 172.21.15.3:0/2512856409 >> 172.21.15.3:0/2512856409 conn(0x7f66bc0f6280 msgr2=0x7f66bc0f7e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:24:41.468 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.901+0000 7f66c3b08700 1 -- 172.21.15.3:0/2512856409 shutdown_connections 2023-11-17T05:24:41.468 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:38.901+0000 7f66c3b08700 1 -- 172.21.15.3:0/2512856409 wait complete. 2023-11-17T05:24:41.468 INFO:teuthology.orchestra.run.smithi003.stderr:mgr not available, waiting (2/15)... 2023-11-17T05:24:41.468 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:39 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:24:38.900408+0000 mon.smithi003 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.3:0/2512856409' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-17T05:24:41.468 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptcuja8tf:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpk7o1jie2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-11-17T05:24:41.521 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:41 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:24:41.156703+0000 mon.smithi003 (mon.0) 9 : cluster [INF] Activating manager daemon smithi003.eddcuc 2023-11-17T05:24:41.521 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:41 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:24:41.159626+0000 mon.smithi003 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi003.eddcuc(active, starting, since 0.00305164s) 2023-11-17T05:24:41.521 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:41 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:24:41.160697+0000 mon.smithi003 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.3:0/2463932015' entity='mgr.smithi003.eddcuc' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-17T05:24:41.521 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:41 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:24:41.160904+0000 mon.smithi003 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.3:0/2463932015' entity='mgr.smithi003.eddcuc' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-17T05:24:41.521 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:41 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:24:41.161096+0000 mon.smithi003 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.3:0/2463932015' entity='mgr.smithi003.eddcuc' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-17T05:24:41.521 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:41 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:24:41.161299+0000 mon.smithi003 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.3:0/2463932015' entity='mgr.smithi003.eddcuc' cmd=[{"prefix": "mon metadata", "id": "smithi003"}]: dispatch 2023-11-17T05:24:41.522 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:41 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:24:41.161576+0000 mon.smithi003 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.3:0/2463932015' entity='mgr.smithi003.eddcuc' cmd=[{"prefix": "mgr metadata", "who": "smithi003.eddcuc", "id": "smithi003.eddcuc"}]: dispatch 2023-11-17T05:24:41.522 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:41 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:24:41.175322+0000 mon.smithi003 (mon.0) 16 : cluster [INF] Manager daemon smithi003.eddcuc is now available 2023-11-17T05:24:41.522 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:41 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:24:41.184258+0000 mon.smithi003 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.3:0/2463932015' entity='mgr.smithi003.eddcuc' 2023-11-17T05:24:41.522 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:41 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:24:41.188792+0000 mon.smithi003 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.3:0/2463932015' entity='mgr.smithi003.eddcuc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.eddcuc/mirror_snapshot_schedule"}]: dispatch 2023-11-17T05:24:41.522 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:41 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:24:41.190132+0000 mon.smithi003 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.3:0/2463932015' entity='mgr.smithi003.eddcuc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.eddcuc/trash_purge_schedule"}]: dispatch 2023-11-17T05:24:41.522 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:41 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:24:41.195476+0000 mon.smithi003 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.3:0/2463932015' entity='mgr.smithi003.eddcuc' 2023-11-17T05:24:41.522 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:41 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:24:41.197749+0000 mon.smithi003 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.3:0/2463932015' entity='mgr.smithi003.eddcuc' 2023-11-17T05:24:41.625 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.624+0000 7f3338be3700 1 Processor -- start 2023-11-17T05:24:41.626 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.624+0000 7f3338be3700 1 -- start start 2023-11-17T05:24:41.626 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.625+0000 7f3338be3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f333406fc10 0x7f33340fe860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-17T05:24:41.626 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.625+0000 7f3338be3700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33340fed50 con 0x7f333406fc10 2023-11-17T05:24:41.626 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.625+0000 7f333259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f333406fc10 0x7f33340fe860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:24:41.626 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.625+0000 7f333259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f333406fc10 0x7f33340fe860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54584/0 (socket says 172.21.15.3:54584) 2023-11-17T05:24:41.628 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.625+0000 7f333259c700 1 -- 172.21.15.3:0/313843285 learned_addr learned my addr 172.21.15.3:0/313843285 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:24:41.628 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.625+0000 7f333259c700 1 -- 172.21.15.3:0/313843285 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33340fee90 con 0x7f333406fc10 2023-11-17T05:24:41.628 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.625+0000 7f333259c700 1 --2- 172.21.15.3:0/313843285 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f333406fc10 0x7f33340fe860 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f332400b5f0 tx=0x7f3324006ee0).ready entity=mon.0 client_cookie=aec4eb5149597a85 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:24:41.628 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.625+0000 7f3331d9b700 1 -- 172.21.15.3:0/313843285 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f332400d600 con 0x7f333406fc10 2023-11-17T05:24:41.628 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.625+0000 7f3331d9b700 1 -- 172.21.15.3:0/313843285 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f332400dbd0 con 0x7f333406fc10 2023-11-17T05:24:41.628 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.625+0000 7f3331d9b700 1 -- 172.21.15.3:0/313843285 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3324011c30 con 0x7f333406fc10 2023-11-17T05:24:41.628 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.626+0000 7f3338be3700 1 -- 172.21.15.3:0/313843285 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f333406fc10 msgr2=0x7f33340fe860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:24:41.629 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.626+0000 7f3338be3700 1 --2- 172.21.15.3:0/313843285 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f333406fc10 0x7f33340fe860 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f332400b5f0 tx=0x7f3324006ee0).stop 2023-11-17T05:24:41.629 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.626+0000 7f3338be3700 1 -- 172.21.15.3:0/313843285 shutdown_connections 2023-11-17T05:24:41.629 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.626+0000 7f3338be3700 1 --2- 172.21.15.3:0/313843285 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f333406fc10 0x7f33340fe860 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:24:41.629 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.626+0000 7f3338be3700 1 -- 172.21.15.3:0/313843285 >> 172.21.15.3:0/313843285 conn(0x7f33340f6280 msgr2=0x7f33340f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:24:41.629 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.626+0000 7f3338be3700 1 -- 172.21.15.3:0/313843285 shutdown_connections 2023-11-17T05:24:41.629 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.626+0000 7f3338be3700 1 -- 172.21.15.3:0/313843285 wait complete. 2023-11-17T05:24:41.629 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.627+0000 7f3338be3700 1 Processor -- start 2023-11-17T05:24:41.629 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.627+0000 7f3338be3700 1 -- start start 2023-11-17T05:24:41.629 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.627+0000 7f3338be3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f333406fc10 0x7f333410a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:24:41.629 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.627+0000 7f3338be3700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f333410aed0 con 0x7f333406fc10 2023-11-17T05:24:41.629 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.627+0000 7f333259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f333406fc10 0x7f333410a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:24:41.629 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.628+0000 7f333259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f333406fc10 0x7f333410a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54586/0 (socket says 172.21.15.3:54586) 2023-11-17T05:24:41.630 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.628+0000 7f333259c700 1 -- 172.21.15.3:0/2718658254 learned_addr learned my addr 172.21.15.3:0/2718658254 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:24:41.630 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.628+0000 7f333259c700 1 -- 172.21.15.3:0/2718658254 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f332400b200 con 0x7f333406fc10 2023-11-17T05:24:41.630 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.628+0000 7f333259c700 1 --2- 172.21.15.3:0/2718658254 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f333406fc10 0x7f333410a9e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f3324011550 tx=0x7f3324006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:24:41.630 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.628+0000 7f332b7fe700 1 -- 172.21.15.3:0/2718658254 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f332401b410 con 0x7f333406fc10 2023-11-17T05:24:41.630 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.628+0000 7f332b7fe700 1 -- 172.21.15.3:0/2718658254 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f332401b9e0 con 0x7f333406fc10 2023-11-17T05:24:41.630 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.628+0000 7f3338be3700 1 -- 172.21.15.3:0/2718658254 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f333410b0d0 con 0x7f333406fc10 2023-11-17T05:24:41.631 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.628+0000 7f332b7fe700 1 -- 172.21.15.3:0/2718658254 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f332401a7f0 con 0x7f333406fc10 2023-11-17T05:24:41.631 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.628+0000 7f3338be3700 1 -- 172.21.15.3:0/2718658254 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f333410b4e0 con 0x7f333406fc10 2023-11-17T05:24:41.631 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.629+0000 7f332b7fe700 1 -- 172.21.15.3:0/2718658254 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 2) v1 ==== 35959+0+0 (secure 0 0 0) 0x7f332400f070 con 0x7f333406fc10 2023-11-17T05:24:41.631 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.629+0000 7f332b7fe700 1 -- 172.21.15.3:0/2718658254 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f33240410a0 con 0x7f333406fc10 2023-11-17T05:24:41.631 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.629+0000 7f3338be3700 1 -- 172.21.15.3:0/2718658254 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3334059100 con 0x7f333406fc10 2023-11-17T05:24:41.632 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.631+0000 7f332b7fe700 1 -- 172.21.15.3:0/2718658254 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f3324022900 con 0x7f333406fc10 2023-11-17T05:24:41.717 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.704+0000 7f3338be3700 1 -- 172.21.15.3:0/2718658254 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f3334104830 con 0x7f333406fc10 2023-11-17T05:24:41.717 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.705+0000 7f332b7fe700 1 -- 172.21.15.3:0/2718658254 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f3324022310 con 0x7f333406fc10 2023-11-17T05:24:41.717 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.718 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: { 2023-11-17T05:24:41.718 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "fsid": "851feb1e-8509-11ee-8dbe-212e2dc638e7", 2023-11-17T05:24:41.718 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "health": { 2023-11-17T05:24:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-17T05:24:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "checks": {}, 2023-11-17T05:24:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "mutes": [] 2023-11-17T05:24:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-17T05:24:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-17T05:24:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "quorum": [ 2023-11-17T05:24:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 0 2023-11-17T05:24:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: ], 2023-11-17T05:24:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-17T05:24:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "smithi003" 2023-11-17T05:24:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: ], 2023-11-17T05:24:41.719 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "quorum_age": 7, 2023-11-17T05:24:41.720 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "monmap": { 2023-11-17T05:24:41.720 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-17T05:24:41.720 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-17T05:24:41.720 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-17T05:24:41.720 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-17T05:24:41.720 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "osdmap": { 2023-11-17T05:24:41.720 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-17T05:24:41.720 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-17T05:24:41.720 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-17T05:24:41.720 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-17T05:24:41.720 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-17T05:24:41.720 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-17T05:24:41.720 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-17T05:24:41.720 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-17T05:24:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "pgmap": { 2023-11-17T05:24:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-17T05:24:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-17T05:24:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-17T05:24:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-17T05:24:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-17T05:24:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-17T05:24:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-17T05:24:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-17T05:24:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-17T05:24:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "fsmap": { 2023-11-17T05:24:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-17T05:24:41.721 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-17T05:24:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-17T05:24:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-17T05:24:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-17T05:24:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "available": false, 2023-11-17T05:24:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-17T05:24:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "modules": [ 2023-11-17T05:24:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "iostat", 2023-11-17T05:24:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "restful" 2023-11-17T05:24:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: ], 2023-11-17T05:24:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "services": {} 2023-11-17T05:24:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-17T05:24:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "servicemap": { 2023-11-17T05:24:41.722 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-17T05:24:41.723 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "modified": "2023-11-17T05:24:30.510925+0000", 2023-11-17T05:24:41.723 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "services": {} 2023-11-17T05:24:41.723 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-17T05:24:41.723 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-17T05:24:41.723 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: } 2023-11-17T05:24:41.723 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.705+0000 7f3338be3700 1 -- 172.21.15.3:0/2718658254 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f333406fc10 msgr2=0x7f333410a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:24:41.723 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.705+0000 7f3338be3700 1 --2- 172.21.15.3:0/2718658254 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f333406fc10 0x7f333410a9e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f3324011550 tx=0x7f3324006ee0).stop 2023-11-17T05:24:41.723 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.705+0000 7f3338be3700 1 -- 172.21.15.3:0/2718658254 shutdown_connections 2023-11-17T05:24:41.723 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.705+0000 7f3338be3700 1 --2- 172.21.15.3:0/2718658254 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f333406fc10 0x7f333410a9e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:24:41.723 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.705+0000 7f3338be3700 1 -- 172.21.15.3:0/2718658254 >> 172.21.15.3:0/2718658254 conn(0x7f33340f6280 msgr2=0x7f33340f7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:24:41.723 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.705+0000 7f3338be3700 1 -- 172.21.15.3:0/2718658254 shutdown_connections 2023-11-17T05:24:41.723 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:41.705+0000 7f3338be3700 1 -- 172.21.15.3:0/2718658254 wait complete. 2023-11-17T05:24:42.123 INFO:teuthology.orchestra.run.smithi003.stderr:mgr not available, waiting (3/15)... 2023-11-17T05:24:42.520 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:42 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:24:41.705359+0000 mon.smithi003 (mon.0) 22 : audit [DBG] from='client.? 172.21.15.3:0/2718658254' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-17T05:24:42.520 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:42 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:24:42.163068+0000 mon.smithi003 (mon.0) 23 : cluster [DBG] mgrmap e3: smithi003.eddcuc(active, since 1.0065s) 2023-11-17T05:24:44.126 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptcuja8tf:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpk7o1jie2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-11-17T05:24:44.507 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.505+0000 7f9d74a6f700 1 Processor -- start 2023-11-17T05:24:44.507 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.506+0000 7f9d74a6f700 1 -- start start 2023-11-17T05:24:44.507 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.506+0000 7f9d74a6f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d700faae0 0x7f9d700faf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-17T05:24:44.507 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.506+0000 7f9d74a6f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d700fb480 con 0x7f9d700faae0 2023-11-17T05:24:44.507 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.506+0000 7f9d6e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d700faae0 0x7f9d700faf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:24:44.507 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.506+0000 7f9d6e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d700faae0 0x7f9d700faf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54598/0 (socket says 172.21.15.3:54598) 2023-11-17T05:24:44.508 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.506+0000 7f9d6e59c700 1 -- 172.21.15.3:0/3244527103 learned_addr learned my addr 172.21.15.3:0/3244527103 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:24:44.508 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.506+0000 7f9d6e59c700 1 -- 172.21.15.3:0/3244527103 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d700fb5c0 con 0x7f9d700faae0 2023-11-17T05:24:44.510 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.506+0000 7f9d6e59c700 1 --2- 172.21.15.3:0/3244527103 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d700faae0 0x7f9d700faf00 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f9d6000b5f0 tx=0x7f9d60006ee0).ready entity=mon.0 client_cookie=e9f01d71fe7d9b0d server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:24:44.510 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.507+0000 7f9d6dd9b700 1 -- 172.21.15.3:0/3244527103 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d6000d600 con 0x7f9d700faae0 2023-11-17T05:24:44.510 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.507+0000 7f9d6dd9b700 1 -- 172.21.15.3:0/3244527103 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9d6000dbd0 con 0x7f9d700faae0 2023-11-17T05:24:44.510 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.507+0000 7f9d6dd9b700 1 -- 172.21.15.3:0/3244527103 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d60011c00 con 0x7f9d700faae0 2023-11-17T05:24:44.510 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.507+0000 7f9d74a6f700 1 -- 172.21.15.3:0/3244527103 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d700faae0 msgr2=0x7f9d700faf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:24:44.510 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.507+0000 7f9d74a6f700 1 --2- 172.21.15.3:0/3244527103 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d700faae0 0x7f9d700faf00 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f9d6000b5f0 tx=0x7f9d60006ee0).stop 2023-11-17T05:24:44.510 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.507+0000 7f9d74a6f700 1 -- 172.21.15.3:0/3244527103 shutdown_connections 2023-11-17T05:24:44.511 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.507+0000 7f9d74a6f700 1 --2- 172.21.15.3:0/3244527103 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d700faae0 0x7f9d700faf00 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:24:44.511 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.507+0000 7f9d74a6f700 1 -- 172.21.15.3:0/3244527103 >> 172.21.15.3:0/3244527103 conn(0x7f9d700f6280 msgr2=0x7f9d700f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:24:44.511 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.507+0000 7f9d74a6f700 1 -- 172.21.15.3:0/3244527103 shutdown_connections 2023-11-17T05:24:44.511 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.507+0000 7f9d74a6f700 1 -- 172.21.15.3:0/3244527103 wait complete. 2023-11-17T05:24:44.511 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.508+0000 7f9d74a6f700 1 Processor -- start 2023-11-17T05:24:44.511 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.508+0000 7f9d74a6f700 1 -- start start 2023-11-17T05:24:44.511 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.508+0000 7f9d74a6f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d700faae0 0x7f9d70100c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:24:44.511 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.508+0000 7f9d74a6f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d700ff0d0 con 0x7f9d700faae0 2023-11-17T05:24:44.511 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.508+0000 7f9d6e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d700faae0 0x7f9d70100c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:24:44.511 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.508+0000 7f9d6e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d700faae0 0x7f9d70100c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54614/0 (socket says 172.21.15.3:54614) 2023-11-17T05:24:44.511 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.508+0000 7f9d6e59c700 1 -- 172.21.15.3:0/975726911 learned_addr learned my addr 172.21.15.3:0/975726911 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:24:44.511 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.509+0000 7f9d6e59c700 1 -- 172.21.15.3:0/975726911 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d6000b200 con 0x7f9d700faae0 2023-11-17T05:24:44.511 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.509+0000 7f9d6e59c700 1 --2- 172.21.15.3:0/975726911 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d700faae0 0x7f9d70100c50 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f9d60002f40 tx=0x7f9d60006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:24:44.512 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.509+0000 7f9d677fe700 1 -- 172.21.15.3:0/975726911 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d60011690 con 0x7f9d700faae0 2023-11-17T05:24:44.512 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.509+0000 7f9d677fe700 1 -- 172.21.15.3:0/975726911 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9d6001b880 con 0x7f9d700faae0 2023-11-17T05:24:44.512 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.509+0000 7f9d74a6f700 1 -- 172.21.15.3:0/975726911 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d700ff270 con 0x7f9d700faae0 2023-11-17T05:24:44.513 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.509+0000 7f9d677fe700 1 -- 172.21.15.3:0/975726911 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d6001a6b0 con 0x7f9d700faae0 2023-11-17T05:24:44.513 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.509+0000 7f9d74a6f700 1 -- 172.21.15.3:0/975726911 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d700ff680 con 0x7f9d700faae0 2023-11-17T05:24:44.513 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.510+0000 7f9d677fe700 1 -- 172.21.15.3:0/975726911 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7f9d6000f070 con 0x7f9d700faae0 2023-11-17T05:24:44.513 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.510+0000 7f9d677fe700 1 --2- 172.21.15.3:0/975726911 >> [v2:172.21.15.3:6800/1850097233,v1:172.21.15.3:6801/1850097233] conn(0x7f9d5002fd50 0x7f9d50032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:24:44.514 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.510+0000 7f9d677fe700 1 -- 172.21.15.3:0/975726911 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f9d60041040 con 0x7f9d700faae0 2023-11-17T05:24:44.514 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.510+0000 7f9d65d9b700 1 --2- 172.21.15.3:0/975726911 >> [v2:172.21.15.3:6800/1850097233,v1:172.21.15.3:6801/1850097233] conn(0x7f9d5002fd50 0x7f9d50032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:24:44.514 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.510+0000 7f9d74a6f700 1 -- 172.21.15.3:0/975726911 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d70059100 con 0x7f9d700faae0 2023-11-17T05:24:44.514 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.511+0000 7f9d65d9b700 1 --2- 172.21.15.3:0/975726911 >> [v2:172.21.15.3:6800/1850097233,v1:172.21.15.3:6801/1850097233] conn(0x7f9d5002fd50 0x7f9d50032210 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f9d580096a0 tx=0x7f9d58006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:24:44.518 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.515+0000 7f9d677fe700 1 -- 172.21.15.3:0/975726911 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9d60016070 con 0x7f9d700faae0 2023-11-17T05:24:44.520 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:44 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:24:43.214513+0000 mon.smithi003 (mon.0) 24 : cluster [DBG] mgrmap e4: smithi003.eddcuc(active, since 2s) 2023-11-17T05:24:44.741 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.740+0000 7f9d74a6f700 1 -- 172.21.15.3:0/975726911 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f9d70045ad0 con 0x7f9d700faae0 2023-11-17T05:24:44.741 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.741+0000 7f9d677fe700 1 -- 172.21.15.3:0/975726911 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f9d60019cd0 con 0x7f9d700faae0 2023-11-17T05:24:44.742 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.745 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: { 2023-11-17T05:24:44.745 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "fsid": "851feb1e-8509-11ee-8dbe-212e2dc638e7", 2023-11-17T05:24:44.745 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "health": { 2023-11-17T05:24:44.745 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-17T05:24:44.745 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "checks": {}, 2023-11-17T05:24:44.745 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "mutes": [] 2023-11-17T05:24:44.745 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-17T05:24:44.745 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-17T05:24:44.745 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "quorum": [ 2023-11-17T05:24:44.745 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 0 2023-11-17T05:24:44.746 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: ], 2023-11-17T05:24:44.746 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-17T05:24:44.746 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "smithi003" 2023-11-17T05:24:44.746 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: ], 2023-11-17T05:24:44.746 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "quorum_age": 10, 2023-11-17T05:24:44.746 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "monmap": { 2023-11-17T05:24:44.746 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-17T05:24:44.746 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-17T05:24:44.746 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-17T05:24:44.746 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-17T05:24:44.746 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "osdmap": { 2023-11-17T05:24:44.746 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-17T05:24:44.747 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-17T05:24:44.747 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-17T05:24:44.747 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-17T05:24:44.747 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-17T05:24:44.747 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-17T05:24:44.747 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-17T05:24:44.747 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-17T05:24:44.747 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "pgmap": { 2023-11-17T05:24:44.748 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-17T05:24:44.748 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-17T05:24:44.748 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-17T05:24:44.748 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-17T05:24:44.748 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-17T05:24:44.749 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-17T05:24:44.749 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-17T05:24:44.749 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-17T05:24:44.749 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-17T05:24:44.749 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "fsmap": { 2023-11-17T05:24:44.751 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-17T05:24:44.752 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-17T05:24:44.752 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-17T05:24:44.752 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-17T05:24:44.752 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-17T05:24:44.752 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "available": true, 2023-11-17T05:24:44.752 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-17T05:24:44.752 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "modules": [ 2023-11-17T05:24:44.752 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "iostat", 2023-11-17T05:24:44.752 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "restful" 2023-11-17T05:24:44.752 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: ], 2023-11-17T05:24:44.752 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "services": {} 2023-11-17T05:24:44.752 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-17T05:24:44.752 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "servicemap": { 2023-11-17T05:24:44.753 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-17T05:24:44.753 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "modified": "2023-11-17T05:24:30.510925+0000", 2023-11-17T05:24:44.753 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "services": {} 2023-11-17T05:24:44.753 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: }, 2023-11-17T05:24:44.753 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-17T05:24:44.753 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: } 2023-11-17T05:24:44.753 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.744+0000 7f9d74a6f700 1 -- 172.21.15.3:0/975726911 >> [v2:172.21.15.3:6800/1850097233,v1:172.21.15.3:6801/1850097233] conn(0x7f9d5002fd50 msgr2=0x7f9d50032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:24:44.753 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.744+0000 7f9d74a6f700 1 --2- 172.21.15.3:0/975726911 >> [v2:172.21.15.3:6800/1850097233,v1:172.21.15.3:6801/1850097233] conn(0x7f9d5002fd50 0x7f9d50032210 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f9d580096a0 tx=0x7f9d58006b40).stop 2023-11-17T05:24:44.753 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.744+0000 7f9d74a6f700 1 -- 172.21.15.3:0/975726911 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d700faae0 msgr2=0x7f9d70100c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:24:44.753 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.744+0000 7f9d74a6f700 1 --2- 172.21.15.3:0/975726911 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d700faae0 0x7f9d70100c50 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f9d60002f40 tx=0x7f9d60006ee0).stop 2023-11-17T05:24:44.753 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.744+0000 7f9d74a6f700 1 -- 172.21.15.3:0/975726911 shutdown_connections 2023-11-17T05:24:44.753 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.744+0000 7f9d74a6f700 1 --2- 172.21.15.3:0/975726911 >> [v2:172.21.15.3:6800/1850097233,v1:172.21.15.3:6801/1850097233] conn(0x7f9d5002fd50 0x7f9d50032210 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:24:44.753 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.744+0000 7f9d74a6f700 1 --2- 172.21.15.3:0/975726911 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d700faae0 0x7f9d70100c50 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:24:44.754 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.744+0000 7f9d74a6f700 1 -- 172.21.15.3:0/975726911 >> 172.21.15.3:0/975726911 conn(0x7f9d700f6280 msgr2=0x7f9d700f6ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:24:44.754 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.744+0000 7f9d74a6f700 1 -- 172.21.15.3:0/975726911 shutdown_connections 2023-11-17T05:24:44.754 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:44.744+0000 7f9d74a6f700 1 -- 172.21.15.3:0/975726911 wait complete. 2023-11-17T05:24:45.107 INFO:teuthology.orchestra.run.smithi003.stderr:mgr is available 2023-11-17T05:24:45.107 INFO:teuthology.orchestra.run.smithi003.stderr:Enabling cephadm module... 2023-11-17T05:24:45.107 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptcuja8tf:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpk7o1jie2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2023-11-17T05:24:45.506 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:45.504+0000 7fce7fc7a700 1 Processor -- start 2023-11-17T05:24:45.506 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:45.505+0000 7fce7fc7a700 1 -- start start 2023-11-17T05:24:45.506 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:45.505+0000 7fce7fc7a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fce7806fcd0 0x7fce780fe7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-17T05:24:45.506 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:45.505+0000 7fce7fc7a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce780fed20 con 0x7fce7806fcd0 2023-11-17T05:24:45.506 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:45.505+0000 7fce7da16700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fce7806fcd0 0x7fce780fe7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:24:45.506 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:45.505+0000 7fce7da16700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fce7806fcd0 0x7fce780fe7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54620/0 (socket says 172.21.15.3:54620) 2023-11-17T05:24:45.507 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:45.505+0000 7fce7da16700 1 -- 172.21.15.3:0/1850496503 learned_addr learned my addr 172.21.15.3:0/1850496503 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:24:45.509 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:45.505+0000 7fce7da16700 1 -- 172.21.15.3:0/1850496503 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce780fee60 con 0x7fce7806fcd0 2023-11-17T05:24:45.510 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:45.506+0000 7fce7da16700 1 --2- 172.21.15.3:0/1850496503 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fce7806fcd0 0x7fce780fe7a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fce7400b5f0 tx=0x7fce74006ee0).ready entity=mon.0 client_cookie=254bb21b84063801 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:24:45.510 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:45.506+0000 7fce7ca14700 1 -- 172.21.15.3:0/1850496503 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce7400d600 con 0x7fce7806fcd0 2023-11-17T05:24:45.511 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:45.506+0000 7fce7ca14700 1 -- 172.21.15.3:0/1850496503 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fce7400dbd0 con 0x7fce7806fcd0 2023-11-17T05:24:45.511 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:45.506+0000 7fce7ca14700 1 -- 172.21.15.3:0/1850496503 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce74011c00 con 0x7fce7806fcd0 2023-11-17T05:24:45.511 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:45.506+0000 7fce7fc7a700 1 -- 172.21.15.3:0/1850496503 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fce7806fcd0 msgr2=0x7fce780fe7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:24:45.511 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:45.506+0000 7fce7fc7a700 1 --2- 172.21.15.3:0/1850496503 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fce7806fcd0 0x7fce780fe7a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fce7400b5f0 tx=0x7fce74006ee0).stop 2023-11-17T05:24:45.511 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:45.507+0000 7fce7fc7a700 1 -- 172.21.15.3:0/1850496503 shutdown_connections 2023-11-17T05:24:45.511 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:45.507+0000 7fce7fc7a700 1 --2- 172.21.15.3:0/1850496503 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fce7806fcd0 0x7fce780fe7a0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:24:45.511 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:45.507+0000 7fce7fc7a700 1 -- 172.21.15.3:0/1850496503 >> 172.21.15.3:0/1850496503 conn(0x7fce780f6280 msgr2=0x7fce780f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:24:45.511 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:45.507+0000 7fce7fc7a700 1 -- 172.21.15.3:0/1850496503 shutdown_connections 2023-11-17T05:24:45.512 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:45.507+0000 7fce7fc7a700 1 -- 172.21.15.3:0/1850496503 wait complete. 2023-11-17T05:24:45.512 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:45.508+0000 7fce7fc7a700 1 Processor -- start 2023-11-17T05:24:45.512 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:45.508+0000 7fce7fc7a700 1 -- start start 2023-11-17T05:24:45.512 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:45.508+0000 7fce7fc7a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fce7806fcd0 0x7fce780fc220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:24:45.513 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:45.508+0000 7fce7fc7a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce780fa6a0 con 0x7fce7806fcd0 2023-11-17T05:24:45.513 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:45.508+0000 7fce7da16700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fce7806fcd0 0x7fce780fc220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:24:45.513 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:45.508+0000 7fce7da16700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fce7806fcd0 0x7fce780fc220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54628/0 (socket says 172.21.15.3:54628) 2023-11-17T05:24:45.513 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:45.508+0000 7fce7da16700 1 -- 172.21.15.3:0/3895629937 learned_addr learned my addr 172.21.15.3:0/3895629937 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:24:45.513 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:45.508+0000 7fce7da16700 1 -- 172.21.15.3:0/3895629937 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce7400b200 con 0x7fce7806fcd0 2023-11-17T05:24:45.513 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:45.508+0000 7fce7da16700 1 --2- 172.21.15.3:0/3895629937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fce7806fcd0 0x7fce780fc220 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fce7400e040 tx=0x7fce74006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:24:45.513 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:45.509+0000 7fce667fc700 1 -- 172.21.15.3:0/3895629937 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce7401b410 con 0x7fce7806fcd0 2023-11-17T05:24:45.513 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:45.509+0000 7fce667fc700 1 -- 172.21.15.3:0/3895629937 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fce7401b9e0 con 0x7fce7806fcd0 2023-11-17T05:24:45.514 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:45.509+0000 7fce7fc7a700 1 -- 172.21.15.3:0/3895629937 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce780fa840 con 0x7fce7806fcd0 2023-11-17T05:24:45.514 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:45.509+0000 7fce667fc700 1 -- 172.21.15.3:0/3895629937 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce7401a770 con 0x7fce7806fcd0 2023-11-17T05:24:45.514 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:45.509+0000 7fce7fc7a700 1 -- 172.21.15.3:0/3895629937 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce780fac50 con 0x7fce7806fcd0 2023-11-17T05:24:45.515 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:45.510+0000 7fce667fc700 1 -- 172.21.15.3:0/3895629937 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7fce7400f070 con 0x7fce7806fcd0 2023-11-17T05:24:45.515 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:45.510+0000 7fce667fc700 1 --2- 172.21.15.3:0/3895629937 >> [v2:172.21.15.3:6800/1850097233,v1:172.21.15.3:6801/1850097233] conn(0x7fce6002fd50 0x7fce60032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:24:45.515 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:45.510+0000 7fce667fc700 1 -- 172.21.15.3:0/3895629937 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fce74040ef0 con 0x7fce7806fcd0 2023-11-17T05:24:45.515 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:45.510+0000 7fce7d215700 1 --2- 172.21.15.3:0/3895629937 >> [v2:172.21.15.3:6800/1850097233,v1:172.21.15.3:6801/1850097233] conn(0x7fce6002fd50 0x7fce60032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:24:45.515 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:45.510+0000 7fce7fc7a700 1 -- 172.21.15.3:0/3895629937 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce78059100 con 0x7fce7806fcd0 2023-11-17T05:24:45.515 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:45.511+0000 7fce7d215700 1 --2- 172.21.15.3:0/3895629937 >> [v2:172.21.15.3:6800/1850097233,v1:172.21.15.3:6801/1850097233] conn(0x7fce6002fd50 0x7fce60032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fce680096a0 tx=0x7fce68006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:24:45.515 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:45.515+0000 7fce667fc700 1 -- 172.21.15.3:0/3895629937 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fce74016070 con 0x7fce7806fcd0 2023-11-17T05:24:45.520 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:45 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:24:44.741709+0000 mon.smithi003 (mon.0) 25 : audit [DBG] from='client.? 172.21.15.3:0/975726911' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-17T05:24:45.729 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:45.728+0000 7fce7fc7a700 1 -- 172.21.15.3:0/3895629937 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fce78104820 con 0x7fce7806fcd0 2023-11-17T05:24:46.218 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:46.218+0000 7fce667fc700 1 -- 172.21.15.3:0/3895629937 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fce7401bb50 con 0x7fce7806fcd0 2023-11-17T05:24:46.219 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:46.219+0000 7fce667fc700 1 -- 172.21.15.3:0/3895629937 <== mon.0 v2:172.21.15.3:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7fce74023900 con 0x7fce7806fcd0 2023-11-17T05:24:46.223 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:46.221+0000 7fce7fc7a700 1 -- 172.21.15.3:0/3895629937 >> [v2:172.21.15.3:6800/1850097233,v1:172.21.15.3:6801/1850097233] conn(0x7fce6002fd50 msgr2=0x7fce60032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:24:46.223 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:46.221+0000 7fce7fc7a700 1 --2- 172.21.15.3:0/3895629937 >> [v2:172.21.15.3:6800/1850097233,v1:172.21.15.3:6801/1850097233] conn(0x7fce6002fd50 0x7fce60032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fce680096a0 tx=0x7fce68006b40).stop 2023-11-17T05:24:46.223 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:46.221+0000 7fce7fc7a700 1 -- 172.21.15.3:0/3895629937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fce7806fcd0 msgr2=0x7fce780fc220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:24:46.223 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:46.221+0000 7fce7fc7a700 1 --2- 172.21.15.3:0/3895629937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fce7806fcd0 0x7fce780fc220 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fce7400e040 tx=0x7fce74006ee0).stop 2023-11-17T05:24:46.223 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:46.221+0000 7fce7fc7a700 1 -- 172.21.15.3:0/3895629937 shutdown_connections 2023-11-17T05:24:46.223 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:46.221+0000 7fce7fc7a700 1 --2- 172.21.15.3:0/3895629937 >> [v2:172.21.15.3:6800/1850097233,v1:172.21.15.3:6801/1850097233] conn(0x7fce6002fd50 0x7fce60032210 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:24:46.223 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:46.221+0000 7fce7fc7a700 1 --2- 172.21.15.3:0/3895629937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fce7806fcd0 0x7fce780fc220 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:24:46.223 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:46.221+0000 7fce7fc7a700 1 -- 172.21.15.3:0/3895629937 >> 172.21.15.3:0/3895629937 conn(0x7fce780f6280 msgr2=0x7fce780f7e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:24:46.223 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:46.221+0000 7fce7fc7a700 1 -- 172.21.15.3:0/3895629937 shutdown_connections 2023-11-17T05:24:46.223 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:46.221+0000 7fce7fc7a700 1 -- 172.21.15.3:0/3895629937 wait complete. 2023-11-17T05:24:46.520 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:46 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:24:45.729374+0000 mon.smithi003 (mon.0) 26 : audit [INF] from='client.? 172.21.15.3:0/3895629937' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-11-17T05:24:47.520 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:47 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:24:46.219567+0000 mon.smithi003 (mon.0) 27 : audit [INF] from='client.? 172.21.15.3:0/3895629937' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-11-17T05:24:47.520 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:47 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:24:46.219656+0000 mon.smithi003 (mon.0) 28 : cluster [DBG] mgrmap e5: smithi003.eddcuc(active, since 5s) 2023-11-17T05:24:47.739 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptcuja8tf:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpk7o1jie2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2023-11-17T05:24:48.146 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.139+0000 7faff8501700 1 Processor -- start 2023-11-17T05:24:48.146 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.139+0000 7faff8501700 1 -- start start 2023-11-17T05:24:48.146 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.139+0000 7faff8501700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faff006fc50 0x7faff0100c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-17T05:24:48.146 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.139+0000 7faff8501700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff00fc6a0 con 0x7faff006fc50 2023-11-17T05:24:48.146 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.140+0000 7faff629d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faff006fc50 0x7faff0100c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:24:48.146 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.140+0000 7faff629d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faff006fc50 0x7faff0100c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54636/0 (socket says 172.21.15.3:54636) 2023-11-17T05:24:48.146 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.140+0000 7faff629d700 1 -- 172.21.15.3:0/3902800551 learned_addr learned my addr 172.21.15.3:0/3902800551 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:24:48.147 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.140+0000 7faff629d700 1 -- 172.21.15.3:0/3902800551 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff0101110 con 0x7faff006fc50 2023-11-17T05:24:48.147 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.140+0000 7faff629d700 1 --2- 172.21.15.3:0/3902800551 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faff006fc50 0x7faff0100c20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fafec00b5f0 tx=0x7fafec006ee0).ready entity=mon.0 client_cookie=feed8f38c0f9691b server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:24:48.147 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.140+0000 7faff529b700 1 -- 172.21.15.3:0/3902800551 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafec00d600 con 0x7faff006fc50 2023-11-17T05:24:48.147 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.140+0000 7faff529b700 1 -- 172.21.15.3:0/3902800551 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fafec00dbd0 con 0x7faff006fc50 2023-11-17T05:24:48.147 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.140+0000 7faff529b700 1 -- 172.21.15.3:0/3902800551 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafec011c00 con 0x7faff006fc50 2023-11-17T05:24:48.147 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.141+0000 7faff8501700 1 -- 172.21.15.3:0/3902800551 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faff006fc50 msgr2=0x7faff0100c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:24:48.147 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.141+0000 7faff8501700 1 --2- 172.21.15.3:0/3902800551 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faff006fc50 0x7faff0100c20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fafec00b5f0 tx=0x7fafec006ee0).stop 2023-11-17T05:24:48.147 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.141+0000 7faff8501700 1 -- 172.21.15.3:0/3902800551 shutdown_connections 2023-11-17T05:24:48.147 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.141+0000 7faff8501700 1 --2- 172.21.15.3:0/3902800551 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faff006fc50 0x7faff0100c20 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:24:48.147 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.141+0000 7faff8501700 1 -- 172.21.15.3:0/3902800551 >> 172.21.15.3:0/3902800551 conn(0x7faff00f6280 msgr2=0x7faff00f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:24:48.147 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.141+0000 7faff8501700 1 -- 172.21.15.3:0/3902800551 shutdown_connections 2023-11-17T05:24:48.147 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.141+0000 7faff8501700 1 -- 172.21.15.3:0/3902800551 wait complete. 2023-11-17T05:24:48.147 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.142+0000 7faff8501700 1 Processor -- start 2023-11-17T05:24:48.148 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.142+0000 7faff8501700 1 -- start start 2023-11-17T05:24:48.148 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.142+0000 7faff8501700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faff006fc50 0x7faff010ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:24:48.148 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.142+0000 7faff8501700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff010d1b0 con 0x7faff006fc50 2023-11-17T05:24:48.148 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.142+0000 7faff629d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faff006fc50 0x7faff010ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:24:48.148 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.142+0000 7faff629d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faff006fc50 0x7faff010ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54644/0 (socket says 172.21.15.3:54644) 2023-11-17T05:24:48.148 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.142+0000 7faff629d700 1 -- 172.21.15.3:0/1003365302 learned_addr learned my addr 172.21.15.3:0/1003365302 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:24:48.148 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.142+0000 7faff629d700 1 -- 172.21.15.3:0/1003365302 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafec00b200 con 0x7faff006fc50 2023-11-17T05:24:48.148 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.143+0000 7faff629d700 1 --2- 172.21.15.3:0/1003365302 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faff006fc50 0x7faff010ccc0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fafec006c70 tx=0x7fafec006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:24:48.148 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.143+0000 7fafdeffd700 1 -- 172.21.15.3:0/1003365302 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafec011690 con 0x7faff006fc50 2023-11-17T05:24:48.148 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.143+0000 7fafdeffd700 1 -- 172.21.15.3:0/1003365302 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fafec01b880 con 0x7faff006fc50 2023-11-17T05:24:48.148 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.143+0000 7fafdeffd700 1 -- 172.21.15.3:0/1003365302 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafec01a650 con 0x7faff006fc50 2023-11-17T05:24:48.148 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.143+0000 7faff8501700 1 -- 172.21.15.3:0/1003365302 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faff010d3b0 con 0x7faff006fc50 2023-11-17T05:24:48.148 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.143+0000 7faff8501700 1 -- 172.21.15.3:0/1003365302 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faff010d7c0 con 0x7faff006fc50 2023-11-17T05:24:48.148 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.144+0000 7fafdeffd700 1 -- 172.21.15.3:0/1003365302 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fafec00f070 con 0x7faff006fc50 2023-11-17T05:24:48.149 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.144+0000 7fafdcff9700 1 -- 172.21.15.3:0/1003365302 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafd4004fa0 con 0x7faff006fc50 2023-11-17T05:24:48.149 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.144+0000 7fafdeffd700 1 --2- 172.21.15.3:0/1003365302 >> [v2:172.21.15.3:6800/1850097233,v1:172.21.15.3:6801/1850097233] conn(0x7fafd802fd50 0x7fafd8032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:24:48.149 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.144+0000 7fafdeffd700 1 -- 172.21.15.3:0/1003365302 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fafec041460 con 0x7faff006fc50 2023-11-17T05:24:48.149 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.144+0000 7faff5a9c700 1 -- 172.21.15.3:0/1003365302 >> [v2:172.21.15.3:6800/1850097233,v1:172.21.15.3:6801/1850097233] conn(0x7fafd802fd50 msgr2=0x7fafd8032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/1850097233 2023-11-17T05:24:48.149 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.144+0000 7faff5a9c700 1 --2- 172.21.15.3:0/1003365302 >> [v2:172.21.15.3:6800/1850097233,v1:172.21.15.3:6801/1850097233] conn(0x7fafd802fd50 0x7fafd8032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-17T05:24:48.149 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.149+0000 7fafdeffd700 1 -- 172.21.15.3:0/1003365302 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fafec01bd60 con 0x7faff006fc50 2023-11-17T05:24:48.345 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.345+0000 7faff5a9c700 1 -- 172.21.15.3:0/1003365302 >> [v2:172.21.15.3:6800/1850097233,v1:172.21.15.3:6801/1850097233] conn(0x7fafd802fd50 msgr2=0x7fafd8032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/1850097233 2023-11-17T05:24:48.346 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.345+0000 7faff5a9c700 1 --2- 172.21.15.3:0/1003365302 >> [v2:172.21.15.3:6800/1850097233,v1:172.21.15.3:6801/1850097233] conn(0x7fafd802fd50 0x7fafd8032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-17T05:24:48.359 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.358+0000 7fafdcff9700 1 -- 172.21.15.3:0/1003365302 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fafd4005e80 con 0x7faff006fc50 2023-11-17T05:24:48.359 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.358+0000 7fafdeffd700 1 -- 172.21.15.3:0/1003365302 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7fafec016020 con 0x7faff006fc50 2023-11-17T05:24:48.359 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: { 2023-11-17T05:24:48.359 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "epoch": 5, 2023-11-17T05:24:48.359 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "available": true, 2023-11-17T05:24:48.359 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "active_name": "smithi003.eddcuc", 2023-11-17T05:24:48.359 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_standby": 0 2023-11-17T05:24:48.359 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: } 2023-11-17T05:24:48.363 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.360+0000 7fafdcff9700 1 -- 172.21.15.3:0/1003365302 >> [v2:172.21.15.3:6800/1850097233,v1:172.21.15.3:6801/1850097233] conn(0x7fafd802fd50 msgr2=0x7fafd8032210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-17T05:24:48.363 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.360+0000 7fafdcff9700 1 --2- 172.21.15.3:0/1003365302 >> [v2:172.21.15.3:6800/1850097233,v1:172.21.15.3:6801/1850097233] conn(0x7fafd802fd50 0x7fafd8032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-17T05:24:48.363 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.360+0000 7fafdcff9700 1 -- 172.21.15.3:0/1003365302 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faff006fc50 msgr2=0x7faff010ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:24:48.363 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.360+0000 7fafdcff9700 1 --2- 172.21.15.3:0/1003365302 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faff006fc50 0x7faff010ccc0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fafec006c70 tx=0x7fafec006ee0).stop 2023-11-17T05:24:48.363 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.360+0000 7fafdcff9700 1 -- 172.21.15.3:0/1003365302 shutdown_connections 2023-11-17T05:24:48.363 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.360+0000 7fafdcff9700 1 --2- 172.21.15.3:0/1003365302 >> [v2:172.21.15.3:6800/1850097233,v1:172.21.15.3:6801/1850097233] conn(0x7fafd802fd50 0x7fafd8032210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-17T05:24:48.363 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.360+0000 7fafdcff9700 1 --2- 172.21.15.3:0/1003365302 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faff006fc50 0x7faff010ccc0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:24:48.363 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.360+0000 7fafdcff9700 1 -- 172.21.15.3:0/1003365302 >> 172.21.15.3:0/1003365302 conn(0x7faff00f6280 msgr2=0x7faff00f7e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:24:48.363 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.361+0000 7fafdcff9700 1 -- 172.21.15.3:0/1003365302 shutdown_connections 2023-11-17T05:24:48.363 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:48.361+0000 7fafdcff9700 1 -- 172.21.15.3:0/1003365302 wait complete. 2023-11-17T05:24:48.771 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:48 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:24:48.358873+0000 mon.smithi003 (mon.0) 29 : audit [DBG] from='client.? 172.21.15.3:0/1003365302' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-11-17T05:24:48.772 INFO:teuthology.orchestra.run.smithi003.stderr:Waiting for the mgr to restart... 2023-11-17T05:24:48.772 INFO:teuthology.orchestra.run.smithi003.stderr:Waiting for mgr epoch 5... 2023-11-17T05:24:48.772 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptcuja8tf:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpk7o1jie2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2023-11-17T05:24:49.157 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:49.152+0000 7f599bd69700 1 Processor -- start 2023-11-17T05:24:49.157 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:49.152+0000 7f599bd69700 1 -- start start 2023-11-17T05:24:49.157 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:49.152+0000 7f599bd69700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f599406fc90 0x7f59940fe7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-17T05:24:49.157 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:49.152+0000 7f599bd69700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59940fed60 con 0x7f599406fc90 2023-11-17T05:24:49.157 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:49.152+0000 7f5999b05700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f599406fc90 0x7f59940fe7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:24:49.157 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:49.152+0000 7f5999b05700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f599406fc90 0x7f59940fe7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36132/0 (socket says 172.21.15.3:36132) 2023-11-17T05:24:49.157 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:49.152+0000 7f5999b05700 1 -- 172.21.15.3:0/1040918574 learned_addr learned my addr 172.21.15.3:0/1040918574 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:24:49.157 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:49.153+0000 7f5999b05700 1 -- 172.21.15.3:0/1040918574 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59940feea0 con 0x7f599406fc90 2023-11-17T05:24:49.157 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:49.153+0000 7f5999b05700 1 --2- 172.21.15.3:0/1040918574 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f599406fc90 0x7f59940fe7e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f599000b5f0 tx=0x7f5990006ee0).ready entity=mon.0 client_cookie=962c363308862953 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:24:49.157 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:49.153+0000 7f5998b03700 1 -- 172.21.15.3:0/1040918574 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f599000d600 con 0x7f599406fc90 2023-11-17T05:24:49.157 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:49.153+0000 7f5998b03700 1 -- 172.21.15.3:0/1040918574 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f599000dbd0 con 0x7f599406fc90 2023-11-17T05:24:49.158 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:49.153+0000 7f5998b03700 1 -- 172.21.15.3:0/1040918574 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5990011c00 con 0x7f599406fc90 2023-11-17T05:24:49.158 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:49.154+0000 7f599bd69700 1 -- 172.21.15.3:0/1040918574 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f599406fc90 msgr2=0x7f59940fe7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:24:49.158 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:49.154+0000 7f599bd69700 1 --2- 172.21.15.3:0/1040918574 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f599406fc90 0x7f59940fe7e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f599000b5f0 tx=0x7f5990006ee0).stop 2023-11-17T05:24:49.158 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:49.154+0000 7f599bd69700 1 -- 172.21.15.3:0/1040918574 shutdown_connections 2023-11-17T05:24:49.158 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:49.154+0000 7f599bd69700 1 --2- 172.21.15.3:0/1040918574 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f599406fc90 0x7f59940fe7e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:24:49.158 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:49.154+0000 7f599bd69700 1 -- 172.21.15.3:0/1040918574 >> 172.21.15.3:0/1040918574 conn(0x7f59940f6280 msgr2=0x7f59940f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:24:49.158 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:49.154+0000 7f599bd69700 1 -- 172.21.15.3:0/1040918574 shutdown_connections 2023-11-17T05:24:49.158 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:49.154+0000 7f599bd69700 1 -- 172.21.15.3:0/1040918574 wait complete. 2023-11-17T05:24:49.161 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:49.155+0000 7f599bd69700 1 Processor -- start 2023-11-17T05:24:49.161 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:49.155+0000 7f599bd69700 1 -- start start 2023-11-17T05:24:49.161 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:49.155+0000 7f599bd69700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f599406fc90 0x7f59940fc260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:24:49.161 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:49.155+0000 7f599bd69700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59940fa6e0 con 0x7f599406fc90 2023-11-17T05:24:49.162 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:49.155+0000 7f5999b05700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f599406fc90 0x7f59940fc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:24:49.162 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:49.155+0000 7f5999b05700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f599406fc90 0x7f59940fc260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36134/0 (socket says 172.21.15.3:36134) 2023-11-17T05:24:49.162 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:49.155+0000 7f5999b05700 1 -- 172.21.15.3:0/3311618392 learned_addr learned my addr 172.21.15.3:0/3311618392 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:24:49.162 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:49.156+0000 7f5999b05700 1 -- 172.21.15.3:0/3311618392 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f599000b200 con 0x7f599406fc90 2023-11-17T05:24:49.162 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:49.156+0000 7f5999b05700 1 --2- 172.21.15.3:0/3311618392 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f599406fc90 0x7f59940fc260 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f599000e040 tx=0x7f5990006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:24:49.162 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:49.156+0000 7f59827fc700 1 -- 172.21.15.3:0/3311618392 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f599001b410 con 0x7f599406fc90 2023-11-17T05:24:49.162 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:49.156+0000 7f59827fc700 1 -- 172.21.15.3:0/3311618392 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f599001b9e0 con 0x7f599406fc90 2023-11-17T05:24:49.162 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:49.156+0000 7f599bd69700 1 -- 172.21.15.3:0/3311618392 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59940fa880 con 0x7f599406fc90 2023-11-17T05:24:49.163 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:49.156+0000 7f59827fc700 1 -- 172.21.15.3:0/3311618392 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f599001a770 con 0x7f599406fc90 2023-11-17T05:24:49.163 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:49.156+0000 7f599bd69700 1 -- 172.21.15.3:0/3311618392 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59940fac90 con 0x7f599406fc90 2023-11-17T05:24:49.163 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:49.157+0000 7f59827fc700 1 -- 172.21.15.3:0/3311618392 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f599000f070 con 0x7f599406fc90 2023-11-17T05:24:49.163 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:49.157+0000 7f59827fc700 1 --2- 172.21.15.3:0/3311618392 >> [v2:172.21.15.3:6800/1850097233,v1:172.21.15.3:6801/1850097233] conn(0x7f597c02fd50 0x7f597c032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:24:49.163 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:49.157+0000 7f59827fc700 1 -- 172.21.15.3:0/3311618392 --> [v2:172.21.15.3:6800/1850097233,v1:172.21.15.3:6801/1850097233] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f597c0328d0 con 0x7f597c02fd50 2023-11-17T05:24:49.163 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:49.157+0000 7f59827fc700 1 -- 172.21.15.3:0/3311618392 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f5990040fd0 con 0x7f599406fc90 2023-11-17T05:24:49.163 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:49.157+0000 7f5999304700 1 -- 172.21.15.3:0/3311618392 >> [v2:172.21.15.3:6800/1850097233,v1:172.21.15.3:6801/1850097233] conn(0x7f597c02fd50 msgr2=0x7f597c032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/1850097233 2023-11-17T05:24:49.164 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:49.157+0000 7f5999304700 1 --2- 172.21.15.3:0/3311618392 >> [v2:172.21.15.3:6800/1850097233,v1:172.21.15.3:6801/1850097233] conn(0x7f597c02fd50 0x7f597c032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-17T05:24:49.358 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:49.358+0000 7f5999304700 1 -- 172.21.15.3:0/3311618392 >> [v2:172.21.15.3:6800/1850097233,v1:172.21.15.3:6801/1850097233] conn(0x7f597c02fd50 msgr2=0x7f597c032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/1850097233 2023-11-17T05:24:49.359 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:49.358+0000 7f5999304700 1 --2- 172.21.15.3:0/3311618392 >> [v2:172.21.15.3:6800/1850097233,v1:172.21.15.3:6801/1850097233] conn(0x7f597c02fd50 0x7f597c032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-17T05:24:49.760 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:49.759+0000 7f5999304700 1 -- 172.21.15.3:0/3311618392 >> [v2:172.21.15.3:6800/1850097233,v1:172.21.15.3:6801/1850097233] conn(0x7f597c02fd50 msgr2=0x7f597c032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/1850097233 2023-11-17T05:24:49.760 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:49.759+0000 7f5999304700 1 --2- 172.21.15.3:0/3311618392 >> [v2:172.21.15.3:6800/1850097233,v1:172.21.15.3:6801/1850097233] conn(0x7f597c02fd50 0x7f597c032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-11-17T05:24:50.561 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:50.560+0000 7f5999304700 1 -- 172.21.15.3:0/3311618392 >> [v2:172.21.15.3:6800/1850097233,v1:172.21.15.3:6801/1850097233] conn(0x7f597c02fd50 msgr2=0x7f597c032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/1850097233 2023-11-17T05:24:50.561 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:50.560+0000 7f5999304700 1 --2- 172.21.15.3:0/3311618392 >> [v2:172.21.15.3:6800/1850097233,v1:172.21.15.3:6801/1850097233] conn(0x7f597c02fd50 0x7f597c032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-11-17T05:24:51.887 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:51.885+0000 7f59827fc700 1 -- 172.21.15.3:0/3311618392 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mgrmap(e 6) v1 ==== 35970+0+0 (secure 0 0 0) 0x7f5990024790 con 0x7f599406fc90 2023-11-17T05:24:51.887 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:51.886+0000 7f59827fc700 1 -- 172.21.15.3:0/3311618392 >> [v2:172.21.15.3:6800/1850097233,v1:172.21.15.3:6801/1850097233] conn(0x7f597c02fd50 msgr2=0x7f597c032210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-17T05:24:51.887 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:51.886+0000 7f59827fc700 1 --2- 172.21.15.3:0/3311618392 >> [v2:172.21.15.3:6800/1850097233,v1:172.21.15.3:6801/1850097233] conn(0x7f597c02fd50 0x7f597c032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-17T05:24:52.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:51 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:24:51.833425+0000 mon.smithi003 (mon.0) 30 : cluster [INF] Active manager daemon smithi003.eddcuc restarted 2023-11-17T05:24:52.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:51 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:24:51.833842+0000 mon.smithi003 (mon.0) 31 : cluster [INF] Activating manager daemon smithi003.eddcuc 2023-11-17T05:24:52.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:51 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:24:51.836092+0000 mon.smithi003 (mon.0) 32 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2023-11-17T05:24:52.894 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:52.894+0000 7f59827fc700 1 -- 172.21.15.3:0/3311618392 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f599001b570 con 0x7f599406fc90 2023-11-17T05:24:52.895 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:52.894+0000 7f59827fc700 1 --2- 172.21.15.3:0/3311618392 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f597c033820 0x7f597c035c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:24:52.895 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:52.894+0000 7f59827fc700 1 -- 172.21.15.3:0/3311618392 --> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f597c0328d0 con 0x7f597c033820 2023-11-17T05:24:52.895 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:52.894+0000 7f5999304700 1 --2- 172.21.15.3:0/3311618392 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f597c033820 0x7f597c035c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:24:52.896 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:52.895+0000 7f5999304700 1 --2- 172.21.15.3:0/3311618392 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f597c033820 0x7f597c035c10 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f59840036d0 tx=0x7f5984006ca0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:24:52.897 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:52.896+0000 7f59827fc700 1 -- 172.21.15.3:0/3311618392 <== mgr.14116 v2:172.21.15.3:6800/2575681100 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f597c0328d0 con 0x7f597c033820 2023-11-17T05:24:52.903 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:52.903+0000 7f599bd69700 1 -- 172.21.15.3:0/3311618392 --> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f5994059100 con 0x7f597c033820 2023-11-17T05:24:52.904 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:52.903+0000 7f59827fc700 1 -- 172.21.15.3:0/3311618392 <== mgr.14116 v2:172.21.15.3:6800/2575681100 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f5994059100 con 0x7f597c033820 2023-11-17T05:24:52.904 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: { 2023-11-17T05:24:52.904 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2023-11-17T05:24:52.904 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "initialized": true 2023-11-17T05:24:52.905 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: } 2023-11-17T05:24:52.905 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:52.904+0000 7f599bd69700 1 -- 172.21.15.3:0/3311618392 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f597c033820 msgr2=0x7f597c035c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:24:52.905 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:52.904+0000 7f599bd69700 1 --2- 172.21.15.3:0/3311618392 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f597c033820 0x7f597c035c10 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f59840036d0 tx=0x7f5984006ca0).stop 2023-11-17T05:24:52.906 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:52.904+0000 7f599bd69700 1 -- 172.21.15.3:0/3311618392 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f599406fc90 msgr2=0x7f59940fc260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:24:52.906 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:52.904+0000 7f599bd69700 1 --2- 172.21.15.3:0/3311618392 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f599406fc90 0x7f59940fc260 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f599000e040 tx=0x7f5990006ee0).stop 2023-11-17T05:24:52.906 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:52.904+0000 7f599bd69700 1 -- 172.21.15.3:0/3311618392 shutdown_connections 2023-11-17T05:24:52.906 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:52.904+0000 7f599bd69700 1 --2- 172.21.15.3:0/3311618392 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f597c033820 0x7f597c035c10 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:24:52.906 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:52.904+0000 7f599bd69700 1 --2- 172.21.15.3:0/3311618392 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f599406fc90 0x7f59940fc260 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:24:52.906 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:52.904+0000 7f599bd69700 1 --2- 172.21.15.3:0/3311618392 >> [v2:172.21.15.3:6800/1850097233,v1:172.21.15.3:6801/1850097233] conn(0x7f597c02fd50 0x7f597c032210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-17T05:24:52.906 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:52.904+0000 7f599bd69700 1 -- 172.21.15.3:0/3311618392 >> 172.21.15.3:0/3311618392 conn(0x7f59940f6280 msgr2=0x7f59940f7e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:24:52.906 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:52.904+0000 7f599bd69700 1 -- 172.21.15.3:0/3311618392 shutdown_connections 2023-11-17T05:24:52.906 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:52.904+0000 7f599bd69700 1 -- 172.21.15.3:0/3311618392 wait complete. 2023-11-17T05:24:53.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:52 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:24:51.887199+0000 mon.smithi003 (mon.0) 33 : cluster [DBG] mgrmap e6: smithi003.eddcuc(active, starting, since 0.0534381s) 2023-11-17T05:24:53.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:52 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:24:51.890016+0000 mon.smithi003 (mon.0) 34 : audit [DBG] from='mgr.14116 172.21.15.3:0/2120313347' entity='mgr.smithi003.eddcuc' cmd=[{"prefix": "mon metadata", "id": "smithi003"}]: dispatch 2023-11-17T05:24:53.271 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:52 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:24:51.890810+0000 mon.smithi003 (mon.0) 35 : audit [DBG] from='mgr.14116 172.21.15.3:0/2120313347' entity='mgr.smithi003.eddcuc' cmd=[{"prefix": "mgr metadata", "who": "smithi003.eddcuc", "id": "smithi003.eddcuc"}]: dispatch 2023-11-17T05:24:53.271 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:52 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:24:51.891604+0000 mon.smithi003 (mon.0) 36 : audit [DBG] from='mgr.14116 172.21.15.3:0/2120313347' entity='mgr.smithi003.eddcuc' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-17T05:24:53.271 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:52 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:24:51.891899+0000 mon.smithi003 (mon.0) 37 : audit [DBG] from='mgr.14116 172.21.15.3:0/2120313347' entity='mgr.smithi003.eddcuc' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-17T05:24:53.271 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:52 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:24:51.892163+0000 mon.smithi003 (mon.0) 38 : audit [DBG] from='mgr.14116 172.21.15.3:0/2120313347' entity='mgr.smithi003.eddcuc' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-17T05:24:53.271 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:52 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:24:51.906079+0000 mon.smithi003 (mon.0) 39 : cluster [INF] Manager daemon smithi003.eddcuc is now available 2023-11-17T05:24:53.271 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:52 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:24:51.921893+0000 mon.smithi003 (mon.0) 40 : audit [INF] from='mgr.14116 172.21.15.3:0/2120313347' entity='mgr.smithi003.eddcuc' 2023-11-17T05:24:53.271 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:52 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:24:51.925094+0000 mon.smithi003 (mon.0) 41 : audit [INF] from='mgr.14116 172.21.15.3:0/2120313347' entity='mgr.smithi003.eddcuc' 2023-11-17T05:24:53.271 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:52 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:24:51.927910+0000 mon.smithi003 (mon.0) 42 : audit [INF] from='mgr.14116 172.21.15.3:0/2120313347' entity='mgr.smithi003.eddcuc' 2023-11-17T05:24:53.271 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:52 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:24:52.029862+0000 mon.smithi003 (mon.0) 43 : audit [INF] from='mgr.14116 172.21.15.3:0/2120313347' entity='mgr.smithi003.eddcuc' 2023-11-17T05:24:53.271 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:52 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:24:52.030646+0000 mon.smithi003 (mon.0) 44 : audit [DBG] from='mgr.14116 172.21.15.3:0/2120313347' entity='mgr.smithi003.eddcuc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:24:53.271 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:52 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:24:52.032059+0000 mon.smithi003 (mon.0) 45 : audit [DBG] from='mgr.14116 172.21.15.3:0/2120313347' entity='mgr.smithi003.eddcuc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:24:53.271 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:52 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:24:52.034583+0000 mon.smithi003 (mon.0) 46 : audit [DBG] from='mgr.14116 172.21.15.3:0/2120313347' entity='mgr.smithi003.eddcuc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:24:53.271 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:52 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:24:52.045783+0000 mon.smithi003 (mon.0) 47 : audit [INF] from='mgr.14116 172.21.15.3:0/2120313347' entity='mgr.smithi003.eddcuc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.eddcuc/mirror_snapshot_schedule"}]: dispatch 2023-11-17T05:24:53.272 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:52 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:24:52.047129+0000 mon.smithi003 (mon.0) 48 : audit [INF] from='mgr.14116 172.21.15.3:0/2120313347' entity='mgr.smithi003.eddcuc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.eddcuc/trash_purge_schedule"}]: dispatch 2023-11-17T05:24:54.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:53 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:24:52.896654+0000 mgr.smithi003.eddcuc (mgr.14116) 1 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-11-17T05:24:54.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:53 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:24:52.896833+0000 mon.smithi003 (mon.0) 49 : cluster [DBG] mgrmap e7: smithi003.eddcuc(active, since 1.06307s) 2023-11-17T05:24:54.271 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:53 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:24:52.904024+0000 mgr.smithi003.eddcuc (mgr.14116) 2 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-11-17T05:24:54.547 INFO:teuthology.orchestra.run.smithi003.stderr:mgr epoch 5 is available 2023-11-17T05:24:54.548 INFO:teuthology.orchestra.run.smithi003.stderr:Setting orchestrator backend to cephadm... 2023-11-17T05:24:54.548 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptcuja8tf:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpk7o1jie2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2023-11-17T05:24:55.164 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:54 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:24:53.899309+0000 mon.smithi003 (mon.0) 50 : cluster [DBG] mgrmap e8: smithi003.eddcuc(active, since 2s) 2023-11-17T05:24:55.901 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.713+0000 7f496fbfd700 1 Processor -- start 2023-11-17T05:24:55.901 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.713+0000 7f496fbfd700 1 -- start start 2023-11-17T05:24:55.902 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.713+0000 7f496fbfd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f496806fc90 0x7f49680fe7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-17T05:24:55.902 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.713+0000 7f496fbfd700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49680fed60 con 0x7f496806fc90 2023-11-17T05:24:55.902 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.713+0000 7f496d999700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f496806fc90 0x7f49680fe7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:24:55.902 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.713+0000 7f496d999700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f496806fc90 0x7f49680fe7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36188/0 (socket says 172.21.15.3:36188) 2023-11-17T05:24:55.902 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.713+0000 7f496d999700 1 -- 172.21.15.3:0/4133848001 learned_addr learned my addr 172.21.15.3:0/4133848001 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:24:55.903 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.714+0000 7f496d999700 1 -- 172.21.15.3:0/4133848001 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49680feea0 con 0x7f496806fc90 2023-11-17T05:24:55.903 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.714+0000 7f496d999700 1 --2- 172.21.15.3:0/4133848001 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f496806fc90 0x7f49680fe7e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f496400b5f0 tx=0x7f4964006ee0).ready entity=mon.0 client_cookie=cc243e018e3812d2 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:24:55.903 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.714+0000 7f496c997700 1 -- 172.21.15.3:0/4133848001 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f496400d600 con 0x7f496806fc90 2023-11-17T05:24:55.903 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.714+0000 7f496c997700 1 -- 172.21.15.3:0/4133848001 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f496400dbd0 con 0x7f496806fc90 2023-11-17T05:24:55.903 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.714+0000 7f496c997700 1 -- 172.21.15.3:0/4133848001 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4964011c00 con 0x7f496806fc90 2023-11-17T05:24:55.903 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.714+0000 7f496fbfd700 1 -- 172.21.15.3:0/4133848001 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f496806fc90 msgr2=0x7f49680fe7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:24:55.904 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.714+0000 7f496fbfd700 1 --2- 172.21.15.3:0/4133848001 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f496806fc90 0x7f49680fe7e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f496400b5f0 tx=0x7f4964006ee0).stop 2023-11-17T05:24:55.904 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.715+0000 7f496fbfd700 1 -- 172.21.15.3:0/4133848001 shutdown_connections 2023-11-17T05:24:55.904 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.715+0000 7f496fbfd700 1 --2- 172.21.15.3:0/4133848001 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f496806fc90 0x7f49680fe7e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:24:55.904 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.715+0000 7f496fbfd700 1 -- 172.21.15.3:0/4133848001 >> 172.21.15.3:0/4133848001 conn(0x7f49680f6280 msgr2=0x7f49680f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:24:55.904 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.715+0000 7f496fbfd700 1 -- 172.21.15.3:0/4133848001 shutdown_connections 2023-11-17T05:24:55.904 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.715+0000 7f496fbfd700 1 -- 172.21.15.3:0/4133848001 wait complete. 2023-11-17T05:24:55.905 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.715+0000 7f496fbfd700 1 Processor -- start 2023-11-17T05:24:55.905 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.715+0000 7f496fbfd700 1 -- start start 2023-11-17T05:24:55.905 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.716+0000 7f496fbfd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f496806fc90 0x7f496810a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:24:55.905 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.716+0000 7f496fbfd700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f496810aee0 con 0x7f496806fc90 2023-11-17T05:24:55.905 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.716+0000 7f496d999700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f496806fc90 0x7f496810a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:24:55.906 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.716+0000 7f496d999700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f496806fc90 0x7f496810a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36202/0 (socket says 172.21.15.3:36202) 2023-11-17T05:24:55.906 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.716+0000 7f496d999700 1 -- 172.21.15.3:0/3031520610 learned_addr learned my addr 172.21.15.3:0/3031520610 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:24:55.906 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.716+0000 7f496d999700 1 -- 172.21.15.3:0/3031520610 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f496400b200 con 0x7f496806fc90 2023-11-17T05:24:55.913 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.716+0000 7f496d999700 1 --2- 172.21.15.3:0/3031520610 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f496806fc90 0x7f496810a9f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f4964000c00 tx=0x7f4964006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:24:55.913 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.716+0000 7f49567fc700 1 -- 172.21.15.3:0/3031520610 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4964011690 con 0x7f496806fc90 2023-11-17T05:24:55.913 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.716+0000 7f49567fc700 1 -- 172.21.15.3:0/3031520610 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f496400d9f0 con 0x7f496806fc90 2023-11-17T05:24:55.913 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.716+0000 7f49567fc700 1 -- 172.21.15.3:0/3031520610 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f496401b710 con 0x7f496806fc90 2023-11-17T05:24:55.914 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.716+0000 7f496fbfd700 1 -- 172.21.15.3:0/3031520610 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f496810b0e0 con 0x7f496806fc90 2023-11-17T05:24:55.914 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.717+0000 7f496fbfd700 1 -- 172.21.15.3:0/3031520610 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f496810b4f0 con 0x7f496806fc90 2023-11-17T05:24:55.914 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.717+0000 7f49567fc700 1 -- 172.21.15.3:0/3031520610 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f496400f070 con 0x7f496806fc90 2023-11-17T05:24:55.914 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.717+0000 7f49567fc700 1 --2- 172.21.15.3:0/3031520610 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f495002fdb0 0x7f4950032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:24:55.914 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.717+0000 7f49567fc700 1 -- 172.21.15.3:0/3031520610 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f4964041230 con 0x7f496806fc90 2023-11-17T05:24:55.915 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.718+0000 7f496d198700 1 --2- 172.21.15.3:0/3031520610 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f495002fdb0 0x7f4950032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:24:55.915 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.718+0000 7f496fbfd700 1 -- 172.21.15.3:0/3031520610 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4968059100 con 0x7f496806fc90 2023-11-17T05:24:55.915 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.718+0000 7f496d198700 1 --2- 172.21.15.3:0/3031520610 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f495002fdb0 0x7f4950032270 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f49580096a0 tx=0x7f4958006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:24:55.915 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.722+0000 7f49567fc700 1 -- 172.21.15.3:0/3031520610 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4964016070 con 0x7f496806fc90 2023-11-17T05:24:55.925 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.924+0000 7f496fbfd700 1 -- 172.21.15.3:0/3031520610 --> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f4968104890 con 0x7f495002fdb0 2023-11-17T05:24:55.930 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.930+0000 7f49567fc700 1 -- 172.21.15.3:0/3031520610 <== mgr.14116 v2:172.21.15.3:6800/2575681100 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4968104890 con 0x7f495002fdb0 2023-11-17T05:24:55.932 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.932+0000 7f496fbfd700 1 -- 172.21.15.3:0/3031520610 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f495002fdb0 msgr2=0x7f4950032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:24:55.933 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.932+0000 7f496fbfd700 1 --2- 172.21.15.3:0/3031520610 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f495002fdb0 0x7f4950032270 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f49580096a0 tx=0x7f4958006b40).stop 2023-11-17T05:24:55.933 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.932+0000 7f496fbfd700 1 -- 172.21.15.3:0/3031520610 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f496806fc90 msgr2=0x7f496810a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:24:55.933 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.932+0000 7f496fbfd700 1 --2- 172.21.15.3:0/3031520610 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f496806fc90 0x7f496810a9f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f4964000c00 tx=0x7f4964006ee0).stop 2023-11-17T05:24:55.934 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.932+0000 7f496fbfd700 1 -- 172.21.15.3:0/3031520610 shutdown_connections 2023-11-17T05:24:55.934 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.932+0000 7f496fbfd700 1 --2- 172.21.15.3:0/3031520610 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f495002fdb0 0x7f4950032270 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:24:55.934 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.932+0000 7f496fbfd700 1 --2- 172.21.15.3:0/3031520610 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f496806fc90 0x7f496810a9f0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:24:55.934 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.932+0000 7f496fbfd700 1 -- 172.21.15.3:0/3031520610 >> 172.21.15.3:0/3031520610 conn(0x7f49680f6280 msgr2=0x7f49680f7e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:24:55.934 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.932+0000 7f496fbfd700 1 -- 172.21.15.3:0/3031520610 shutdown_connections 2023-11-17T05:24:55.934 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:55.932+0000 7f496fbfd700 1 -- 172.21.15.3:0/3031520610 wait complete. 2023-11-17T05:24:57.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:56 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:24:55.926152+0000 mgr.smithi003.eddcuc (mgr.14116) 3 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:24:57.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:56 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:24:55.930430+0000 mon.smithi003 (mon.0) 51 : audit [INF] from='mgr.14116 172.21.15.3:0/2120313347' entity='mgr.smithi003.eddcuc' 2023-11-17T05:24:57.271 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:24:56 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:24:55.934963+0000 mon.smithi003 (mon.0) 52 : audit [DBG] from='mgr.14116 172.21.15.3:0/2120313347' entity='mgr.smithi003.eddcuc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:24:58.005 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptcuja8tf:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpk7o1jie2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2023-11-17T05:24:59.178 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.141+0000 7f512f984700 1 Processor -- start 2023-11-17T05:24:59.179 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.141+0000 7f512f984700 1 -- start start 2023-11-17T05:24:59.179 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.142+0000 7f512f984700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51280faa50 0x7f51280fae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-17T05:24:59.179 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.142+0000 7f512f984700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51280fb360 con 0x7f51280faa50 2023-11-17T05:24:59.179 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.142+0000 7f512d720700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51280faa50 0x7f51280fae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:24:59.179 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.142+0000 7f512d720700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51280faa50 0x7f51280fae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35404/0 (socket says 172.21.15.3:35404) 2023-11-17T05:24:59.179 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.142+0000 7f512d720700 1 -- 172.21.15.3:0/1702099171 learned_addr learned my addr 172.21.15.3:0/1702099171 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:24:59.179 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.142+0000 7f512d720700 1 -- 172.21.15.3:0/1702099171 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51280fb4a0 con 0x7f51280faa50 2023-11-17T05:24:59.179 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.143+0000 7f512d720700 1 --2- 172.21.15.3:0/1702099171 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51280faa50 0x7f51280fae70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f511800b5f0 tx=0x7f5118006ee0).ready entity=mon.0 client_cookie=6398be38737f943d server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:24:59.179 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.143+0000 7f5117fff700 1 -- 172.21.15.3:0/1702099171 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f511800d600 con 0x7f51280faa50 2023-11-17T05:24:59.179 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.143+0000 7f5117fff700 1 -- 172.21.15.3:0/1702099171 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f511800dbd0 con 0x7f51280faa50 2023-11-17T05:24:59.179 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.143+0000 7f5117fff700 1 -- 172.21.15.3:0/1702099171 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5118011c00 con 0x7f51280faa50 2023-11-17T05:24:59.180 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.143+0000 7f512f984700 1 -- 172.21.15.3:0/1702099171 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51280faa50 msgr2=0x7f51280fae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:24:59.180 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.143+0000 7f512f984700 1 --2- 172.21.15.3:0/1702099171 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51280faa50 0x7f51280fae70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f511800b5f0 tx=0x7f5118006ee0).stop 2023-11-17T05:24:59.180 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.143+0000 7f512f984700 1 -- 172.21.15.3:0/1702099171 shutdown_connections 2023-11-17T05:24:59.180 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.143+0000 7f512f984700 1 --2- 172.21.15.3:0/1702099171 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51280faa50 0x7f51280fae70 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:24:59.180 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.143+0000 7f512f984700 1 -- 172.21.15.3:0/1702099171 >> 172.21.15.3:0/1702099171 conn(0x7f51280f6270 msgr2=0x7f51280f8690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:24:59.180 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.143+0000 7f512f984700 1 -- 172.21.15.3:0/1702099171 shutdown_connections 2023-11-17T05:24:59.180 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.143+0000 7f512f984700 1 -- 172.21.15.3:0/1702099171 wait complete. 2023-11-17T05:24:59.180 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.144+0000 7f512f984700 1 Processor -- start 2023-11-17T05:24:59.180 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.144+0000 7f512f984700 1 -- start start 2023-11-17T05:24:59.180 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.144+0000 7f512f984700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51280faa50 0x7f512810a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:24:59.180 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.144+0000 7f512f984700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f512810adb0 con 0x7f51280faa50 2023-11-17T05:24:59.180 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.144+0000 7f512d720700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51280faa50 0x7f512810a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:24:59.180 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.144+0000 7f512d720700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51280faa50 0x7f512810a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35412/0 (socket says 172.21.15.3:35412) 2023-11-17T05:24:59.181 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.144+0000 7f512d720700 1 -- 172.21.15.3:0/2147791421 learned_addr learned my addr 172.21.15.3:0/2147791421 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:24:59.181 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.145+0000 7f512d720700 1 -- 172.21.15.3:0/2147791421 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f511800b200 con 0x7f51280faa50 2023-11-17T05:24:59.181 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.145+0000 7f512d720700 1 --2- 172.21.15.3:0/2147791421 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51280faa50 0x7f512810a8c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f511800e040 tx=0x7f5118006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:24:59.181 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.145+0000 7f5115ffb700 1 -- 172.21.15.3:0/2147791421 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f511801b410 con 0x7f51280faa50 2023-11-17T05:24:59.181 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.145+0000 7f5115ffb700 1 -- 172.21.15.3:0/2147791421 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f511801b9e0 con 0x7f51280faa50 2023-11-17T05:24:59.181 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.145+0000 7f5115ffb700 1 -- 172.21.15.3:0/2147791421 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f511801a820 con 0x7f51280faa50 2023-11-17T05:24:59.181 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.145+0000 7f512f984700 1 -- 172.21.15.3:0/2147791421 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f512810afb0 con 0x7f51280faa50 2023-11-17T05:24:59.181 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.145+0000 7f512f984700 1 -- 172.21.15.3:0/2147791421 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f512810b440 con 0x7f51280faa50 2023-11-17T05:24:59.181 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.146+0000 7f5115ffb700 1 -- 172.21.15.3:0/2147791421 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f511800f070 con 0x7f51280faa50 2023-11-17T05:24:59.181 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.146+0000 7f512f984700 1 -- 172.21.15.3:0/2147791421 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5128045ad0 con 0x7f51280faa50 2023-11-17T05:24:59.181 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.146+0000 7f5115ffb700 1 --2- 172.21.15.3:0/2147791421 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f511002fdb0 0x7f5110032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:24:59.181 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.146+0000 7f5115ffb700 1 -- 172.21.15.3:0/2147791421 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f5118041310 con 0x7f51280faa50 2023-11-17T05:24:59.182 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.147+0000 7f512cf1f700 1 --2- 172.21.15.3:0/2147791421 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f511002fdb0 0x7f5110032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:24:59.182 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.147+0000 7f512cf1f700 1 --2- 172.21.15.3:0/2147791421 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f511002fdb0 0x7f5110032270 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f51240096a0 tx=0x7f5124006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:24:59.182 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.151+0000 7f5115ffb700 1 -- 172.21.15.3:0/2147791421 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5118019950 con 0x7f51280faa50 2023-11-17T05:24:59.321 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.320+0000 7f512f984700 1 -- 172.21.15.3:0/2147791421 --> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f512810bd60 con 0x7f511002fdb0 2023-11-17T05:24:59.321 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.320+0000 7f5115ffb700 1 -- 172.21.15.3:0/2147791421 <== mgr.14116 v2:172.21.15.3:6800/2575681100 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f512810bd60 con 0x7f511002fdb0 2023-11-17T05:24:59.321 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: value unchanged 2023-11-17T05:24:59.323 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.322+0000 7f512f984700 1 -- 172.21.15.3:0/2147791421 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f511002fdb0 msgr2=0x7f5110032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:24:59.323 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.322+0000 7f512f984700 1 --2- 172.21.15.3:0/2147791421 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f511002fdb0 0x7f5110032270 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f51240096a0 tx=0x7f5124006b40).stop 2023-11-17T05:24:59.323 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.322+0000 7f512f984700 1 -- 172.21.15.3:0/2147791421 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51280faa50 msgr2=0x7f512810a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:24:59.323 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.322+0000 7f512f984700 1 --2- 172.21.15.3:0/2147791421 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51280faa50 0x7f512810a8c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f511800e040 tx=0x7f5118006ee0).stop 2023-11-17T05:24:59.324 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.322+0000 7f512f984700 1 -- 172.21.15.3:0/2147791421 shutdown_connections 2023-11-17T05:24:59.324 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.322+0000 7f512f984700 1 --2- 172.21.15.3:0/2147791421 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f511002fdb0 0x7f5110032270 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:24:59.324 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.322+0000 7f512f984700 1 --2- 172.21.15.3:0/2147791421 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51280faa50 0x7f512810a8c0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:24:59.324 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.322+0000 7f512f984700 1 -- 172.21.15.3:0/2147791421 >> 172.21.15.3:0/2147791421 conn(0x7f51280f6270 msgr2=0x7f512806f830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:24:59.324 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.322+0000 7f512f984700 1 -- 172.21.15.3:0/2147791421 shutdown_connections 2023-11-17T05:24:59.324 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:24:59.322+0000 7f512f984700 1 -- 172.21.15.3:0/2147791421 wait complete. 2023-11-17T05:25:00.520 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:00 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:24:59.320960+0000 mgr.smithi003.eddcuc (mgr.14116) 4 : audit [DBG] from='client.14128 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:25:01.174 INFO:teuthology.orchestra.run.smithi003.stderr:Generating ssh key... 2023-11-17T05:25:01.175 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptcuja8tf:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpk7o1jie2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2023-11-17T05:25:01.754 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:01.751+0000 7f6c0c9ab700 1 Processor -- start 2023-11-17T05:25:01.754 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:01.751+0000 7f6c0c9ab700 1 -- start start 2023-11-17T05:25:01.754 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:01.752+0000 7f6c0c9ab700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c080faae0 0x7f6c080faf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:01.754 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:01.752+0000 7f6c0c9ab700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c080fb480 con 0x7f6c080faae0 2023-11-17T05:25:01.794 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:01.752+0000 7f6c0659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c080faae0 0x7f6c080faf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:01.794 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:01.752+0000 7f6c0659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c080faae0 0x7f6c080faf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35426/0 (socket says 172.21.15.3:35426) 2023-11-17T05:25:01.794 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:01.752+0000 7f6c0659c700 1 -- 172.21.15.3:0/2127990834 learned_addr learned my addr 172.21.15.3:0/2127990834 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:25:01.794 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:01.752+0000 7f6c0659c700 1 -- 172.21.15.3:0/2127990834 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c080fb5c0 con 0x7f6c080faae0 2023-11-17T05:25:01.797 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:01.752+0000 7f6c0659c700 1 --2- 172.21.15.3:0/2127990834 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c080faae0 0x7f6c080faf00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f6bf800b870 tx=0x7f6bf8006f70).ready entity=mon.0 client_cookie=4d6b3a29bfc36281 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:01.797 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:01.752+0000 7f6c0559a700 1 -- 172.21.15.3:0/2127990834 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bf800da20 con 0x7f6c080faae0 2023-11-17T05:25:01.797 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:01.753+0000 7f6c0559a700 1 -- 172.21.15.3:0/2127990834 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f6bf8010470 con 0x7f6c080faae0 2023-11-17T05:25:01.797 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:01.753+0000 7f6c0559a700 1 -- 172.21.15.3:0/2127990834 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bf800f5b0 con 0x7f6c080faae0 2023-11-17T05:25:01.797 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:01.753+0000 7f6c0c9ab700 1 -- 172.21.15.3:0/2127990834 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c080faae0 msgr2=0x7f6c080faf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:01.797 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:01.753+0000 7f6c0c9ab700 1 --2- 172.21.15.3:0/2127990834 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c080faae0 0x7f6c080faf00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f6bf800b870 tx=0x7f6bf8006f70).stop 2023-11-17T05:25:01.797 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:01.753+0000 7f6c0c9ab700 1 -- 172.21.15.3:0/2127990834 shutdown_connections 2023-11-17T05:25:01.798 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:01.753+0000 7f6c0c9ab700 1 --2- 172.21.15.3:0/2127990834 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c080faae0 0x7f6c080faf00 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:01.891 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:01.753+0000 7f6c0c9ab700 1 -- 172.21.15.3:0/2127990834 >> 172.21.15.3:0/2127990834 conn(0x7f6c080f6280 msgr2=0x7f6c080f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:01.891 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:01.753+0000 7f6c0c9ab700 1 -- 172.21.15.3:0/2127990834 shutdown_connections 2023-11-17T05:25:01.891 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:01.753+0000 7f6c0c9ab700 1 -- 172.21.15.3:0/2127990834 wait complete. 2023-11-17T05:25:01.891 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:01.754+0000 7f6c0c9ab700 1 Processor -- start 2023-11-17T05:25:01.891 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:01.754+0000 7f6c0c9ab700 1 -- start start 2023-11-17T05:25:01.891 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:01.754+0000 7f6c0c9ab700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c080faae0 0x7f6c08108750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:01.891 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:01.754+0000 7f6c0c9ab700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c08108c40 con 0x7f6c080faae0 2023-11-17T05:25:01.891 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:01.754+0000 7f6c0659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c080faae0 0x7f6c08108750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:01.892 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:01.754+0000 7f6c0659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c080faae0 0x7f6c08108750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35438/0 (socket says 172.21.15.3:35438) 2023-11-17T05:25:01.892 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:01.754+0000 7f6c0659c700 1 -- 172.21.15.3:0/995165863 learned_addr learned my addr 172.21.15.3:0/995165863 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:25:01.892 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:01.754+0000 7f6c0659c700 1 -- 172.21.15.3:0/995165863 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf800b480 con 0x7f6c080faae0 2023-11-17T05:25:01.892 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:01.755+0000 7f6c0659c700 1 --2- 172.21.15.3:0/995165863 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c080faae0 0x7f6c08108750 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f6bf8010df0 tx=0x7f6bf8006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:01.892 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:01.755+0000 7f6bf6ffd700 1 -- 172.21.15.3:0/995165863 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bf800f820 con 0x7f6c080faae0 2023-11-17T05:25:01.892 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:01.755+0000 7f6bf6ffd700 1 -- 172.21.15.3:0/995165863 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f6bf800fdf0 con 0x7f6c080faae0 2023-11-17T05:25:01.892 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:01.755+0000 7f6c0c9ab700 1 -- 172.21.15.3:0/995165863 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c08108e40 con 0x7f6c080faae0 2023-11-17T05:25:01.892 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:01.755+0000 7f6bf6ffd700 1 -- 172.21.15.3:0/995165863 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bf8019c70 con 0x7f6c080faae0 2023-11-17T05:25:01.892 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:01.755+0000 7f6c0c9ab700 1 -- 172.21.15.3:0/995165863 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c08109250 con 0x7f6c080faae0 2023-11-17T05:25:01.892 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:01.756+0000 7f6bf6ffd700 1 -- 172.21.15.3:0/995165863 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f6bf8016030 con 0x7f6c080faae0 2023-11-17T05:25:01.892 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:01.756+0000 7f6bf6ffd700 1 --2- 172.21.15.3:0/995165863 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f6bec02fdb0 0x7f6bec032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:01.892 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:01.756+0000 7f6bf6ffd700 1 -- 172.21.15.3:0/995165863 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f6bf8043e60 con 0x7f6c080faae0 2023-11-17T05:25:01.892 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:01.756+0000 7f6c05d9b700 1 --2- 172.21.15.3:0/995165863 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f6bec02fdb0 0x7f6bec032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:01.892 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:01.757+0000 7f6c05d9b700 1 --2- 172.21.15.3:0/995165863 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f6bec02fdb0 0x7f6bec032270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f6bfc0096a0 tx=0x7f6bfc006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:01.893 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:01.757+0000 7f6c0c9ab700 1 -- 172.21.15.3:0/995165863 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c08059100 con 0x7f6c080faae0 2023-11-17T05:25:01.893 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:01.761+0000 7f6bf6ffd700 1 -- 172.21.15.3:0/995165863 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6bf8014070 con 0x7f6c080faae0 2023-11-17T05:25:01.930 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:01.928+0000 7f6c0c9ab700 1 -- 172.21.15.3:0/995165863 --> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f6c08102590 con 0x7f6bec02fdb0 2023-11-17T05:25:02.021 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.019+0000 7f6bf6ffd700 1 -- 172.21.15.3:0/995165863 <== mgr.14116 v2:172.21.15.3:6800/2575681100 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6c08102590 con 0x7f6bec02fdb0 2023-11-17T05:25:02.023 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.021+0000 7f6c0c9ab700 1 -- 172.21.15.3:0/995165863 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f6bec02fdb0 msgr2=0x7f6bec032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:02.023 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.021+0000 7f6c0c9ab700 1 --2- 172.21.15.3:0/995165863 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f6bec02fdb0 0x7f6bec032270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f6bfc0096a0 tx=0x7f6bfc006b40).stop 2023-11-17T05:25:02.023 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.021+0000 7f6c0c9ab700 1 -- 172.21.15.3:0/995165863 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c080faae0 msgr2=0x7f6c08108750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:02.023 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.021+0000 7f6c0c9ab700 1 --2- 172.21.15.3:0/995165863 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c080faae0 0x7f6c08108750 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f6bf8010df0 tx=0x7f6bf8006f70).stop 2023-11-17T05:25:02.023 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.021+0000 7f6c0c9ab700 1 -- 172.21.15.3:0/995165863 shutdown_connections 2023-11-17T05:25:02.023 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.021+0000 7f6c0c9ab700 1 --2- 172.21.15.3:0/995165863 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f6bec02fdb0 0x7f6bec032270 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:02.024 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.021+0000 7f6c0c9ab700 1 --2- 172.21.15.3:0/995165863 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c080faae0 0x7f6c08108750 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:02.024 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.021+0000 7f6c0c9ab700 1 -- 172.21.15.3:0/995165863 >> 172.21.15.3:0/995165863 conn(0x7f6c080f6280 msgr2=0x7f6c080ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:02.024 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.021+0000 7f6c0c9ab700 1 -- 172.21.15.3:0/995165863 shutdown_connections 2023-11-17T05:25:02.024 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.021+0000 7f6c0c9ab700 1 -- 172.21.15.3:0/995165863 wait complete. 2023-11-17T05:25:02.387 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptcuja8tf:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpk7o1jie2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2023-11-17T05:25:02.757 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.754+0000 7f5b66d01700 1 Processor -- start 2023-11-17T05:25:02.757 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.754+0000 7f5b66d01700 1 -- start start 2023-11-17T05:25:02.757 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.754+0000 7f5b66d01700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b6006fc50 0x7f5b60100c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:02.757 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.754+0000 7f5b66d01700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b600fc6a0 con 0x7f5b6006fc50 2023-11-17T05:25:02.757 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.754+0000 7f5b64a9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b6006fc50 0x7f5b60100c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:02.757 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.754+0000 7f5b64a9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b6006fc50 0x7f5b60100c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35446/0 (socket says 172.21.15.3:35446) 2023-11-17T05:25:02.757 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.754+0000 7f5b64a9d700 1 -- 172.21.15.3:0/880482635 learned_addr learned my addr 172.21.15.3:0/880482635 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:25:02.757 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.755+0000 7f5b64a9d700 1 -- 172.21.15.3:0/880482635 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b60101110 con 0x7f5b6006fc50 2023-11-17T05:25:02.758 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.755+0000 7f5b64a9d700 1 --2- 172.21.15.3:0/880482635 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b6006fc50 0x7f5b60100c20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f5b5400b5f0 tx=0x7f5b54006ee0).ready entity=mon.0 client_cookie=dde6ce78c0630e15 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:02.758 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.755+0000 7f5b5f7fe700 1 -- 172.21.15.3:0/880482635 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b5400d600 con 0x7f5b6006fc50 2023-11-17T05:25:02.758 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.755+0000 7f5b5f7fe700 1 -- 172.21.15.3:0/880482635 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f5b5400dbd0 con 0x7f5b6006fc50 2023-11-17T05:25:02.760 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.755+0000 7f5b5f7fe700 1 -- 172.21.15.3:0/880482635 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b54011c00 con 0x7f5b6006fc50 2023-11-17T05:25:02.760 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.756+0000 7f5b66d01700 1 -- 172.21.15.3:0/880482635 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b6006fc50 msgr2=0x7f5b60100c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:02.760 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.756+0000 7f5b66d01700 1 --2- 172.21.15.3:0/880482635 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b6006fc50 0x7f5b60100c20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f5b5400b5f0 tx=0x7f5b54006ee0).stop 2023-11-17T05:25:02.760 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.756+0000 7f5b66d01700 1 -- 172.21.15.3:0/880482635 shutdown_connections 2023-11-17T05:25:02.760 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.756+0000 7f5b66d01700 1 --2- 172.21.15.3:0/880482635 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b6006fc50 0x7f5b60100c20 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:02.760 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.756+0000 7f5b66d01700 1 -- 172.21.15.3:0/880482635 >> 172.21.15.3:0/880482635 conn(0x7f5b600f6280 msgr2=0x7f5b600f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:02.760 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.756+0000 7f5b66d01700 1 -- 172.21.15.3:0/880482635 shutdown_connections 2023-11-17T05:25:02.760 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.756+0000 7f5b66d01700 1 -- 172.21.15.3:0/880482635 wait complete. 2023-11-17T05:25:02.760 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.757+0000 7f5b66d01700 1 Processor -- start 2023-11-17T05:25:02.760 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.757+0000 7f5b66d01700 1 -- start start 2023-11-17T05:25:02.761 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.757+0000 7f5b66d01700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b6006fc50 0x7f5b6010cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:02.761 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.757+0000 7f5b66d01700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6010d0c0 con 0x7f5b6006fc50 2023-11-17T05:25:02.761 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.757+0000 7f5b64a9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b6006fc50 0x7f5b6010cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:02.761 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.757+0000 7f5b64a9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b6006fc50 0x7f5b6010cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35448/0 (socket says 172.21.15.3:35448) 2023-11-17T05:25:02.761 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.757+0000 7f5b64a9d700 1 -- 172.21.15.3:0/1288597261 learned_addr learned my addr 172.21.15.3:0/1288597261 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:25:02.761 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.757+0000 7f5b64a9d700 1 -- 172.21.15.3:0/1288597261 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b5400b200 con 0x7f5b6006fc50 2023-11-17T05:25:02.761 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.758+0000 7f5b64a9d700 1 --2- 172.21.15.3:0/1288597261 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b6006fc50 0x7f5b6010cbd0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f5b5400b1d0 tx=0x7f5b54006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:02.761 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.758+0000 7f5b5d7fa700 1 -- 172.21.15.3:0/1288597261 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b54011690 con 0x7f5b6006fc50 2023-11-17T05:25:02.761 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.758+0000 7f5b66d01700 1 -- 172.21.15.3:0/1288597261 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b6010d2c0 con 0x7f5b6006fc50 2023-11-17T05:25:02.761 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.758+0000 7f5b5d7fa700 1 -- 172.21.15.3:0/1288597261 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f5b5401b880 con 0x7f5b6006fc50 2023-11-17T05:25:02.761 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.758+0000 7f5b5d7fa700 1 -- 172.21.15.3:0/1288597261 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b5401a650 con 0x7f5b6006fc50 2023-11-17T05:25:02.761 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.758+0000 7f5b66d01700 1 -- 172.21.15.3:0/1288597261 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b6010d6d0 con 0x7f5b6006fc50 2023-11-17T05:25:02.762 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.759+0000 7f5b5d7fa700 1 -- 172.21.15.3:0/1288597261 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f5b5400f070 con 0x7f5b6006fc50 2023-11-17T05:25:02.762 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.759+0000 7f5b5d7fa700 1 --2- 172.21.15.3:0/1288597261 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f5b4402fd50 0x7f5b44032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:02.762 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.759+0000 7f5b5d7fa700 1 -- 172.21.15.3:0/1288597261 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f5b540410a0 con 0x7f5b6006fc50 2023-11-17T05:25:02.762 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.759+0000 7f5b66d01700 1 -- 172.21.15.3:0/1288597261 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b60045ad0 con 0x7f5b6006fc50 2023-11-17T05:25:02.763 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.759+0000 7f5b5ffff700 1 --2- 172.21.15.3:0/1288597261 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f5b4402fd50 0x7f5b44032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:02.763 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.760+0000 7f5b5ffff700 1 --2- 172.21.15.3:0/1288597261 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f5b4402fd50 0x7f5b44032210 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f5b4c0096a0 tx=0x7f5b4c006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:02.765 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.764+0000 7f5b5d7fa700 1 -- 172.21.15.3:0/1288597261 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5b540226f0 con 0x7f5b6006fc50 2023-11-17T05:25:02.933 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.931+0000 7f5b66d01700 1 -- 172.21.15.3:0/1288597261 --> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f5b60106b00 con 0x7f5b4402fd50 2023-11-17T05:25:02.934 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.932+0000 7f5b5d7fa700 1 -- 172.21.15.3:0/1288597261 <== mgr.14116 v2:172.21.15.3:6800/2575681100 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f5b60106b00 con 0x7f5b4402fd50 2023-11-17T05:25:02.934 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDrzTHMzFX2NWVwNhuoufWlyf/uJpnw9PgXmv2GlMM71C50xs9YU4jzcVV3I6GU9YnqHwEjMLwjwA1H0VePmw+83pyfhzq7WbqCIvAZUXNHnMKD272bIQ8jLVM+2piyZOCo+tvmpcvncbTV34nouTQ0+IDcHOzDHNXq8KOmU3t9e/mGS+YHU3upQ/C9a8t2JQlxFfIBjYjxSRa7znX+gFikk2eABQ3Xst5jpMhuJWrTAOnA9Mlq9MhVyr8NXJmdRmZ5tuu6Lzi6Za31q3/0Gtq8rhpty7SwYN6BFY1achyEZUOI6gN4GyHT7sD/pRJGw3IAOhjTBaQ3ygjq10I9/5Q9R4pgaS2QTBbG6WRGwRuhYg1h0YwBWQ8uvNOBByYTU+RsjNU+KDi0fnlVYhj4qQkeBN7QggLcob8maIDJv92STaKZnoD/ZaoUispkCxJGlN3xq4WxK0J8O1XGQ7V5d6Fz8N14p4C1iMmMPsP+Q5Y89PLknYcPq2k3eBwJu1FA0wk= ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7 2023-11-17T05:25:02.935 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.934+0000 7f5b66d01700 1 -- 172.21.15.3:0/1288597261 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f5b4402fd50 msgr2=0x7f5b44032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:02.935 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.934+0000 7f5b66d01700 1 --2- 172.21.15.3:0/1288597261 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f5b4402fd50 0x7f5b44032210 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f5b4c0096a0 tx=0x7f5b4c006b40).stop 2023-11-17T05:25:02.936 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.934+0000 7f5b66d01700 1 -- 172.21.15.3:0/1288597261 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b6006fc50 msgr2=0x7f5b6010cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:02.936 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.934+0000 7f5b66d01700 1 --2- 172.21.15.3:0/1288597261 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b6006fc50 0x7f5b6010cbd0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f5b5400b1d0 tx=0x7f5b54006ee0).stop 2023-11-17T05:25:02.936 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.934+0000 7f5b66d01700 1 -- 172.21.15.3:0/1288597261 shutdown_connections 2023-11-17T05:25:02.936 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.934+0000 7f5b66d01700 1 --2- 172.21.15.3:0/1288597261 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f5b4402fd50 0x7f5b44032210 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:02.937 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.934+0000 7f5b66d01700 1 --2- 172.21.15.3:0/1288597261 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b6006fc50 0x7f5b6010cbd0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:02.937 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.934+0000 7f5b66d01700 1 -- 172.21.15.3:0/1288597261 >> 172.21.15.3:0/1288597261 conn(0x7f5b600f6280 msgr2=0x7f5b600f7e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:02.937 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.934+0000 7f5b66d01700 1 -- 172.21.15.3:0/1288597261 shutdown_connections 2023-11-17T05:25:02.937 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:02.934+0000 7f5b66d01700 1 -- 172.21.15.3:0/1288597261 wait complete. 2023-11-17T05:25:03.225 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:03 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cephadm 2023-11-17T05:25:01.931060+0000 mgr.smithi003.eddcuc (mgr.14116) 6 : cephadm [INF] Generating ssh key... 2023-11-17T05:25:03.325 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:03 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:02.017978+0000 mon.smithi003 (mon.0) 53 : audit [INF] from='mgr.14116 172.21.15.3:0/2120313347' entity='mgr.smithi003.eddcuc' 2023-11-17T05:25:03.326 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:03 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:02.019839+0000 mon.smithi003 (mon.0) 54 : audit [INF] from='mgr.14116 172.21.15.3:0/2120313347' entity='mgr.smithi003.eddcuc' 2023-11-17T05:25:03.326 INFO:teuthology.orchestra.run.smithi003.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-11-17T05:25:03.327 INFO:teuthology.orchestra.run.smithi003.stderr:Adding key to root@localhost authorized_keys... 2023-11-17T05:25:03.327 INFO:teuthology.orchestra.run.smithi003.stderr:Adding host smithi003... 2023-11-17T05:25:03.327 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptcuja8tf:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpk7o1jie2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi003 172.21.15.3 2023-11-17T05:25:03.671 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:03.668+0000 7fd664935700 1 Processor -- start 2023-11-17T05:25:03.671 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:03.668+0000 7fd664935700 1 -- start start 2023-11-17T05:25:03.671 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:03.669+0000 7fd664935700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6600faae0 0x7fd6600faf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:03.671 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:03.669+0000 7fd664935700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6600fb480 con 0x7fd6600faae0 2023-11-17T05:25:03.671 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:03.669+0000 7fd65e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6600faae0 0x7fd6600faf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:03.671 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:03.669+0000 7fd65e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6600faae0 0x7fd6600faf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35454/0 (socket says 172.21.15.3:35454) 2023-11-17T05:25:03.671 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:03.669+0000 7fd65e59c700 1 -- 172.21.15.3:0/3740144815 learned_addr learned my addr 172.21.15.3:0/3740144815 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:25:03.671 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:03.669+0000 7fd65e59c700 1 -- 172.21.15.3:0/3740144815 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6600fb5c0 con 0x7fd6600faae0 2023-11-17T05:25:03.671 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:03.669+0000 7fd65e59c700 1 --2- 172.21.15.3:0/3740144815 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6600faae0 0x7fd6600faf00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fd65400b5f0 tx=0x7fd654006ee0).ready entity=mon.0 client_cookie=8185b8055dd410d8 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:03.672 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:03.669+0000 7fd65d59a700 1 -- 172.21.15.3:0/3740144815 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd65400d600 con 0x7fd6600faae0 2023-11-17T05:25:03.673 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:03.670+0000 7fd65d59a700 1 -- 172.21.15.3:0/3740144815 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd65400dbd0 con 0x7fd6600faae0 2023-11-17T05:25:03.673 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:03.670+0000 7fd65d59a700 1 -- 172.21.15.3:0/3740144815 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd654011c00 con 0x7fd6600faae0 2023-11-17T05:25:03.673 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:03.670+0000 7fd664935700 1 -- 172.21.15.3:0/3740144815 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6600faae0 msgr2=0x7fd6600faf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:03.673 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:03.670+0000 7fd664935700 1 --2- 172.21.15.3:0/3740144815 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6600faae0 0x7fd6600faf00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fd65400b5f0 tx=0x7fd654006ee0).stop 2023-11-17T05:25:03.674 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:03.670+0000 7fd664935700 1 -- 172.21.15.3:0/3740144815 shutdown_connections 2023-11-17T05:25:03.674 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:03.670+0000 7fd664935700 1 --2- 172.21.15.3:0/3740144815 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6600faae0 0x7fd6600faf00 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:03.674 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:03.670+0000 7fd664935700 1 -- 172.21.15.3:0/3740144815 >> 172.21.15.3:0/3740144815 conn(0x7fd6600f6280 msgr2=0x7fd6600f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:03.674 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:03.670+0000 7fd664935700 1 -- 172.21.15.3:0/3740144815 shutdown_connections 2023-11-17T05:25:03.674 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:03.670+0000 7fd664935700 1 -- 172.21.15.3:0/3740144815 wait complete. 2023-11-17T05:25:03.674 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:03.671+0000 7fd664935700 1 Processor -- start 2023-11-17T05:25:03.674 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:03.671+0000 7fd664935700 1 -- start start 2023-11-17T05:25:03.674 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:03.671+0000 7fd664935700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6600faae0 0x7fd660100df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:03.674 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:03.671+0000 7fd664935700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6600ff270 con 0x7fd6600faae0 2023-11-17T05:25:03.674 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:03.671+0000 7fd65e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6600faae0 0x7fd660100df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:03.674 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:03.671+0000 7fd65e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6600faae0 0x7fd660100df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35458/0 (socket says 172.21.15.3:35458) 2023-11-17T05:25:03.674 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:03.671+0000 7fd65e59c700 1 -- 172.21.15.3:0/1417506963 learned_addr learned my addr 172.21.15.3:0/1417506963 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:25:03.675 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:03.672+0000 7fd65e59c700 1 -- 172.21.15.3:0/1417506963 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd65400b200 con 0x7fd6600faae0 2023-11-17T05:25:03.675 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:03.672+0000 7fd65e59c700 1 --2- 172.21.15.3:0/1417506963 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6600faae0 0x7fd660100df0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fd654002f40 tx=0x7fd654006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:03.676 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:03.672+0000 7fd64effd700 1 -- 172.21.15.3:0/1417506963 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd654011690 con 0x7fd6600faae0 2023-11-17T05:25:03.676 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:03.672+0000 7fd64effd700 1 -- 172.21.15.3:0/1417506963 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd65401b880 con 0x7fd6600faae0 2023-11-17T05:25:03.676 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:03.672+0000 7fd664935700 1 -- 172.21.15.3:0/1417506963 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6600ff410 con 0x7fd6600faae0 2023-11-17T05:25:03.676 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:03.672+0000 7fd64effd700 1 -- 172.21.15.3:0/1417506963 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd65401a6b0 con 0x7fd6600faae0 2023-11-17T05:25:03.676 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:03.672+0000 7fd664935700 1 -- 172.21.15.3:0/1417506963 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6600ff820 con 0x7fd6600faae0 2023-11-17T05:25:03.676 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:03.673+0000 7fd64effd700 1 -- 172.21.15.3:0/1417506963 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fd65400f070 con 0x7fd6600faae0 2023-11-17T05:25:03.676 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:03.673+0000 7fd64effd700 1 --2- 172.21.15.3:0/1417506963 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7fd64402fd50 0x7fd644032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:03.676 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:03.673+0000 7fd64effd700 1 -- 172.21.15.3:0/1417506963 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fd654041140 con 0x7fd6600faae0 2023-11-17T05:25:03.676 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:03.673+0000 7fd65dd9b700 1 --2- 172.21.15.3:0/1417506963 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7fd64402fd50 0x7fd644032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:03.676 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:03.673+0000 7fd664935700 1 -- 172.21.15.3:0/1417506963 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd660059100 con 0x7fd6600faae0 2023-11-17T05:25:03.676 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:03.674+0000 7fd65dd9b700 1 --2- 172.21.15.3:0/1417506963 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7fd64402fd50 0x7fd644032210 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fd6500096a0 tx=0x7fd650006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:03.679 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:03.678+0000 7fd64effd700 1 -- 172.21.15.3:0/1417506963 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd654016070 con 0x7fd6600faae0 2023-11-17T05:25:03.870 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:03.868+0000 7fd664935700 1 -- 172.21.15.3:0/1417506963 --> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi003", "addr": "172.21.15.3", "target": ["mon-mgr", ""]}) v1 -- 0x7fd6600fc930 con 0x7fd64402fd50 2023-11-17T05:25:04.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:04 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:02.933750+0000 mgr.smithi003.eddcuc (mgr.14116) 7 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:25:04.514 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: Added host 'smithi003' 2023-11-17T05:25:04.514 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:04.511+0000 7fd64effd700 1 -- 172.21.15.3:0/1417506963 <== mgr.14116 v2:172.21.15.3:6800/2575681100 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7fd6600fc930 con 0x7fd64402fd50 2023-11-17T05:25:04.516 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:04.514+0000 7fd664935700 1 -- 172.21.15.3:0/1417506963 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7fd64402fd50 msgr2=0x7fd644032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:04.516 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:04.514+0000 7fd664935700 1 --2- 172.21.15.3:0/1417506963 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7fd64402fd50 0x7fd644032210 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fd6500096a0 tx=0x7fd650006b40).stop 2023-11-17T05:25:04.517 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:04.514+0000 7fd664935700 1 -- 172.21.15.3:0/1417506963 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6600faae0 msgr2=0x7fd660100df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:04.517 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:04.514+0000 7fd664935700 1 --2- 172.21.15.3:0/1417506963 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6600faae0 0x7fd660100df0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fd654002f40 tx=0x7fd654006ee0).stop 2023-11-17T05:25:04.517 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:04.514+0000 7fd664935700 1 -- 172.21.15.3:0/1417506963 shutdown_connections 2023-11-17T05:25:04.517 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:04.514+0000 7fd664935700 1 --2- 172.21.15.3:0/1417506963 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7fd64402fd50 0x7fd644032210 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:04.517 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:04.514+0000 7fd664935700 1 --2- 172.21.15.3:0/1417506963 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6600faae0 0x7fd660100df0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:04.517 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:04.514+0000 7fd664935700 1 -- 172.21.15.3:0/1417506963 >> 172.21.15.3:0/1417506963 conn(0x7fd6600f6280 msgr2=0x7fd66006f890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:04.517 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:04.514+0000 7fd664935700 1 -- 172.21.15.3:0/1417506963 shutdown_connections 2023-11-17T05:25:04.517 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:04.515+0000 7fd664935700 1 -- 172.21.15.3:0/1417506963 wait complete. 2023-11-17T05:25:04.826 INFO:teuthology.orchestra.run.smithi003.stderr:Deploying mon service with default placement... 2023-11-17T05:25:04.826 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptcuja8tf:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpk7o1jie2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2023-11-17T05:25:05.213 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.209+0000 7f8778380700 1 Processor -- start 2023-11-17T05:25:05.213 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.209+0000 7f8778380700 1 -- start start 2023-11-17T05:25:05.213 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.210+0000 7f8778380700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f877006fc50 0x7f8770100c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:05.213 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.210+0000 7f8778380700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87700fc6a0 con 0x7f877006fc50 2023-11-17T05:25:05.214 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.210+0000 7f877611c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f877006fc50 0x7f8770100c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:05.214 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.210+0000 7f877611c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f877006fc50 0x7f8770100c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35470/0 (socket says 172.21.15.3:35470) 2023-11-17T05:25:05.214 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.210+0000 7f877611c700 1 -- 172.21.15.3:0/2640433701 learned_addr learned my addr 172.21.15.3:0/2640433701 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:25:05.214 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.210+0000 7f877611c700 1 -- 172.21.15.3:0/2640433701 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8770101110 con 0x7f877006fc50 2023-11-17T05:25:05.214 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.210+0000 7f877611c700 1 --2- 172.21.15.3:0/2640433701 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f877006fc50 0x7f8770100c20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f876c00b5f0 tx=0x7f876c006ee0).ready entity=mon.0 client_cookie=46aeff03fb161242 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:05.214 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.211+0000 7f877511a700 1 -- 172.21.15.3:0/2640433701 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f876c00d600 con 0x7f877006fc50 2023-11-17T05:25:05.214 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.211+0000 7f877511a700 1 -- 172.21.15.3:0/2640433701 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f876c00dbd0 con 0x7f877006fc50 2023-11-17T05:25:05.214 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.211+0000 7f877511a700 1 -- 172.21.15.3:0/2640433701 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f876c011c00 con 0x7f877006fc50 2023-11-17T05:25:05.214 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.211+0000 7f8778380700 1 -- 172.21.15.3:0/2640433701 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f877006fc50 msgr2=0x7f8770100c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:05.214 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.211+0000 7f8778380700 1 --2- 172.21.15.3:0/2640433701 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f877006fc50 0x7f8770100c20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f876c00b5f0 tx=0x7f876c006ee0).stop 2023-11-17T05:25:05.214 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.211+0000 7f8778380700 1 -- 172.21.15.3:0/2640433701 shutdown_connections 2023-11-17T05:25:05.215 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.211+0000 7f8778380700 1 --2- 172.21.15.3:0/2640433701 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f877006fc50 0x7f8770100c20 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:05.215 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.211+0000 7f8778380700 1 -- 172.21.15.3:0/2640433701 >> 172.21.15.3:0/2640433701 conn(0x7f87700f6280 msgr2=0x7f87700f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:05.215 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.211+0000 7f8778380700 1 -- 172.21.15.3:0/2640433701 shutdown_connections 2023-11-17T05:25:05.215 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.211+0000 7f8778380700 1 -- 172.21.15.3:0/2640433701 wait complete. 2023-11-17T05:25:05.215 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.212+0000 7f8778380700 1 Processor -- start 2023-11-17T05:25:05.215 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.212+0000 7f8778380700 1 -- start start 2023-11-17T05:25:05.217 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.212+0000 7f8778380700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f877006fc50 0x7f877010ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:05.217 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.212+0000 7f8778380700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f877010d1b0 con 0x7f877006fc50 2023-11-17T05:25:05.217 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.212+0000 7f877611c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f877006fc50 0x7f877010ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:05.217 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.212+0000 7f877611c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f877006fc50 0x7f877010ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35472/0 (socket says 172.21.15.3:35472) 2023-11-17T05:25:05.217 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.212+0000 7f877611c700 1 -- 172.21.15.3:0/2387815131 learned_addr learned my addr 172.21.15.3:0/2387815131 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:25:05.217 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.213+0000 7f877611c700 1 -- 172.21.15.3:0/2387815131 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f876c00b200 con 0x7f877006fc50 2023-11-17T05:25:05.217 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.213+0000 7f877611c700 1 --2- 172.21.15.3:0/2387815131 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f877006fc50 0x7f877010ccc0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f876c006c70 tx=0x7f876c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:05.217 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.213+0000 7f875effd700 1 -- 172.21.15.3:0/2387815131 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f876c011690 con 0x7f877006fc50 2023-11-17T05:25:05.217 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.213+0000 7f8778380700 1 -- 172.21.15.3:0/2387815131 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f877010d3b0 con 0x7f877006fc50 2023-11-17T05:25:05.218 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.213+0000 7f875effd700 1 -- 172.21.15.3:0/2387815131 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f876c01b880 con 0x7f877006fc50 2023-11-17T05:25:05.218 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.213+0000 7f875effd700 1 -- 172.21.15.3:0/2387815131 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f876c01a650 con 0x7f877006fc50 2023-11-17T05:25:05.218 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.213+0000 7f8778380700 1 -- 172.21.15.3:0/2387815131 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f877010d7c0 con 0x7f877006fc50 2023-11-17T05:25:05.218 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.214+0000 7f875effd700 1 -- 172.21.15.3:0/2387815131 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f876c00f070 con 0x7f877006fc50 2023-11-17T05:25:05.218 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.214+0000 7f875effd700 1 --2- 172.21.15.3:0/2387815131 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f875802fd50 0x7f8758032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:05.218 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.214+0000 7f875effd700 1 -- 172.21.15.3:0/2387815131 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f876c041230 con 0x7f877006fc50 2023-11-17T05:25:05.218 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.214+0000 7f8778380700 1 -- 172.21.15.3:0/2387815131 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8770045ad0 con 0x7f877006fc50 2023-11-17T05:25:05.218 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.214+0000 7f877591b700 1 --2- 172.21.15.3:0/2387815131 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f875802fd50 0x7f8758032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:05.218 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.215+0000 7f877591b700 1 --2- 172.21.15.3:0/2387815131 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f875802fd50 0x7f8758032210 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f87600096a0 tx=0x7f8760006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:05.220 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.219+0000 7f875effd700 1 -- 172.21.15.3:0/2387815131 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f876c016070 con 0x7f877006fc50 2023-11-17T05:25:05.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:05 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:03.870722+0000 mgr.smithi003.eddcuc (mgr.14116) 8 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi003", "addr": "172.21.15.3", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:25:05.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:05 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:04.512250+0000 mon.smithi003 (mon.0) 55 : audit [INF] from='mgr.14116 172.21.15.3:0/2120313347' entity='mgr.smithi003.eddcuc' 2023-11-17T05:25:05.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:05 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:04.512898+0000 mon.smithi003 (mon.0) 56 : audit [DBG] from='mgr.14116 172.21.15.3:0/2120313347' entity='mgr.smithi003.eddcuc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:25:05.415 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.414+0000 7f8778380700 1 -- 172.21.15.3:0/2387815131 --> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f877010ecf0 con 0x7f875802fd50 2023-11-17T05:25:05.423 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.418+0000 7f875effd700 1 -- 172.21.15.3:0/2387815131 <== mgr.14116 v2:172.21.15.3:6800/2575681100 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f877010ecf0 con 0x7f875802fd50 2023-11-17T05:25:05.423 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: Scheduled mon update... 2023-11-17T05:25:05.423 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.420+0000 7f8778380700 1 -- 172.21.15.3:0/2387815131 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f875802fd50 msgr2=0x7f8758032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:05.424 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.420+0000 7f8778380700 1 --2- 172.21.15.3:0/2387815131 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f875802fd50 0x7f8758032210 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f87600096a0 tx=0x7f8760006b40).stop 2023-11-17T05:25:05.424 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.420+0000 7f8778380700 1 -- 172.21.15.3:0/2387815131 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f877006fc50 msgr2=0x7f877010ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:05.424 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.420+0000 7f8778380700 1 --2- 172.21.15.3:0/2387815131 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f877006fc50 0x7f877010ccc0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f876c006c70 tx=0x7f876c006ee0).stop 2023-11-17T05:25:05.424 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.420+0000 7f8778380700 1 -- 172.21.15.3:0/2387815131 shutdown_connections 2023-11-17T05:25:05.424 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.420+0000 7f8778380700 1 --2- 172.21.15.3:0/2387815131 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f875802fd50 0x7f8758032210 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:05.424 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.420+0000 7f8778380700 1 --2- 172.21.15.3:0/2387815131 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f877006fc50 0x7f877010ccc0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:05.424 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.420+0000 7f8778380700 1 -- 172.21.15.3:0/2387815131 >> 172.21.15.3:0/2387815131 conn(0x7f87700f6280 msgr2=0x7f87700f7e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:05.424 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.420+0000 7f8778380700 1 -- 172.21.15.3:0/2387815131 shutdown_connections 2023-11-17T05:25:05.424 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:05.420+0000 7f8778380700 1 -- 172.21.15.3:0/2387815131 wait complete. 2023-11-17T05:25:05.771 INFO:teuthology.orchestra.run.smithi003.stderr:Deploying mgr service with default placement... 2023-11-17T05:25:05.771 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptcuja8tf:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpk7o1jie2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2023-11-17T05:25:06.173 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.167+0000 7f9adb728700 1 Processor -- start 2023-11-17T05:25:06.173 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.167+0000 7f9adb728700 1 -- start start 2023-11-17T05:25:06.173 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.167+0000 7f9adb728700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ad406fc90 0x7f9ad40fe7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:06.173 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.167+0000 7f9adb728700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad40fed60 con 0x7f9ad406fc90 2023-11-17T05:25:06.173 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.167+0000 7f9ad94c4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ad406fc90 0x7f9ad40fe7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:06.173 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.167+0000 7f9ad94c4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ad406fc90 0x7f9ad40fe7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35480/0 (socket says 172.21.15.3:35480) 2023-11-17T05:25:06.173 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.168+0000 7f9ad94c4700 1 -- 172.21.15.3:0/1323592439 learned_addr learned my addr 172.21.15.3:0/1323592439 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:25:06.173 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.168+0000 7f9ad94c4700 1 -- 172.21.15.3:0/1323592439 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad40feea0 con 0x7f9ad406fc90 2023-11-17T05:25:06.174 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.168+0000 7f9ad94c4700 1 --2- 172.21.15.3:0/1323592439 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ad406fc90 0x7f9ad40fe7e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f9ad000b5f0 tx=0x7f9ad0006ee0).ready entity=mon.0 client_cookie=fef077febeee045a server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:06.174 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.168+0000 7f9ac3fff700 1 -- 172.21.15.3:0/1323592439 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ad000d600 con 0x7f9ad406fc90 2023-11-17T05:25:06.174 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.168+0000 7f9ac3fff700 1 -- 172.21.15.3:0/1323592439 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9ad000dbd0 con 0x7f9ad406fc90 2023-11-17T05:25:06.175 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.168+0000 7f9ac3fff700 1 -- 172.21.15.3:0/1323592439 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ad0011c00 con 0x7f9ad406fc90 2023-11-17T05:25:06.175 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.169+0000 7f9adb728700 1 -- 172.21.15.3:0/1323592439 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ad406fc90 msgr2=0x7f9ad40fe7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:06.175 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.169+0000 7f9adb728700 1 --2- 172.21.15.3:0/1323592439 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ad406fc90 0x7f9ad40fe7e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f9ad000b5f0 tx=0x7f9ad0006ee0).stop 2023-11-17T05:25:06.176 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.169+0000 7f9adb728700 1 -- 172.21.15.3:0/1323592439 shutdown_connections 2023-11-17T05:25:06.176 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.169+0000 7f9adb728700 1 --2- 172.21.15.3:0/1323592439 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ad406fc90 0x7f9ad40fe7e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:06.176 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.169+0000 7f9adb728700 1 -- 172.21.15.3:0/1323592439 >> 172.21.15.3:0/1323592439 conn(0x7f9ad40f6280 msgr2=0x7f9ad40f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:06.176 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.169+0000 7f9adb728700 1 -- 172.21.15.3:0/1323592439 shutdown_connections 2023-11-17T05:25:06.176 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.169+0000 7f9adb728700 1 -- 172.21.15.3:0/1323592439 wait complete. 2023-11-17T05:25:06.176 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.170+0000 7f9adb728700 1 Processor -- start 2023-11-17T05:25:06.176 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.170+0000 7f9adb728700 1 -- start start 2023-11-17T05:25:06.176 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.170+0000 7f9adb728700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ad406fc90 0x7f9ad40fc260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:06.176 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.170+0000 7f9adb728700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad40fa6e0 con 0x7f9ad406fc90 2023-11-17T05:25:06.176 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.170+0000 7f9ad94c4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ad406fc90 0x7f9ad40fc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:06.177 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.170+0000 7f9ad94c4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ad406fc90 0x7f9ad40fc260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35488/0 (socket says 172.21.15.3:35488) 2023-11-17T05:25:06.177 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.170+0000 7f9ad94c4700 1 -- 172.21.15.3:0/3896167169 learned_addr learned my addr 172.21.15.3:0/3896167169 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:25:06.177 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.171+0000 7f9ad94c4700 1 -- 172.21.15.3:0/3896167169 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad000b200 con 0x7f9ad406fc90 2023-11-17T05:25:06.177 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.171+0000 7f9ad94c4700 1 --2- 172.21.15.3:0/3896167169 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ad406fc90 0x7f9ad40fc260 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f9ad000e040 tx=0x7f9ad0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:06.177 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.171+0000 7f9ac1ffb700 1 -- 172.21.15.3:0/3896167169 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ad001b410 con 0x7f9ad406fc90 2023-11-17T05:25:06.177 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.171+0000 7f9adb728700 1 -- 172.21.15.3:0/3896167169 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ad40fa880 con 0x7f9ad406fc90 2023-11-17T05:25:06.177 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.171+0000 7f9ac1ffb700 1 -- 172.21.15.3:0/3896167169 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9ad001b9e0 con 0x7f9ad406fc90 2023-11-17T05:25:06.177 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.171+0000 7f9ac1ffb700 1 -- 172.21.15.3:0/3896167169 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ad001a770 con 0x7f9ad406fc90 2023-11-17T05:25:06.178 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.171+0000 7f9adb728700 1 -- 172.21.15.3:0/3896167169 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ad40fac90 con 0x7f9ad406fc90 2023-11-17T05:25:06.178 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.172+0000 7f9ac1ffb700 1 -- 172.21.15.3:0/3896167169 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f9ad000f070 con 0x7f9ad406fc90 2023-11-17T05:25:06.178 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.172+0000 7f9ac1ffb700 1 --2- 172.21.15.3:0/3896167169 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f9abc02fd50 0x7f9abc032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:06.178 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.172+0000 7f9ac1ffb700 1 -- 172.21.15.3:0/3896167169 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f9ad0041040 con 0x7f9ad406fc90 2023-11-17T05:25:06.178 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.172+0000 7f9ad8cc3700 1 --2- 172.21.15.3:0/3896167169 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f9abc02fd50 0x7f9abc032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:06.178 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.173+0000 7f9ad8cc3700 1 --2- 172.21.15.3:0/3896167169 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f9abc02fd50 0x7f9abc032210 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f9ac40096a0 tx=0x7f9ac4006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:06.178 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.173+0000 7f9adb728700 1 -- 172.21.15.3:0/3896167169 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ad4059100 con 0x7f9ad406fc90 2023-11-17T05:25:06.178 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.177+0000 7f9ac1ffb700 1 -- 172.21.15.3:0/3896167169 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9ad0016070 con 0x7f9ad406fc90 2023-11-17T05:25:06.378 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.375+0000 7f9adb728700 1 -- 172.21.15.3:0/3896167169 --> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ad40ffc40 con 0x7f9abc02fd50 2023-11-17T05:25:06.381 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.379+0000 7f9ac1ffb700 1 -- 172.21.15.3:0/3896167169 <== mgr.14116 v2:172.21.15.3:6800/2575681100 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f9ad40ffc40 con 0x7f9abc02fd50 2023-11-17T05:25:06.381 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: Scheduled mgr update... 2023-11-17T05:25:06.383 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.381+0000 7f9adb728700 1 -- 172.21.15.3:0/3896167169 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f9abc02fd50 msgr2=0x7f9abc032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:06.384 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.381+0000 7f9adb728700 1 --2- 172.21.15.3:0/3896167169 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f9abc02fd50 0x7f9abc032210 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f9ac40096a0 tx=0x7f9ac4006b40).stop 2023-11-17T05:25:06.384 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.381+0000 7f9adb728700 1 -- 172.21.15.3:0/3896167169 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ad406fc90 msgr2=0x7f9ad40fc260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:06.384 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.381+0000 7f9adb728700 1 --2- 172.21.15.3:0/3896167169 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ad406fc90 0x7f9ad40fc260 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f9ad000e040 tx=0x7f9ad0006ee0).stop 2023-11-17T05:25:06.384 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.381+0000 7f9adb728700 1 -- 172.21.15.3:0/3896167169 shutdown_connections 2023-11-17T05:25:06.384 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.381+0000 7f9adb728700 1 --2- 172.21.15.3:0/3896167169 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f9abc02fd50 0x7f9abc032210 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:06.384 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.381+0000 7f9adb728700 1 --2- 172.21.15.3:0/3896167169 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ad406fc90 0x7f9ad40fc260 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:06.384 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.381+0000 7f9adb728700 1 -- 172.21.15.3:0/3896167169 >> 172.21.15.3:0/3896167169 conn(0x7f9ad40f6280 msgr2=0x7f9ad40f7e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:06.384 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.382+0000 7f9adb728700 1 -- 172.21.15.3:0/3896167169 shutdown_connections 2023-11-17T05:25:06.385 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:06.382+0000 7f9adb728700 1 -- 172.21.15.3:0/3896167169 wait complete. 2023-11-17T05:25:06.415 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:06 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cephadm 2023-11-17T05:25:04.512622+0000 mgr.smithi003.eddcuc (mgr.14116) 9 : cephadm [INF] Added host smithi003 2023-11-17T05:25:06.415 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:06 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:05.419845+0000 mon.smithi003 (mon.0) 57 : audit [INF] from='mgr.14116 172.21.15.3:0/2120313347' entity='mgr.smithi003.eddcuc' 2023-11-17T05:25:06.828 INFO:teuthology.orchestra.run.smithi003.stderr:Deploying crash service with default placement... 2023-11-17T05:25:06.828 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptcuja8tf:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpk7o1jie2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2023-11-17T05:25:07.222 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.217+0000 7f740fc29700 1 Processor -- start 2023-11-17T05:25:07.222 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.217+0000 7f740fc29700 1 -- start start 2023-11-17T05:25:07.222 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.217+0000 7f740fc29700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74080faae0 0x7f74080faf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:07.222 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.217+0000 7f740fc29700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74080fb480 con 0x7f74080faae0 2023-11-17T05:25:07.222 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.217+0000 7f740d9c5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74080faae0 0x7f74080faf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:07.222 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.217+0000 7f740d9c5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74080faae0 0x7f74080faf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35504/0 (socket says 172.21.15.3:35504) 2023-11-17T05:25:07.223 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.217+0000 7f740d9c5700 1 -- 172.21.15.3:0/1964022847 learned_addr learned my addr 172.21.15.3:0/1964022847 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:25:07.223 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.218+0000 7f740d9c5700 1 -- 172.21.15.3:0/1964022847 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74080fb5c0 con 0x7f74080faae0 2023-11-17T05:25:07.223 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.218+0000 7f740d9c5700 1 --2- 172.21.15.3:0/1964022847 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74080faae0 0x7f74080faf00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f740400b5f0 tx=0x7f7404006ee0).ready entity=mon.0 client_cookie=111ed068378dc946 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:07.223 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.218+0000 7f740c9c3700 1 -- 172.21.15.3:0/1964022847 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f740400d600 con 0x7f74080faae0 2023-11-17T05:25:07.223 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.218+0000 7f740c9c3700 1 -- 172.21.15.3:0/1964022847 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f740400dbd0 con 0x7f74080faae0 2023-11-17T05:25:07.223 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.218+0000 7f740c9c3700 1 -- 172.21.15.3:0/1964022847 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7404011c00 con 0x7f74080faae0 2023-11-17T05:25:07.223 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.219+0000 7f740fc29700 1 -- 172.21.15.3:0/1964022847 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74080faae0 msgr2=0x7f74080faf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:07.224 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.219+0000 7f740fc29700 1 --2- 172.21.15.3:0/1964022847 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74080faae0 0x7f74080faf00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f740400b5f0 tx=0x7f7404006ee0).stop 2023-11-17T05:25:07.224 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.219+0000 7f740fc29700 1 -- 172.21.15.3:0/1964022847 shutdown_connections 2023-11-17T05:25:07.224 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.219+0000 7f740fc29700 1 --2- 172.21.15.3:0/1964022847 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74080faae0 0x7f74080faf00 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:07.224 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.219+0000 7f740fc29700 1 -- 172.21.15.3:0/1964022847 >> 172.21.15.3:0/1964022847 conn(0x7f74080f6280 msgr2=0x7f74080f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:07.224 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.219+0000 7f740fc29700 1 -- 172.21.15.3:0/1964022847 shutdown_connections 2023-11-17T05:25:07.224 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.219+0000 7f740fc29700 1 -- 172.21.15.3:0/1964022847 wait complete. 2023-11-17T05:25:07.224 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.220+0000 7f740fc29700 1 Processor -- start 2023-11-17T05:25:07.225 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.220+0000 7f740fc29700 1 -- start start 2023-11-17T05:25:07.225 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.220+0000 7f740fc29700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74080faae0 0x7f740810d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:07.225 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.220+0000 7f740fc29700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f740806fc50 con 0x7f74080faae0 2023-11-17T05:25:07.225 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.220+0000 7f740d9c5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74080faae0 0x7f740810d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:07.225 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.220+0000 7f740d9c5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74080faae0 0x7f740810d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35514/0 (socket says 172.21.15.3:35514) 2023-11-17T05:25:07.225 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.220+0000 7f740d9c5700 1 -- 172.21.15.3:0/3328144882 learned_addr learned my addr 172.21.15.3:0/3328144882 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:25:07.225 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.220+0000 7f740d9c5700 1 -- 172.21.15.3:0/3328144882 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f740400b200 con 0x7f74080faae0 2023-11-17T05:25:07.226 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.221+0000 7f740d9c5700 1 --2- 172.21.15.3:0/3328144882 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74080faae0 0x7f740810d6f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f740400b1d0 tx=0x7f7404006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:07.226 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.221+0000 7f73f67fc700 1 -- 172.21.15.3:0/3328144882 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7404011a20 con 0x7f74080faae0 2023-11-17T05:25:07.226 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.221+0000 7f73f67fc700 1 -- 172.21.15.3:0/3328144882 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f740401b880 con 0x7f74080faae0 2023-11-17T05:25:07.226 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.221+0000 7f740fc29700 1 -- 172.21.15.3:0/3328144882 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f740806fe50 con 0x7f74080faae0 2023-11-17T05:25:07.226 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.221+0000 7f73f67fc700 1 -- 172.21.15.3:0/3328144882 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f740401a700 con 0x7f74080faae0 2023-11-17T05:25:07.226 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.221+0000 7f740fc29700 1 -- 172.21.15.3:0/3328144882 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f740810dda0 con 0x7f74080faae0 2023-11-17T05:25:07.227 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.222+0000 7f73f67fc700 1 -- 172.21.15.3:0/3328144882 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f740400f070 con 0x7f74080faae0 2023-11-17T05:25:07.227 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.222+0000 7f740fc29700 1 -- 172.21.15.3:0/3328144882 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7408045ad0 con 0x7f74080faae0 2023-11-17T05:25:07.227 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.222+0000 7f73f67fc700 1 --2- 172.21.15.3:0/3328144882 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f73f002fdb0 0x7f73f0032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:07.227 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.222+0000 7f73f67fc700 1 -- 172.21.15.3:0/3328144882 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f7404041190 con 0x7f74080faae0 2023-11-17T05:25:07.227 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.222+0000 7f740d1c4700 1 --2- 172.21.15.3:0/3328144882 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f73f002fdb0 0x7f73f0032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:07.227 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.223+0000 7f740d1c4700 1 --2- 172.21.15.3:0/3328144882 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f73f002fdb0 0x7f73f0032270 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f73f80096a0 tx=0x7f73f8006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:07.228 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.227+0000 7f73f67fc700 1 -- 172.21.15.3:0/3328144882 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7404022310 con 0x7f74080faae0 2023-11-17T05:25:07.271 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:07 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:05.416444+0000 mgr.smithi003.eddcuc (mgr.14116) 10 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:25:07.271 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:07 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cephadm 2023-11-17T05:25:05.417244+0000 mgr.smithi003.eddcuc (mgr.14116) 11 : cephadm [INF] Saving service mon spec with placement count:5 2023-11-17T05:25:07.271 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:07 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:06.380631+0000 mon.smithi003 (mon.0) 58 : audit [INF] from='mgr.14116 172.21.15.3:0/2120313347' entity='mgr.smithi003.eddcuc' 2023-11-17T05:25:07.423 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.421+0000 7f740fc29700 1 -- 172.21.15.3:0/3328144882 --> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f74080fca30 con 0x7f73f002fdb0 2023-11-17T05:25:07.427 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.426+0000 7f73f67fc700 1 -- 172.21.15.3:0/3328144882 <== mgr.14116 v2:172.21.15.3:6800/2575681100 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f74080fca30 con 0x7f73f002fdb0 2023-11-17T05:25:07.427 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: Scheduled crash update... 2023-11-17T05:25:07.430 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.428+0000 7f740fc29700 1 -- 172.21.15.3:0/3328144882 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f73f002fdb0 msgr2=0x7f73f0032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:07.430 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.428+0000 7f740fc29700 1 --2- 172.21.15.3:0/3328144882 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f73f002fdb0 0x7f73f0032270 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f73f80096a0 tx=0x7f73f8006b40).stop 2023-11-17T05:25:07.430 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.428+0000 7f740fc29700 1 -- 172.21.15.3:0/3328144882 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74080faae0 msgr2=0x7f740810d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:07.430 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.428+0000 7f740fc29700 1 --2- 172.21.15.3:0/3328144882 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74080faae0 0x7f740810d6f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f740400b1d0 tx=0x7f7404006ee0).stop 2023-11-17T05:25:07.430 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.428+0000 7f740fc29700 1 -- 172.21.15.3:0/3328144882 shutdown_connections 2023-11-17T05:25:07.430 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.428+0000 7f740fc29700 1 --2- 172.21.15.3:0/3328144882 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f73f002fdb0 0x7f73f0032270 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:07.430 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.428+0000 7f740fc29700 1 --2- 172.21.15.3:0/3328144882 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f74080faae0 0x7f740810d6f0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:07.431 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.428+0000 7f740fc29700 1 -- 172.21.15.3:0/3328144882 >> 172.21.15.3:0/3328144882 conn(0x7f74080f6280 msgr2=0x7f74080f7e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:07.431 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.428+0000 7f740fc29700 1 -- 172.21.15.3:0/3328144882 shutdown_connections 2023-11-17T05:25:07.431 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:07.428+0000 7f740fc29700 1 -- 172.21.15.3:0/3328144882 wait complete. 2023-11-17T05:25:07.778 INFO:teuthology.orchestra.run.smithi003.stderr:Enabling mgr prometheus module... 2023-11-17T05:25:07.778 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptcuja8tf:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpk7o1jie2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2023-11-17T05:25:08.168 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:08.166+0000 7f1d347f4700 1 Processor -- start 2023-11-17T05:25:08.168 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:08.166+0000 7f1d347f4700 1 -- start start 2023-11-17T05:25:08.168 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:08.167+0000 7f1d347f4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d2c0fc170 0x7f1d2c0fc590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:08.170 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:08.167+0000 7f1d347f4700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2c0fcb10 con 0x7f1d2c0fc170 2023-11-17T05:25:08.170 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:08.167+0000 7f1d32590700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d2c0fc170 0x7f1d2c0fc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:08.170 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:08.167+0000 7f1d32590700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d2c0fc170 0x7f1d2c0fc590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35524/0 (socket says 172.21.15.3:35524) 2023-11-17T05:25:08.170 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:08.167+0000 7f1d32590700 1 -- 172.21.15.3:0/1822091871 learned_addr learned my addr 172.21.15.3:0/1822091871 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:25:08.170 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:08.167+0000 7f1d32590700 1 -- 172.21.15.3:0/1822091871 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d2c0fcc50 con 0x7f1d2c0fc170 2023-11-17T05:25:08.170 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:08.167+0000 7f1d32590700 1 --2- 172.21.15.3:0/1822091871 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d2c0fc170 0x7f1d2c0fc590 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f1d2800b5f0 tx=0x7f1d28006ee0).ready entity=mon.0 client_cookie=782085645daac004 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:08.171 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:08.168+0000 7f1d3158e700 1 -- 172.21.15.3:0/1822091871 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d2800d600 con 0x7f1d2c0fc170 2023-11-17T05:25:08.171 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:08.168+0000 7f1d3158e700 1 -- 172.21.15.3:0/1822091871 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1d2800dbd0 con 0x7f1d2c0fc170 2023-11-17T05:25:08.171 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:08.168+0000 7f1d3158e700 1 -- 172.21.15.3:0/1822091871 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d28011c00 con 0x7f1d2c0fc170 2023-11-17T05:25:08.171 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:08.168+0000 7f1d347f4700 1 -- 172.21.15.3:0/1822091871 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d2c0fc170 msgr2=0x7f1d2c0fc590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:08.171 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:08.168+0000 7f1d347f4700 1 --2- 172.21.15.3:0/1822091871 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d2c0fc170 0x7f1d2c0fc590 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f1d2800b5f0 tx=0x7f1d28006ee0).stop 2023-11-17T05:25:08.171 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:08.168+0000 7f1d347f4700 1 -- 172.21.15.3:0/1822091871 shutdown_connections 2023-11-17T05:25:08.171 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:08.168+0000 7f1d347f4700 1 --2- 172.21.15.3:0/1822091871 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d2c0fc170 0x7f1d2c0fc590 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:08.171 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:08.168+0000 7f1d347f4700 1 -- 172.21.15.3:0/1822091871 >> 172.21.15.3:0/1822091871 conn(0x7f1d2c0f59e0 msgr2=0x7f1d2c0f7e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:08.171 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:08.168+0000 7f1d347f4700 1 -- 172.21.15.3:0/1822091871 shutdown_connections 2023-11-17T05:25:08.171 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:08.168+0000 7f1d347f4700 1 -- 172.21.15.3:0/1822091871 wait complete. 2023-11-17T05:25:08.172 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:08.169+0000 7f1d347f4700 1 Processor -- start 2023-11-17T05:25:08.172 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:08.169+0000 7f1d347f4700 1 -- start start 2023-11-17T05:25:08.172 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:08.169+0000 7f1d347f4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d2c0fc170 0x7f1d2c10cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:08.172 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:08.169+0000 7f1d347f4700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2c10d0c0 con 0x7f1d2c0fc170 2023-11-17T05:25:08.172 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:08.169+0000 7f1d32590700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d2c0fc170 0x7f1d2c10cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:08.172 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:08.169+0000 7f1d32590700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d2c0fc170 0x7f1d2c10cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35528/0 (socket says 172.21.15.3:35528) 2023-11-17T05:25:08.173 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:08.169+0000 7f1d32590700 1 -- 172.21.15.3:0/4127090098 learned_addr learned my addr 172.21.15.3:0/4127090098 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:25:08.173 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:08.170+0000 7f1d32590700 1 -- 172.21.15.3:0/4127090098 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d2800b200 con 0x7f1d2c0fc170 2023-11-17T05:25:08.173 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:08.170+0000 7f1d32590700 1 --2- 172.21.15.3:0/4127090098 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d2c0fc170 0x7f1d2c10cbd0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f1d2800e040 tx=0x7f1d28006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:08.174 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:08.170+0000 7f1d1affd700 1 -- 172.21.15.3:0/4127090098 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d2801b410 con 0x7f1d2c0fc170 2023-11-17T05:25:08.174 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:08.170+0000 7f1d1affd700 1 -- 172.21.15.3:0/4127090098 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1d2801b9e0 con 0x7f1d2c0fc170 2023-11-17T05:25:08.174 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:08.170+0000 7f1d347f4700 1 -- 172.21.15.3:0/4127090098 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d2c10d2c0 con 0x7f1d2c0fc170 2023-11-17T05:25:08.174 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:08.170+0000 7f1d1affd700 1 -- 172.21.15.3:0/4127090098 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d2801a770 con 0x7f1d2c0fc170 2023-11-17T05:25:08.174 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:08.170+0000 7f1d347f4700 1 -- 172.21.15.3:0/4127090098 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d2c10d6d0 con 0x7f1d2c0fc170 2023-11-17T05:25:08.174 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:08.171+0000 7f1d1affd700 1 -- 172.21.15.3:0/4127090098 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f1d2800f070 con 0x7f1d2c0fc170 2023-11-17T05:25:08.174 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:08.171+0000 7f1d1affd700 1 --2- 172.21.15.3:0/4127090098 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f1d1402fdb0 0x7f1d14032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:08.174 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:08.171+0000 7f1d1affd700 1 -- 172.21.15.3:0/4127090098 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f1d28040fe0 con 0x7f1d2c0fc170 2023-11-17T05:25:08.174 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:08.172+0000 7f1d31d8f700 1 --2- 172.21.15.3:0/4127090098 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f1d1402fdb0 0x7f1d14032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:08.174 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:08.172+0000 7f1d347f4700 1 -- 172.21.15.3:0/4127090098 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d2c059100 con 0x7f1d2c0fc170 2023-11-17T05:25:08.174 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:08.172+0000 7f1d31d8f700 1 --2- 172.21.15.3:0/4127090098 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f1d1402fdb0 0x7f1d14032270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f1d1c0096a0 tx=0x7f1d1c006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:08.177 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:08.176+0000 7f1d1affd700 1 -- 172.21.15.3:0/4127090098 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1d28016080 con 0x7f1d2c0fc170 2023-11-17T05:25:08.384 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:08.382+0000 7f1d347f4700 1 -- 172.21.15.3:0/4127090098 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7f1d2c045ad0 con 0x7f1d2c0fc170 2023-11-17T05:25:08.522 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:08 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:06.377467+0000 mgr.smithi003.eddcuc (mgr.14116) 12 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:25:08.523 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:08 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cephadm 2023-11-17T05:25:06.378060+0000 mgr.smithi003.eddcuc (mgr.14116) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2023-11-17T05:25:08.524 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:08 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:07.427352+0000 mon.smithi003 (mon.0) 59 : audit [INF] from='mgr.14116 172.21.15.3:0/2120313347' entity='mgr.smithi003.eddcuc' 2023-11-17T05:25:09.027 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.023+0000 7f1d1affd700 1 -- 172.21.15.3:0/4127090098 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f1d2801a8d0 con 0x7f1d2c0fc170 2023-11-17T05:25:09.027 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.024+0000 7f1d1affd700 1 -- 172.21.15.3:0/4127090098 <== mon.0 v2:172.21.15.3:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v9) v1 ==== 89+0+0 (secure 0 0 0) 0x7f1d28022920 con 0x7f1d2c0fc170 2023-11-17T05:25:09.029 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.027+0000 7f1d347f4700 1 -- 172.21.15.3:0/4127090098 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f1d1402fdb0 msgr2=0x7f1d14032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:09.029 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.027+0000 7f1d347f4700 1 --2- 172.21.15.3:0/4127090098 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f1d1402fdb0 0x7f1d14032270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f1d1c0096a0 tx=0x7f1d1c006b40).stop 2023-11-17T05:25:09.029 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.027+0000 7f1d347f4700 1 -- 172.21.15.3:0/4127090098 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d2c0fc170 msgr2=0x7f1d2c10cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:09.029 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.027+0000 7f1d347f4700 1 --2- 172.21.15.3:0/4127090098 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d2c0fc170 0x7f1d2c10cbd0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f1d2800e040 tx=0x7f1d28006ee0).stop 2023-11-17T05:25:09.030 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.027+0000 7f1d347f4700 1 -- 172.21.15.3:0/4127090098 shutdown_connections 2023-11-17T05:25:09.030 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.027+0000 7f1d347f4700 1 --2- 172.21.15.3:0/4127090098 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f1d1402fdb0 0x7f1d14032270 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:09.030 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.027+0000 7f1d347f4700 1 --2- 172.21.15.3:0/4127090098 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d2c0fc170 0x7f1d2c10cbd0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:09.030 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.027+0000 7f1d347f4700 1 -- 172.21.15.3:0/4127090098 >> 172.21.15.3:0/4127090098 conn(0x7f1d2c0f59e0 msgr2=0x7f1d2c0f7540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:09.030 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.027+0000 7f1d347f4700 1 -- 172.21.15.3:0/4127090098 shutdown_connections 2023-11-17T05:25:09.030 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.027+0000 7f1d347f4700 1 -- 172.21.15.3:0/4127090098 wait complete. 2023-11-17T05:25:09.380 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:09 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:07.424152+0000 mgr.smithi003.eddcuc (mgr.14116) 14 : audit [DBG] from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:25:09.380 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:09 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cephadm 2023-11-17T05:25:07.424862+0000 mgr.smithi003.eddcuc (mgr.14116) 15 : cephadm [INF] Saving service crash spec with placement * 2023-11-17T05:25:09.380 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:09 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:08.384345+0000 mon.smithi003 (mon.0) 60 : audit [INF] from='client.? 172.21.15.3:0/4127090098' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2023-11-17T05:25:09.402 INFO:teuthology.orchestra.run.smithi003.stderr:Deploying prometheus service with default placement... 2023-11-17T05:25:09.402 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptcuja8tf:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpk7o1jie2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2023-11-17T05:25:09.804 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.801+0000 7f117f079700 1 Processor -- start 2023-11-17T05:25:09.804 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.801+0000 7f117f079700 1 -- start start 2023-11-17T05:25:09.804 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.801+0000 7f117f079700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f11780faae0 0x7f11780faf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:09.804 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.801+0000 7f117f079700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11780fb480 con 0x7f11780faae0 2023-11-17T05:25:09.804 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.801+0000 7f117ce15700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f11780faae0 0x7f11780faf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:09.804 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.801+0000 7f117ce15700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f11780faae0 0x7f11780faf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43314/0 (socket says 172.21.15.3:43314) 2023-11-17T05:25:09.804 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.801+0000 7f117ce15700 1 -- 172.21.15.3:0/1807661811 learned_addr learned my addr 172.21.15.3:0/1807661811 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:25:09.804 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.801+0000 7f117ce15700 1 -- 172.21.15.3:0/1807661811 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11780fb5c0 con 0x7f11780faae0 2023-11-17T05:25:09.804 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.802+0000 7f117ce15700 1 --2- 172.21.15.3:0/1807661811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f11780faae0 0x7f11780faf00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f116c00b5f0 tx=0x7f116c006ee0).ready entity=mon.0 client_cookie=9251a9b515750ef3 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:09.806 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.802+0000 7f11777fe700 1 -- 172.21.15.3:0/1807661811 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f116c00d600 con 0x7f11780faae0 2023-11-17T05:25:09.806 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.802+0000 7f11777fe700 1 -- 172.21.15.3:0/1807661811 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f116c00dbd0 con 0x7f11780faae0 2023-11-17T05:25:09.807 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.802+0000 7f11777fe700 1 -- 172.21.15.3:0/1807661811 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f116c011c00 con 0x7f11780faae0 2023-11-17T05:25:09.807 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.803+0000 7f117f079700 1 -- 172.21.15.3:0/1807661811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f11780faae0 msgr2=0x7f11780faf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:09.807 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.803+0000 7f117f079700 1 --2- 172.21.15.3:0/1807661811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f11780faae0 0x7f11780faf00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f116c00b5f0 tx=0x7f116c006ee0).stop 2023-11-17T05:25:09.807 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.803+0000 7f117f079700 1 -- 172.21.15.3:0/1807661811 shutdown_connections 2023-11-17T05:25:09.807 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.803+0000 7f117f079700 1 --2- 172.21.15.3:0/1807661811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f11780faae0 0x7f11780faf00 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:09.807 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.803+0000 7f117f079700 1 -- 172.21.15.3:0/1807661811 >> 172.21.15.3:0/1807661811 conn(0x7f11780f6280 msgr2=0x7f11780f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:09.807 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.803+0000 7f117f079700 1 -- 172.21.15.3:0/1807661811 shutdown_connections 2023-11-17T05:25:09.807 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.803+0000 7f117f079700 1 -- 172.21.15.3:0/1807661811 wait complete. 2023-11-17T05:25:09.807 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.804+0000 7f117f079700 1 Processor -- start 2023-11-17T05:25:09.808 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.804+0000 7f117f079700 1 -- start start 2023-11-17T05:25:09.808 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.804+0000 7f117f079700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f11780faae0 0x7f117810faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:09.808 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.804+0000 7f117f079700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117810b4e0 con 0x7f11780faae0 2023-11-17T05:25:09.808 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.804+0000 7f117ce15700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f11780faae0 0x7f117810faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:09.808 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.804+0000 7f117ce15700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f11780faae0 0x7f117810faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43330/0 (socket says 172.21.15.3:43330) 2023-11-17T05:25:09.809 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.804+0000 7f117ce15700 1 -- 172.21.15.3:0/3359220716 learned_addr learned my addr 172.21.15.3:0/3359220716 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:25:09.809 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.805+0000 7f117ce15700 1 -- 172.21.15.3:0/3359220716 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f116c00b200 con 0x7f11780faae0 2023-11-17T05:25:09.809 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.805+0000 7f117ce15700 1 --2- 172.21.15.3:0/3359220716 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f11780faae0 0x7f117810faf0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f116c00b1d0 tx=0x7f116c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:09.809 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.805+0000 7f11757fa700 1 -- 172.21.15.3:0/3359220716 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f116c011a20 con 0x7f11780faae0 2023-11-17T05:25:09.809 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.805+0000 7f11757fa700 1 -- 172.21.15.3:0/3359220716 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f116c01b880 con 0x7f11780faae0 2023-11-17T05:25:09.809 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.805+0000 7f117f079700 1 -- 172.21.15.3:0/3359220716 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f117806fc50 con 0x7f11780faae0 2023-11-17T05:25:09.809 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.805+0000 7f11757fa700 1 -- 172.21.15.3:0/3359220716 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f116c01a700 con 0x7f11780faae0 2023-11-17T05:25:09.810 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.805+0000 7f117f079700 1 -- 172.21.15.3:0/3359220716 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f117810ffe0 con 0x7f11780faae0 2023-11-17T05:25:09.810 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.806+0000 7f11757fa700 1 -- 172.21.15.3:0/3359220716 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f116c00f070 con 0x7f11780faae0 2023-11-17T05:25:09.810 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.806+0000 7f11757fa700 1 --2- 172.21.15.3:0/3359220716 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f115c02fdb0 0x7f115c032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:09.811 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.806+0000 7f11757fa700 1 -- 172.21.15.3:0/3359220716 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f116c0410f0 con 0x7f11780faae0 2023-11-17T05:25:09.811 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.806+0000 7f117f079700 1 -- 172.21.15.3:0/3359220716 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1178045ad0 con 0x7f11780faae0 2023-11-17T05:25:09.811 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.806+0000 7f1177fff700 1 -- 172.21.15.3:0/3359220716 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f115c02fdb0 msgr2=0x7f115c032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/2575681100 2023-11-17T05:25:09.811 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.806+0000 7f1177fff700 1 --2- 172.21.15.3:0/3359220716 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f115c02fdb0 0x7f115c032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-17T05:25:09.816 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:09.815+0000 7f11757fa700 1 -- 172.21.15.3:0/3359220716 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f116c016070 con 0x7f11780faae0 2023-11-17T05:25:10.008 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:10.007+0000 7f1177fff700 1 -- 172.21.15.3:0/3359220716 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f115c02fdb0 msgr2=0x7f115c032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/2575681100 2023-11-17T05:25:10.009 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:10.007+0000 7f1177fff700 1 --2- 172.21.15.3:0/3359220716 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f115c02fdb0 0x7f115c032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-17T05:25:10.010 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:10.009+0000 7f117f079700 1 -- 172.21.15.3:0/3359220716 --> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f1178057f10 con 0x7f115c02fdb0 2023-11-17T05:25:10.409 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:10.408+0000 7f1177fff700 1 -- 172.21.15.3:0/3359220716 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f115c02fdb0 msgr2=0x7f115c032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/2575681100 2023-11-17T05:25:10.409 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:10.408+0000 7f1177fff700 1 --2- 172.21.15.3:0/3359220716 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f115c02fdb0 0x7f115c032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-11-17T05:25:10.520 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:10 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:09.025852+0000 mon.smithi003 (mon.0) 61 : audit [INF] from='client.? 172.21.15.3:0/4127090098' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2023-11-17T05:25:10.520 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:10 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:25:09.025915+0000 mon.smithi003 (mon.0) 62 : cluster [DBG] mgrmap e9: smithi003.eddcuc(active, since 17s) 2023-11-17T05:25:11.211 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:11.209+0000 7f1177fff700 1 -- 172.21.15.3:0/3359220716 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f115c02fdb0 msgr2=0x7f115c032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/2575681100 2023-11-17T05:25:11.211 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:11.209+0000 7f1177fff700 1 --2- 172.21.15.3:0/3359220716 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f115c02fdb0 0x7f115c032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-11-17T05:25:12.813 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:12.810+0000 7f1177fff700 1 -- 172.21.15.3:0/3359220716 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f115c02fdb0 msgr2=0x7f115c032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/2575681100 2023-11-17T05:25:12.813 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:12.810+0000 7f1177fff700 1 --2- 172.21.15.3:0/3359220716 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f115c02fdb0 0x7f115c032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-11-17T05:25:14.909 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:14.907+0000 7f11757fa700 1 -- 172.21.15.3:0/3359220716 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mgrmap(e 10) v1 ==== 35984+0+0 (secure 0 0 0) 0x7f116c01a970 con 0x7f11780faae0 2023-11-17T05:25:14.909 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:14.907+0000 7f11757fa700 1 -- 172.21.15.3:0/3359220716 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f115c02fdb0 msgr2=0x7f115c032270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-17T05:25:14.909 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:14.907+0000 7f11757fa700 1 --2- 172.21.15.3:0/3359220716 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f115c02fdb0 0x7f115c032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-17T05:25:15.161 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:14 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:25:14.704584+0000 mon.smithi003 (mon.0) 63 : cluster [INF] Active manager daemon smithi003.eddcuc restarted 2023-11-17T05:25:15.162 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:14 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:25:14.705002+0000 mon.smithi003 (mon.0) 64 : cluster [INF] Activating manager daemon smithi003.eddcuc 2023-11-17T05:25:15.162 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:14 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:25:14.706898+0000 mon.smithi003 (mon.0) 65 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2023-11-17T05:25:15.914 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:15.911+0000 7f11757fa700 1 -- 172.21.15.3:0/3359220716 <== mon.0 v2:172.21.15.3:3300/0 8 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f116c043430 con 0x7f11780faae0 2023-11-17T05:25:15.914 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:15.911+0000 7f11757fa700 1 --2- 172.21.15.3:0/3359220716 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7f115c033c20 0x7f115c036010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:15.914 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:15.911+0000 7f11757fa700 1 -- 172.21.15.3:0/3359220716 --> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f1178057f10 con 0x7f115c033c20 2023-11-17T05:25:15.915 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:15.912+0000 7f1177fff700 1 --2- 172.21.15.3:0/3359220716 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7f115c033c20 0x7f115c036010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:15.915 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:15.912+0000 7f1177fff700 1 --2- 172.21.15.3:0/3359220716 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7f115c033c20 0x7f115c036010 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f11640036d0 tx=0x7f1164006cb0).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:15.919 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:15.917+0000 7f11757fa700 1 -- 172.21.15.3:0/3359220716 <== mgr.14144 v2:172.21.15.3:6800/2830354403 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f1178057f10 con 0x7f115c033c20 2023-11-17T05:25:15.919 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: Scheduled prometheus update... 2023-11-17T05:25:15.921 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:15.920+0000 7f117f079700 1 -- 172.21.15.3:0/3359220716 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7f115c033c20 msgr2=0x7f115c036010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:15.921 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:15.920+0000 7f117f079700 1 --2- 172.21.15.3:0/3359220716 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7f115c033c20 0x7f115c036010 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f11640036d0 tx=0x7f1164006cb0).stop 2023-11-17T05:25:15.921 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:15.920+0000 7f117f079700 1 -- 172.21.15.3:0/3359220716 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f11780faae0 msgr2=0x7f117810faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:15.922 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:15.920+0000 7f117f079700 1 --2- 172.21.15.3:0/3359220716 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f11780faae0 0x7f117810faf0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f116c00b1d0 tx=0x7f116c006ee0).stop 2023-11-17T05:25:15.922 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:15.920+0000 7f117f079700 1 -- 172.21.15.3:0/3359220716 shutdown_connections 2023-11-17T05:25:15.922 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:15.920+0000 7f117f079700 1 --2- 172.21.15.3:0/3359220716 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7f115c033c20 0x7f115c036010 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:15.923 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:15.920+0000 7f117f079700 1 --2- 172.21.15.3:0/3359220716 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f11780faae0 0x7f117810faf0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:15.923 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:15.920+0000 7f117f079700 1 --2- 172.21.15.3:0/3359220716 >> [v2:172.21.15.3:6800/2575681100,v1:172.21.15.3:6801/2575681100] conn(0x7f115c02fdb0 0x7f115c032270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-17T05:25:15.923 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:15.920+0000 7f117f079700 1 -- 172.21.15.3:0/3359220716 >> 172.21.15.3:0/3359220716 conn(0x7f11780f6280 msgr2=0x7f11780f7e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:15.923 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:15.920+0000 7f117f079700 1 -- 172.21.15.3:0/3359220716 shutdown_connections 2023-11-17T05:25:15.923 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:15.920+0000 7f117f079700 1 -- 172.21.15.3:0/3359220716 wait complete. 2023-11-17T05:25:16.235 INFO:teuthology.orchestra.run.smithi003.stderr:Deploying grafana service with default placement... 2023-11-17T05:25:16.235 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptcuja8tf:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpk7o1jie2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2023-11-17T05:25:16.238 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:15 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:25:14.909851+0000 mon.smithi003 (mon.0) 66 : cluster [DBG] mgrmap e10: smithi003.eddcuc(active, starting, since 0.204934s) 2023-11-17T05:25:16.238 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:15 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:14.911147+0000 mon.smithi003 (mon.0) 67 : audit [DBG] from='mgr.14144 172.21.15.3:0/3813803755' entity='mgr.smithi003.eddcuc' cmd=[{"prefix": "mon metadata", "id": "smithi003"}]: dispatch 2023-11-17T05:25:16.238 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:15 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:14.911888+0000 mon.smithi003 (mon.0) 68 : audit [DBG] from='mgr.14144 172.21.15.3:0/3813803755' entity='mgr.smithi003.eddcuc' cmd=[{"prefix": "mgr metadata", "who": "smithi003.eddcuc", "id": "smithi003.eddcuc"}]: dispatch 2023-11-17T05:25:16.238 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:15 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:14.912821+0000 mon.smithi003 (mon.0) 69 : audit [DBG] from='mgr.14144 172.21.15.3:0/3813803755' entity='mgr.smithi003.eddcuc' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-17T05:25:16.238 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:15 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:14.913085+0000 mon.smithi003 (mon.0) 70 : audit [DBG] from='mgr.14144 172.21.15.3:0/3813803755' entity='mgr.smithi003.eddcuc' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-17T05:25:16.238 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:15 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:14.913283+0000 mon.smithi003 (mon.0) 71 : audit [DBG] from='mgr.14144 172.21.15.3:0/3813803755' entity='mgr.smithi003.eddcuc' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-17T05:25:16.238 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:15 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:25:14.927240+0000 mon.smithi003 (mon.0) 72 : cluster [INF] Manager daemon smithi003.eddcuc is now available 2023-11-17T05:25:16.239 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:15 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:15.042923+0000 mon.smithi003 (mon.0) 73 : audit [DBG] from='mgr.14144 172.21.15.3:0/3813803755' entity='mgr.smithi003.eddcuc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:25:16.239 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:15 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:15.048455+0000 mon.smithi003 (mon.0) 74 : audit [DBG] from='mgr.14144 172.21.15.3:0/3813803755' entity='mgr.smithi003.eddcuc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:25:16.239 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:15 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:15.061716+0000 mon.smithi003 (mon.0) 75 : audit [INF] from='mgr.14144 172.21.15.3:0/3813803755' entity='mgr.smithi003.eddcuc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.eddcuc/mirror_snapshot_schedule"}]: dispatch 2023-11-17T05:25:16.239 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:15 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:15.072078+0000 mon.smithi003 (mon.0) 76 : audit [INF] from='mgr.14144 172.21.15.3:0/3813803755' entity='mgr.smithi003.eddcuc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.eddcuc/trash_purge_schedule"}]: dispatch 2023-11-17T05:25:16.630 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.627+0000 7fa5113a0700 1 Processor -- start 2023-11-17T05:25:16.630 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.628+0000 7fa5113a0700 1 -- start start 2023-11-17T05:25:16.630 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.628+0000 7fa5113a0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa50c0fa8f0 0x7fa50c0fad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:16.630 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.628+0000 7fa5113a0700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa50c0fb200 con 0x7fa50c0fa8f0 2023-11-17T05:25:16.632 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.628+0000 7fa50bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa50c0fa8f0 0x7fa50c0fad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:16.632 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.628+0000 7fa50bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa50c0fa8f0 0x7fa50c0fad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43376/0 (socket says 172.21.15.3:43376) 2023-11-17T05:25:16.632 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.628+0000 7fa50bfff700 1 -- 172.21.15.3:0/1359422889 learned_addr learned my addr 172.21.15.3:0/1359422889 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:25:16.632 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.628+0000 7fa50bfff700 1 -- 172.21.15.3:0/1359422889 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa50c0fb340 con 0x7fa50c0fa8f0 2023-11-17T05:25:16.632 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.628+0000 7fa50bfff700 1 --2- 172.21.15.3:0/1359422889 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa50c0fa8f0 0x7fa50c0fad10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fa4fc00b5f0 tx=0x7fa4fc006ee0).ready entity=mon.0 client_cookie=a4f7f84f93c12be3 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:16.632 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.629+0000 7fa50affd700 1 -- 172.21.15.3:0/1359422889 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4fc00d600 con 0x7fa50c0fa8f0 2023-11-17T05:25:16.632 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.629+0000 7fa50affd700 1 -- 172.21.15.3:0/1359422889 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa4fc00dbd0 con 0x7fa50c0fa8f0 2023-11-17T05:25:16.632 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.629+0000 7fa50affd700 1 -- 172.21.15.3:0/1359422889 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4fc011c00 con 0x7fa50c0fa8f0 2023-11-17T05:25:16.632 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.629+0000 7fa5113a0700 1 -- 172.21.15.3:0/1359422889 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa50c0fa8f0 msgr2=0x7fa50c0fad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:16.632 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.629+0000 7fa5113a0700 1 --2- 172.21.15.3:0/1359422889 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa50c0fa8f0 0x7fa50c0fad10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fa4fc00b5f0 tx=0x7fa4fc006ee0).stop 2023-11-17T05:25:16.632 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.629+0000 7fa5113a0700 1 -- 172.21.15.3:0/1359422889 shutdown_connections 2023-11-17T05:25:16.633 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.629+0000 7fa5113a0700 1 --2- 172.21.15.3:0/1359422889 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa50c0fa8f0 0x7fa50c0fad10 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:16.633 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.629+0000 7fa5113a0700 1 -- 172.21.15.3:0/1359422889 >> 172.21.15.3:0/1359422889 conn(0x7fa50c0f6130 msgr2=0x7fa50c0f8550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:16.633 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.629+0000 7fa5113a0700 1 -- 172.21.15.3:0/1359422889 shutdown_connections 2023-11-17T05:25:16.633 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.629+0000 7fa5113a0700 1 -- 172.21.15.3:0/1359422889 wait complete. 2023-11-17T05:25:16.633 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.630+0000 7fa5113a0700 1 Processor -- start 2023-11-17T05:25:16.633 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.630+0000 7fa5113a0700 1 -- start start 2023-11-17T05:25:16.634 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.630+0000 7fa5113a0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa50c0fa8f0 0x7fa50c10a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:16.634 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.630+0000 7fa5113a0700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa50c10ac80 con 0x7fa50c0fa8f0 2023-11-17T05:25:16.635 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.631+0000 7fa50bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa50c0fa8f0 0x7fa50c10a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:16.635 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.631+0000 7fa50bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa50c0fa8f0 0x7fa50c10a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43380/0 (socket says 172.21.15.3:43380) 2023-11-17T05:25:16.635 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.631+0000 7fa50bfff700 1 -- 172.21.15.3:0/3716197585 learned_addr learned my addr 172.21.15.3:0/3716197585 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:25:16.635 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.631+0000 7fa50bfff700 1 -- 172.21.15.3:0/3716197585 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4fc00b200 con 0x7fa50c0fa8f0 2023-11-17T05:25:16.635 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.631+0000 7fa50bfff700 1 --2- 172.21.15.3:0/3716197585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa50c0fa8f0 0x7fa50c10a790 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fa4fc00e040 tx=0x7fa4fc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:16.635 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.631+0000 7fa508ff9700 1 -- 172.21.15.3:0/3716197585 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4fc01b410 con 0x7fa50c0fa8f0 2023-11-17T05:25:16.635 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.631+0000 7fa5113a0700 1 -- 172.21.15.3:0/3716197585 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa50c10ae80 con 0x7fa50c0fa8f0 2023-11-17T05:25:16.635 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.631+0000 7fa508ff9700 1 -- 172.21.15.3:0/3716197585 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa4fc01b9e0 con 0x7fa50c0fa8f0 2023-11-17T05:25:16.635 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.631+0000 7fa508ff9700 1 -- 172.21.15.3:0/3716197585 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4fc01a820 con 0x7fa50c0fa8f0 2023-11-17T05:25:16.635 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.631+0000 7fa5113a0700 1 -- 172.21.15.3:0/3716197585 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa50c10b290 con 0x7fa50c0fa8f0 2023-11-17T05:25:16.635 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.632+0000 7fa5113a0700 1 -- 172.21.15.3:0/3716197585 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa50c045ad0 con 0x7fa50c0fa8f0 2023-11-17T05:25:16.636 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.632+0000 7fa508ff9700 1 -- 172.21.15.3:0/3716197585 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7fa4fc00f070 con 0x7fa50c0fa8f0 2023-11-17T05:25:16.636 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.633+0000 7fa508ff9700 1 --2- 172.21.15.3:0/3716197585 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7fa4f802fcf0 0x7fa4f80321b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:16.636 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.633+0000 7fa508ff9700 1 -- 172.21.15.3:0/3716197585 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fa4fc041430 con 0x7fa50c0fa8f0 2023-11-17T05:25:16.636 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.633+0000 7fa50b7fe700 1 --2- 172.21.15.3:0/3716197585 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7fa4f802fcf0 0x7fa4f80321b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:16.636 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.633+0000 7fa50b7fe700 1 --2- 172.21.15.3:0/3716197585 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7fa4f802fcf0 0x7fa4f80321b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fa5000096a0 tx=0x7fa500006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:16.639 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.638+0000 7fa508ff9700 1 -- 172.21.15.3:0/3716197585 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa4fc019cd0 con 0x7fa50c0fa8f0 2023-11-17T05:25:16.838 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.834+0000 7fa5113a0700 1 -- 172.21.15.3:0/3716197585 --> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7fa50c10bb70 con 0x7fa4f802fcf0 2023-11-17T05:25:16.840 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.838+0000 7fa508ff9700 1 -- 172.21.15.3:0/3716197585 <== mgr.14144 v2:172.21.15.3:6800/2830354403 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fa50c10bb70 con 0x7fa4f802fcf0 2023-11-17T05:25:16.840 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: Scheduled grafana update... 2023-11-17T05:25:16.843 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.840+0000 7fa5113a0700 1 -- 172.21.15.3:0/3716197585 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7fa4f802fcf0 msgr2=0x7fa4f80321b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:16.843 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.840+0000 7fa5113a0700 1 --2- 172.21.15.3:0/3716197585 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7fa4f802fcf0 0x7fa4f80321b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fa5000096a0 tx=0x7fa500006b40).stop 2023-11-17T05:25:16.843 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.840+0000 7fa5113a0700 1 -- 172.21.15.3:0/3716197585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa50c0fa8f0 msgr2=0x7fa50c10a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:16.843 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.840+0000 7fa5113a0700 1 --2- 172.21.15.3:0/3716197585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa50c0fa8f0 0x7fa50c10a790 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fa4fc00e040 tx=0x7fa4fc006ee0).stop 2023-11-17T05:25:16.843 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.840+0000 7fa5113a0700 1 -- 172.21.15.3:0/3716197585 shutdown_connections 2023-11-17T05:25:16.843 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.840+0000 7fa5113a0700 1 --2- 172.21.15.3:0/3716197585 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7fa4f802fcf0 0x7fa4f80321b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:16.843 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.840+0000 7fa5113a0700 1 --2- 172.21.15.3:0/3716197585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa50c0fa8f0 0x7fa50c10a790 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:16.843 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.840+0000 7fa5113a0700 1 -- 172.21.15.3:0/3716197585 >> 172.21.15.3:0/3716197585 conn(0x7fa50c0f6130 msgr2=0x7fa50c06f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:16.843 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.841+0000 7fa5113a0700 1 -- 172.21.15.3:0/3716197585 shutdown_connections 2023-11-17T05:25:16.844 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:16.841+0000 7fa5113a0700 1 -- 172.21.15.3:0/3716197585 wait complete. 2023-11-17T05:25:17.162 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:16 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:25:15.915183+0000 mon.smithi003 (mon.0) 77 : cluster [DBG] mgrmap e11: smithi003.eddcuc(active, since 1.21027s) 2023-11-17T05:25:17.163 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:16 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cephadm 2023-11-17T05:25:15.916548+0000 mgr.smithi003.eddcuc (mgr.14144) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2023-11-17T05:25:17.163 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:16 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:15.918795+0000 mon.smithi003 (mon.0) 78 : audit [INF] from='mgr.14144 172.21.15.3:0/3813803755' entity='mgr.smithi003.eddcuc' 2023-11-17T05:25:17.163 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:16 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:16.840058+0000 mon.smithi003 (mon.0) 79 : audit [INF] from='mgr.14144 172.21.15.3:0/3813803755' entity='mgr.smithi003.eddcuc' 2023-11-17T05:25:17.193 INFO:teuthology.orchestra.run.smithi003.stderr:Deploying node-exporter service with default placement... 2023-11-17T05:25:17.193 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptcuja8tf:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpk7o1jie2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2023-11-17T05:25:17.560 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.557+0000 7f6524339700 1 Processor -- start 2023-11-17T05:25:17.561 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.558+0000 7f6524339700 1 -- start start 2023-11-17T05:25:17.561 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.558+0000 7f6524339700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f651c0fc090 0x7f651c0fc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:17.561 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.558+0000 7f6524339700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f651c0fc9a0 con 0x7f651c0fc090 2023-11-17T05:25:17.561 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.559+0000 7f65220d5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f651c0fc090 0x7f651c0fc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:17.561 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.559+0000 7f65220d5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f651c0fc090 0x7f651c0fc4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43396/0 (socket says 172.21.15.3:43396) 2023-11-17T05:25:17.561 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.559+0000 7f65220d5700 1 -- 172.21.15.3:0/3302199656 learned_addr learned my addr 172.21.15.3:0/3302199656 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:25:17.561 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.559+0000 7f65220d5700 1 -- 172.21.15.3:0/3302199656 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f651c0fcae0 con 0x7f651c0fc090 2023-11-17T05:25:17.563 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.559+0000 7f65220d5700 1 --2- 172.21.15.3:0/3302199656 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f651c0fc090 0x7f651c0fc4b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f651800b870 tx=0x7f6518006f70).ready entity=mon.0 client_cookie=b65ce6ee410994b1 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:17.563 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.559+0000 7f65210d3700 1 -- 172.21.15.3:0/3302199656 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f651800da20 con 0x7f651c0fc090 2023-11-17T05:25:17.563 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.559+0000 7f65210d3700 1 -- 172.21.15.3:0/3302199656 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f6518010470 con 0x7f651c0fc090 2023-11-17T05:25:17.563 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.559+0000 7f65210d3700 1 -- 172.21.15.3:0/3302199656 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f651800f5b0 con 0x7f651c0fc090 2023-11-17T05:25:17.563 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.560+0000 7f6524339700 1 -- 172.21.15.3:0/3302199656 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f651c0fc090 msgr2=0x7f651c0fc4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:17.563 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.560+0000 7f6524339700 1 --2- 172.21.15.3:0/3302199656 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f651c0fc090 0x7f651c0fc4b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f651800b870 tx=0x7f6518006f70).stop 2023-11-17T05:25:17.564 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.560+0000 7f6524339700 1 -- 172.21.15.3:0/3302199656 shutdown_connections 2023-11-17T05:25:17.564 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.560+0000 7f6524339700 1 --2- 172.21.15.3:0/3302199656 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f651c0fc090 0x7f651c0fc4b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:17.564 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.560+0000 7f6524339700 1 -- 172.21.15.3:0/3302199656 >> 172.21.15.3:0/3302199656 conn(0x7f651c0f5a80 msgr2=0x7f651c0f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:17.564 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.560+0000 7f6524339700 1 -- 172.21.15.3:0/3302199656 shutdown_connections 2023-11-17T05:25:17.564 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.560+0000 7f6524339700 1 -- 172.21.15.3:0/3302199656 wait complete. 2023-11-17T05:25:17.564 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.561+0000 7f6524339700 1 Processor -- start 2023-11-17T05:25:17.564 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.561+0000 7f6524339700 1 -- start start 2023-11-17T05:25:17.564 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.561+0000 7f6524339700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f651c0fc090 0x7f651c10cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:17.564 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.561+0000 7f6524339700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f651c10d050 con 0x7f651c0fc090 2023-11-17T05:25:17.564 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.561+0000 7f65220d5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f651c0fc090 0x7f651c10cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:17.564 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.561+0000 7f65220d5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f651c0fc090 0x7f651c10cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43410/0 (socket says 172.21.15.3:43410) 2023-11-17T05:25:17.564 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.561+0000 7f65220d5700 1 -- 172.21.15.3:0/957205149 learned_addr learned my addr 172.21.15.3:0/957205149 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:25:17.565 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.562+0000 7f65220d5700 1 -- 172.21.15.3:0/957205149 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f651800b480 con 0x7f651c0fc090 2023-11-17T05:25:17.565 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.562+0000 7f65220d5700 1 --2- 172.21.15.3:0/957205149 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f651c0fc090 0x7f651c10cb60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f6518006b80 tx=0x7f6518006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:17.566 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.562+0000 7f650affd700 1 -- 172.21.15.3:0/957205149 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f651800de10 con 0x7f651c0fc090 2023-11-17T05:25:17.566 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.562+0000 7f650affd700 1 -- 172.21.15.3:0/957205149 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f651800fef0 con 0x7f651c0fc090 2023-11-17T05:25:17.566 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.562+0000 7f650affd700 1 -- 172.21.15.3:0/957205149 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6518019de0 con 0x7f651c0fc090 2023-11-17T05:25:17.566 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.562+0000 7f6524339700 1 -- 172.21.15.3:0/957205149 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f651c10d250 con 0x7f651c0fc090 2023-11-17T05:25:17.566 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.562+0000 7f6524339700 1 -- 172.21.15.3:0/957205149 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f651c10d660 con 0x7f651c0fc090 2023-11-17T05:25:17.566 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.563+0000 7f650affd700 1 -- 172.21.15.3:0/957205149 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f6518016030 con 0x7f651c0fc090 2023-11-17T05:25:17.566 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.563+0000 7f650affd700 1 --2- 172.21.15.3:0/957205149 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7f650402feb0 0x7f6504032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:17.566 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.563+0000 7f650affd700 1 -- 172.21.15.3:0/957205149 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f6518041cb0 con 0x7f651c0fc090 2023-11-17T05:25:17.566 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.563+0000 7f65218d4700 1 --2- 172.21.15.3:0/957205149 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7f650402feb0 0x7f6504032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:17.567 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.563+0000 7f6524339700 1 -- 172.21.15.3:0/957205149 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f651c059100 con 0x7f651c0fc090 2023-11-17T05:25:17.567 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.564+0000 7f65218d4700 1 --2- 172.21.15.3:0/957205149 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7f650402feb0 0x7f6504032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f650c0096a0 tx=0x7f650c006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:17.569 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.568+0000 7f650affd700 1 -- 172.21.15.3:0/957205149 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6518014020 con 0x7f651c0fc090 2023-11-17T05:25:17.764 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.762+0000 7f6524339700 1 -- 172.21.15.3:0/957205149 --> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f651c102620 con 0x7f650402feb0 2023-11-17T05:25:17.768 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.767+0000 7f650affd700 1 -- 172.21.15.3:0/957205149 <== mgr.14144 v2:172.21.15.3:6800/2830354403 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f651c102620 con 0x7f650402feb0 2023-11-17T05:25:17.768 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2023-11-17T05:25:17.771 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.769+0000 7f6524339700 1 -- 172.21.15.3:0/957205149 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7f650402feb0 msgr2=0x7f6504032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:17.771 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.769+0000 7f6524339700 1 --2- 172.21.15.3:0/957205149 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7f650402feb0 0x7f6504032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f650c0096a0 tx=0x7f650c006b40).stop 2023-11-17T05:25:17.771 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.769+0000 7f6524339700 1 -- 172.21.15.3:0/957205149 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f651c0fc090 msgr2=0x7f651c10cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:17.771 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.769+0000 7f6524339700 1 --2- 172.21.15.3:0/957205149 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f651c0fc090 0x7f651c10cb60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f6518006b80 tx=0x7f6518006f70).stop 2023-11-17T05:25:17.771 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.769+0000 7f6524339700 1 -- 172.21.15.3:0/957205149 shutdown_connections 2023-11-17T05:25:17.771 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.769+0000 7f6524339700 1 --2- 172.21.15.3:0/957205149 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7f650402feb0 0x7f6504032370 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:17.772 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.769+0000 7f6524339700 1 --2- 172.21.15.3:0/957205149 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f651c0fc090 0x7f651c10cb60 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:17.772 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.769+0000 7f6524339700 1 -- 172.21.15.3:0/957205149 >> 172.21.15.3:0/957205149 conn(0x7f651c0f5a80 msgr2=0x7f651c0ff890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:17.772 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.769+0000 7f6524339700 1 -- 172.21.15.3:0/957205149 shutdown_connections 2023-11-17T05:25:17.772 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:17.769+0000 7f6524339700 1 -- 172.21.15.3:0/957205149 wait complete. 2023-11-17T05:25:18.071 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:17 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:16.837137+0000 mgr.smithi003.eddcuc (mgr.14144) 3 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:25:18.071 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:17 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cephadm 2023-11-17T05:25:16.837713+0000 mgr.smithi003.eddcuc (mgr.14144) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2023-11-17T05:25:18.071 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:17 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:25:16.920609+0000 mon.smithi003 (mon.0) 80 : cluster [DBG] mgrmap e12: smithi003.eddcuc(active, since 2s) 2023-11-17T05:25:18.071 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:17 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:17.768381+0000 mon.smithi003 (mon.0) 81 : audit [INF] from='mgr.14144 172.21.15.3:0/3813803755' entity='mgr.smithi003.eddcuc' 2023-11-17T05:25:18.126 INFO:teuthology.orchestra.run.smithi003.stderr:Deploying alertmanager service with default placement... 2023-11-17T05:25:18.126 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptcuja8tf:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpk7o1jie2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2023-11-17T05:25:18.558 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.552+0000 7f01757ca700 1 Processor -- start 2023-11-17T05:25:18.558 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.552+0000 7f01757ca700 1 -- start start 2023-11-17T05:25:18.558 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.552+0000 7f01757ca700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01700fc0c0 0x7f01700fc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:18.558 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.552+0000 7f01757ca700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01700fc9d0 con 0x7f01700fc0c0 2023-11-17T05:25:18.558 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.553+0000 7f016effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01700fc0c0 0x7f01700fc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:18.558 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.553+0000 7f016effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01700fc0c0 0x7f01700fc4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55358/0 (socket says 172.21.15.3:55358) 2023-11-17T05:25:18.558 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.553+0000 7f016effd700 1 -- 172.21.15.3:0/145794975 learned_addr learned my addr 172.21.15.3:0/145794975 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:25:18.558 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.553+0000 7f016effd700 1 -- 172.21.15.3:0/145794975 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01700fcb10 con 0x7f01700fc0c0 2023-11-17T05:25:18.558 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.553+0000 7f016effd700 1 --2- 172.21.15.3:0/145794975 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01700fc0c0 0x7f01700fc4e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f0160009960 tx=0x7f01600092d0).ready entity=mon.0 client_cookie=b5318f223c94ce86 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:18.558 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.553+0000 7f016dffb700 1 -- 172.21.15.3:0/145794975 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f016000b9c0 con 0x7f01700fc0c0 2023-11-17T05:25:18.559 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.553+0000 7f016dffb700 1 -- 172.21.15.3:0/145794975 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0160011400 con 0x7f01700fc0c0 2023-11-17T05:25:18.559 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.553+0000 7f016dffb700 1 -- 172.21.15.3:0/145794975 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01600103a0 con 0x7f01700fc0c0 2023-11-17T05:25:18.559 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.554+0000 7f01757ca700 1 -- 172.21.15.3:0/145794975 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01700fc0c0 msgr2=0x7f01700fc4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:18.559 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.554+0000 7f01757ca700 1 --2- 172.21.15.3:0/145794975 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01700fc0c0 0x7f01700fc4e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f0160009960 tx=0x7f01600092d0).stop 2023-11-17T05:25:18.559 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.554+0000 7f01757ca700 1 -- 172.21.15.3:0/145794975 shutdown_connections 2023-11-17T05:25:18.559 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.554+0000 7f01757ca700 1 --2- 172.21.15.3:0/145794975 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01700fc0c0 0x7f01700fc4e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:18.559 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.554+0000 7f01757ca700 1 -- 172.21.15.3:0/145794975 >> 172.21.15.3:0/145794975 conn(0x7f01700f5a80 msgr2=0x7f01700f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:18.559 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.554+0000 7f01757ca700 1 -- 172.21.15.3:0/145794975 shutdown_connections 2023-11-17T05:25:18.559 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.554+0000 7f01757ca700 1 -- 172.21.15.3:0/145794975 wait complete. 2023-11-17T05:25:18.559 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.555+0000 7f01757ca700 1 Processor -- start 2023-11-17T05:25:18.559 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.555+0000 7f01757ca700 1 -- start start 2023-11-17T05:25:18.560 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.555+0000 7f01757ca700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01700fc0c0 0x7f01700ff450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:18.560 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.555+0000 7f01757ca700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01700fd8d0 con 0x7f01700fc0c0 2023-11-17T05:25:18.560 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.555+0000 7f016effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01700fc0c0 0x7f01700ff450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:18.560 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.555+0000 7f016effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01700fc0c0 0x7f01700ff450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55364/0 (socket says 172.21.15.3:55364) 2023-11-17T05:25:18.560 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.555+0000 7f016effd700 1 -- 172.21.15.3:0/604012438 learned_addr learned my addr 172.21.15.3:0/604012438 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:25:18.560 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.555+0000 7f016effd700 1 -- 172.21.15.3:0/604012438 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0160009570 con 0x7f01700fc0c0 2023-11-17T05:25:18.560 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.556+0000 7f016effd700 1 --2- 172.21.15.3:0/604012438 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01700fc0c0 0x7f01700ff450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f016000bac0 tx=0x7f01600092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:18.560 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.556+0000 7f01677fe700 1 -- 172.21.15.3:0/604012438 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0160010870 con 0x7f01700fc0c0 2023-11-17T05:25:18.560 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.556+0000 7f01677fe700 1 -- 172.21.15.3:0/604012438 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0160010e40 con 0x7f01700fc0c0 2023-11-17T05:25:18.560 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.556+0000 7f01757ca700 1 -- 172.21.15.3:0/604012438 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01700fda70 con 0x7f01700fc0c0 2023-11-17T05:25:18.560 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.556+0000 7f01677fe700 1 -- 172.21.15.3:0/604012438 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f016001ae00 con 0x7f01700fc0c0 2023-11-17T05:25:18.560 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.556+0000 7f01757ca700 1 -- 172.21.15.3:0/604012438 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01700fde80 con 0x7f01700fc0c0 2023-11-17T05:25:18.561 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.557+0000 7f01677fe700 1 -- 172.21.15.3:0/604012438 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f016000f070 con 0x7f01700fc0c0 2023-11-17T05:25:18.561 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.557+0000 7f01677fe700 1 --2- 172.21.15.3:0/604012438 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7f015002fe40 0x7f0150032300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:18.561 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.557+0000 7f01677fe700 1 -- 172.21.15.3:0/604012438 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f0160041fd0 con 0x7f01700fc0c0 2023-11-17T05:25:18.562 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.557+0000 7f016e7fc700 1 --2- 172.21.15.3:0/604012438 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7f015002fe40 0x7f0150032300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:18.562 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.558+0000 7f01757ca700 1 -- 172.21.15.3:0/604012438 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0170059100 con 0x7f01700fc0c0 2023-11-17T05:25:18.562 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.558+0000 7f016e7fc700 1 --2- 172.21.15.3:0/604012438 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7f015002fe40 0x7f0150032300 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f01580096a0 tx=0x7f0158006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:18.563 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.562+0000 7f01677fe700 1 -- 172.21.15.3:0/604012438 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f016001e020 con 0x7f01700fc0c0 2023-11-17T05:25:18.758 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.756+0000 7f01757ca700 1 -- 172.21.15.3:0/604012438 --> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f01701044a0 con 0x7f015002fe40 2023-11-17T05:25:18.761 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.760+0000 7f01677fe700 1 -- 172.21.15.3:0/604012438 <== mgr.14144 v2:172.21.15.3:6800/2830354403 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f01701044a0 con 0x7f015002fe40 2023-11-17T05:25:18.762 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2023-11-17T05:25:18.764 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.762+0000 7f01757ca700 1 -- 172.21.15.3:0/604012438 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7f015002fe40 msgr2=0x7f0150032300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:18.764 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.762+0000 7f01757ca700 1 --2- 172.21.15.3:0/604012438 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7f015002fe40 0x7f0150032300 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f01580096a0 tx=0x7f0158006b40).stop 2023-11-17T05:25:18.764 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.762+0000 7f01757ca700 1 -- 172.21.15.3:0/604012438 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01700fc0c0 msgr2=0x7f01700ff450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:18.764 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.762+0000 7f01757ca700 1 --2- 172.21.15.3:0/604012438 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01700fc0c0 0x7f01700ff450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f016000bac0 tx=0x7f01600092d0).stop 2023-11-17T05:25:18.764 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.762+0000 7f01757ca700 1 -- 172.21.15.3:0/604012438 shutdown_connections 2023-11-17T05:25:18.764 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.762+0000 7f01757ca700 1 --2- 172.21.15.3:0/604012438 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7f015002fe40 0x7f0150032300 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:18.764 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.762+0000 7f01757ca700 1 --2- 172.21.15.3:0/604012438 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01700fc0c0 0x7f01700ff450 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:18.765 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.762+0000 7f01757ca700 1 -- 172.21.15.3:0/604012438 >> 172.21.15.3:0/604012438 conn(0x7f01700f5a80 msgr2=0x7f01700fa670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:18.765 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.762+0000 7f01757ca700 1 -- 172.21.15.3:0/604012438 shutdown_connections 2023-11-17T05:25:18.765 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:18.762+0000 7f01757ca700 1 -- 172.21.15.3:0/604012438 wait complete. 2023-11-17T05:25:19.046 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:18 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:17.765202+0000 mgr.smithi003.eddcuc (mgr.14144) 5 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:25:19.047 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:18 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cephadm 2023-11-17T05:25:17.765961+0000 mgr.smithi003.eddcuc (mgr.14144) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2023-11-17T05:25:19.047 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:18 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:18.761583+0000 mon.smithi003 (mon.0) 82 : audit [INF] from='mgr.14144 172.21.15.3:0/3813803755' entity='mgr.smithi003.eddcuc' 2023-11-17T05:25:19.111 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptcuja8tf:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpk7o1jie2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2023-11-17T05:25:19.549 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.542+0000 7f3a07b36700 1 Processor -- start 2023-11-17T05:25:19.549 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.542+0000 7f3a07b36700 1 -- start start 2023-11-17T05:25:19.550 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.542+0000 7f3a07b36700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3a000fd2a0 0x7f3a000fd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:19.550 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.542+0000 7f3a07b36700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a000fdbb0 con 0x7f3a000fd2a0 2023-11-17T05:25:19.554 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.542+0000 7f3a058d2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3a000fd2a0 0x7f3a000fd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:19.554 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.542+0000 7f3a058d2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3a000fd2a0 0x7f3a000fd6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55366/0 (socket says 172.21.15.3:55366) 2023-11-17T05:25:19.554 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.542+0000 7f3a058d2700 1 -- 172.21.15.3:0/185554992 learned_addr learned my addr 172.21.15.3:0/185554992 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:25:19.554 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.543+0000 7f3a058d2700 1 -- 172.21.15.3:0/185554992 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a000fdcf0 con 0x7f3a000fd2a0 2023-11-17T05:25:19.555 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.543+0000 7f3a058d2700 1 --2- 172.21.15.3:0/185554992 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3a000fd2a0 0x7f3a000fd6c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f39fc00b5f0 tx=0x7f39fc006ee0).ready entity=mon.0 client_cookie=65d98a9a42d08f3b server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:19.555 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.543+0000 7f3a048d0700 1 -- 172.21.15.3:0/185554992 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39fc00d600 con 0x7f3a000fd2a0 2023-11-17T05:25:19.555 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.543+0000 7f3a048d0700 1 -- 172.21.15.3:0/185554992 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f39fc00dbd0 con 0x7f3a000fd2a0 2023-11-17T05:25:19.555 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.543+0000 7f3a048d0700 1 -- 172.21.15.3:0/185554992 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39fc011c00 con 0x7f3a000fd2a0 2023-11-17T05:25:19.555 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.544+0000 7f3a07b36700 1 -- 172.21.15.3:0/185554992 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3a000fd2a0 msgr2=0x7f3a000fd6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:19.556 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.544+0000 7f3a07b36700 1 --2- 172.21.15.3:0/185554992 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3a000fd2a0 0x7f3a000fd6c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f39fc00b5f0 tx=0x7f39fc006ee0).stop 2023-11-17T05:25:19.556 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.544+0000 7f3a07b36700 1 -- 172.21.15.3:0/185554992 shutdown_connections 2023-11-17T05:25:19.556 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.544+0000 7f3a07b36700 1 --2- 172.21.15.3:0/185554992 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3a000fd2a0 0x7f3a000fd6c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:19.556 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.544+0000 7f3a07b36700 1 -- 172.21.15.3:0/185554992 >> 172.21.15.3:0/185554992 conn(0x7f3a000f5a80 msgr2=0x7f3a000f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:19.556 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.544+0000 7f3a07b36700 1 -- 172.21.15.3:0/185554992 shutdown_connections 2023-11-17T05:25:19.557 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.544+0000 7f3a07b36700 1 -- 172.21.15.3:0/185554992 wait complete. 2023-11-17T05:25:19.557 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.544+0000 7f3a07b36700 1 Processor -- start 2023-11-17T05:25:19.557 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.544+0000 7f3a07b36700 1 -- start start 2023-11-17T05:25:19.557 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.545+0000 7f3a07b36700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3a000fd2a0 0x7f3a0010aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:19.557 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.545+0000 7f3a07b36700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0010af20 con 0x7f3a000fd2a0 2023-11-17T05:25:19.558 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.545+0000 7f3a058d2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3a000fd2a0 0x7f3a0010aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:19.558 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.545+0000 7f3a058d2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3a000fd2a0 0x7f3a0010aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55372/0 (socket says 172.21.15.3:55372) 2023-11-17T05:25:19.558 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.545+0000 7f3a058d2700 1 -- 172.21.15.3:0/1534610966 learned_addr learned my addr 172.21.15.3:0/1534610966 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:25:19.558 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.545+0000 7f3a058d2700 1 -- 172.21.15.3:0/1534610966 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39fc00b200 con 0x7f3a000fd2a0 2023-11-17T05:25:19.559 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.545+0000 7f3a058d2700 1 --2- 172.21.15.3:0/1534610966 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3a000fd2a0 0x7f3a0010aa30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f39fc006c70 tx=0x7f39fc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:19.559 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.545+0000 7f39ee7fc700 1 -- 172.21.15.3:0/1534610966 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39fc011690 con 0x7f3a000fd2a0 2023-11-17T05:25:19.559 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.545+0000 7f39ee7fc700 1 -- 172.21.15.3:0/1534610966 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f39fc01b880 con 0x7f3a000fd2a0 2023-11-17T05:25:19.559 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.546+0000 7f3a07b36700 1 -- 172.21.15.3:0/1534610966 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a0010b120 con 0x7f3a000fd2a0 2023-11-17T05:25:19.559 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.546+0000 7f39ee7fc700 1 -- 172.21.15.3:0/1534610966 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39fc01a650 con 0x7f3a000fd2a0 2023-11-17T05:25:19.560 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.546+0000 7f3a07b36700 1 -- 172.21.15.3:0/1534610966 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a0010b530 con 0x7f3a000fd2a0 2023-11-17T05:25:19.622 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.547+0000 7f39ee7fc700 1 -- 172.21.15.3:0/1534610966 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f39fc00f070 con 0x7f3a000fd2a0 2023-11-17T05:25:19.622 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.547+0000 7f39ee7fc700 1 --2- 172.21.15.3:0/1534610966 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7f39e802feb0 0x7f39e8032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:19.622 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.547+0000 7f39ee7fc700 1 -- 172.21.15.3:0/1534610966 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f39fc041380 con 0x7f3a000fd2a0 2023-11-17T05:25:19.622 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.547+0000 7f3a050d1700 1 --2- 172.21.15.3:0/1534610966 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7f39e802feb0 0x7f39e8032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:19.623 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.547+0000 7f3a050d1700 1 --2- 172.21.15.3:0/1534610966 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7f39e802feb0 0x7f39e8032370 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f39f00096a0 tx=0x7f39f0006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:19.623 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.548+0000 7f3a07b36700 1 -- 172.21.15.3:0/1534610966 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a00059100 con 0x7f3a000fd2a0 2023-11-17T05:25:19.624 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.552+0000 7f39ee7fc700 1 -- 172.21.15.3:0/1534610966 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f39fc016070 con 0x7f3a000fd2a0 2023-11-17T05:25:19.718 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.716+0000 7f3a07b36700 1 -- 172.21.15.3:0/1534610966 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f3a00104550 con 0x7f3a000fd2a0 2023-11-17T05:25:19.721 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.720+0000 7f39ee7fc700 1 -- 172.21.15.3:0/1534610966 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v8)=0 v8) v1 ==== 142+0+0 (secure 0 0 0) 0x7f39fc01b410 con 0x7f3a000fd2a0 2023-11-17T05:25:19.723 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.721+0000 7f3a07b36700 1 -- 172.21.15.3:0/1534610966 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7f39e802feb0 msgr2=0x7f39e8032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:19.723 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.721+0000 7f3a07b36700 1 --2- 172.21.15.3:0/1534610966 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7f39e802feb0 0x7f39e8032370 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f39f00096a0 tx=0x7f39f0006b40).stop 2023-11-17T05:25:19.744 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.721+0000 7f3a07b36700 1 -- 172.21.15.3:0/1534610966 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3a000fd2a0 msgr2=0x7f3a0010aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:19.774 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.721+0000 7f3a07b36700 1 --2- 172.21.15.3:0/1534610966 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3a000fd2a0 0x7f3a0010aa30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f39fc006c70 tx=0x7f39fc006ee0).stop 2023-11-17T05:25:19.775 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.722+0000 7f3a07b36700 1 -- 172.21.15.3:0/1534610966 shutdown_connections 2023-11-17T05:25:19.775 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.722+0000 7f3a07b36700 1 --2- 172.21.15.3:0/1534610966 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7f39e802feb0 0x7f39e8032370 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:19.775 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.722+0000 7f3a07b36700 1 --2- 172.21.15.3:0/1534610966 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3a000fd2a0 0x7f3a0010aa30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:19.775 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.722+0000 7f3a07b36700 1 -- 172.21.15.3:0/1534610966 >> 172.21.15.3:0/1534610966 conn(0x7f3a000f5a80 msgr2=0x7f3a000ff300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:19.775 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.722+0000 7f3a07b36700 1 -- 172.21.15.3:0/1534610966 shutdown_connections 2023-11-17T05:25:19.775 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:19.722+0000 7f3a07b36700 1 -- 172.21.15.3:0/1534610966 wait complete. 2023-11-17T05:25:20.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:19 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:18.758568+0000 mgr.smithi003.eddcuc (mgr.14144) 7 : audit [DBG] from='client.14156 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:25:20.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:19 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cephadm 2023-11-17T05:25:18.759131+0000 mgr.smithi003.eddcuc (mgr.14144) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2023-11-17T05:25:20.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:19 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:19.721338+0000 mon.smithi003 (mon.0) 83 : audit [INF] from='client.? 172.21.15.3:0/1534610966' entity='client.admin' 2023-11-17T05:25:20.093 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptcuja8tf:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpk7o1jie2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2023-11-17T05:25:20.469 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.455+0000 7f042cb7d700 1 Processor -- start 2023-11-17T05:25:20.469 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.455+0000 7f042cb7d700 1 -- start start 2023-11-17T05:25:20.469 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.455+0000 7f042cb7d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04280f9f80 0x7f04280fa3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:20.470 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.455+0000 7f042cb7d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04280fa920 con 0x7f04280f9f80 2023-11-17T05:25:20.470 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.455+0000 7f042659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04280f9f80 0x7f04280fa3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:20.471 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.455+0000 7f042659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04280f9f80 0x7f04280fa3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55376/0 (socket says 172.21.15.3:55376) 2023-11-17T05:25:20.471 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.455+0000 7f042659c700 1 -- 172.21.15.3:0/3853837212 learned_addr learned my addr 172.21.15.3:0/3853837212 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:25:20.471 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.456+0000 7f042659c700 1 -- 172.21.15.3:0/3853837212 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04280faa60 con 0x7f04280f9f80 2023-11-17T05:25:20.472 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.456+0000 7f042659c700 1 --2- 172.21.15.3:0/3853837212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04280f9f80 0x7f04280fa3a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f041800b5f0 tx=0x7f0418006ee0).ready entity=mon.0 client_cookie=c8a247288955a97f server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:20.472 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.456+0000 7f042559a700 1 -- 172.21.15.3:0/3853837212 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f041800d600 con 0x7f04280f9f80 2023-11-17T05:25:20.472 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.456+0000 7f042559a700 1 -- 172.21.15.3:0/3853837212 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f041800dbd0 con 0x7f04280f9f80 2023-11-17T05:25:20.473 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.456+0000 7f042559a700 1 -- 172.21.15.3:0/3853837212 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0418011c00 con 0x7f04280f9f80 2023-11-17T05:25:20.473 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.457+0000 7f042cb7d700 1 -- 172.21.15.3:0/3853837212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04280f9f80 msgr2=0x7f04280fa3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:20.474 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.457+0000 7f042cb7d700 1 --2- 172.21.15.3:0/3853837212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04280f9f80 0x7f04280fa3a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f041800b5f0 tx=0x7f0418006ee0).stop 2023-11-17T05:25:20.474 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.457+0000 7f042cb7d700 1 -- 172.21.15.3:0/3853837212 shutdown_connections 2023-11-17T05:25:20.474 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.457+0000 7f042cb7d700 1 --2- 172.21.15.3:0/3853837212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04280f9f80 0x7f04280fa3a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:20.475 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.457+0000 7f042cb7d700 1 -- 172.21.15.3:0/3853837212 >> 172.21.15.3:0/3853837212 conn(0x7f04280f59e0 msgr2=0x7f04280f7e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:20.475 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.457+0000 7f042cb7d700 1 -- 172.21.15.3:0/3853837212 shutdown_connections 2023-11-17T05:25:20.475 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.457+0000 7f042cb7d700 1 -- 172.21.15.3:0/3853837212 wait complete. 2023-11-17T05:25:20.476 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.458+0000 7f042cb7d700 1 Processor -- start 2023-11-17T05:25:20.476 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.458+0000 7f042cb7d700 1 -- start start 2023-11-17T05:25:20.476 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.458+0000 7f042cb7d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04280f9f80 0x7f04280fcb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:20.476 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.458+0000 7f042cb7d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04280faff0 con 0x7f04280f9f80 2023-11-17T05:25:20.476 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.458+0000 7f042659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04280f9f80 0x7f04280fcb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:20.477 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.458+0000 7f042659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04280f9f80 0x7f04280fcb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55390/0 (socket says 172.21.15.3:55390) 2023-11-17T05:25:20.477 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.458+0000 7f042659c700 1 -- 172.21.15.3:0/4233187025 learned_addr learned my addr 172.21.15.3:0/4233187025 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:25:20.477 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.458+0000 7f042659c700 1 -- 172.21.15.3:0/4233187025 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f041800b200 con 0x7f04280f9f80 2023-11-17T05:25:20.477 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.459+0000 7f042659c700 1 --2- 172.21.15.3:0/4233187025 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04280f9f80 0x7f04280fcb70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f0418002f40 tx=0x7f0418006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:20.477 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.459+0000 7f0416ffd700 1 -- 172.21.15.3:0/4233187025 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0418011690 con 0x7f04280f9f80 2023-11-17T05:25:20.477 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.459+0000 7f0416ffd700 1 -- 172.21.15.3:0/4233187025 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f041801b880 con 0x7f04280f9f80 2023-11-17T05:25:20.477 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.459+0000 7f042cb7d700 1 -- 172.21.15.3:0/4233187025 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04280fb190 con 0x7f04280f9f80 2023-11-17T05:25:20.478 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.459+0000 7f0416ffd700 1 -- 172.21.15.3:0/4233187025 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f041801a6b0 con 0x7f04280f9f80 2023-11-17T05:25:20.479 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.459+0000 7f042cb7d700 1 -- 172.21.15.3:0/4233187025 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04280fb5a0 con 0x7f04280f9f80 2023-11-17T05:25:20.479 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.460+0000 7f0416ffd700 1 -- 172.21.15.3:0/4233187025 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f041800f070 con 0x7f04280f9f80 2023-11-17T05:25:20.479 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.460+0000 7f042cb7d700 1 -- 172.21.15.3:0/4233187025 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0428045ad0 con 0x7f04280f9f80 2023-11-17T05:25:20.481 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.460+0000 7f0416ffd700 1 --2- 172.21.15.3:0/4233187025 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7f040c02fe60 0x7f040c032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:20.481 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.460+0000 7f0416ffd700 1 -- 172.21.15.3:0/4233187025 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f04180412f0 con 0x7f04280f9f80 2023-11-17T05:25:20.481 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.460+0000 7f0425d9b700 1 --2- 172.21.15.3:0/4233187025 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7f040c02fe60 0x7f040c032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:20.481 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.460+0000 7f0425d9b700 1 --2- 172.21.15.3:0/4233187025 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7f040c02fe60 0x7f040c032320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f041c0096a0 tx=0x7f041c006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:20.481 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.464+0000 7f0416ffd700 1 -- 172.21.15.3:0/4233187025 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f04180198d0 con 0x7f04280f9f80 2023-11-17T05:25:20.632 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.630+0000 7f042cb7d700 1 -- 172.21.15.3:0/4233187025 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f04280ffbe0 con 0x7f04280f9f80 2023-11-17T05:25:20.634 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.633+0000 7f0416ffd700 1 -- 172.21.15.3:0/4233187025 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v9)=0 v9) v1 ==== 130+0+0 (secure 0 0 0) 0x7f0418016070 con 0x7f04280f9f80 2023-11-17T05:25:20.637 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.635+0000 7f042cb7d700 1 -- 172.21.15.3:0/4233187025 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7f040c02fe60 msgr2=0x7f040c032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:20.637 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.635+0000 7f042cb7d700 1 --2- 172.21.15.3:0/4233187025 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7f040c02fe60 0x7f040c032320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f041c0096a0 tx=0x7f041c006b40).stop 2023-11-17T05:25:20.637 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.635+0000 7f042cb7d700 1 -- 172.21.15.3:0/4233187025 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04280f9f80 msgr2=0x7f04280fcb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:20.637 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.635+0000 7f042cb7d700 1 --2- 172.21.15.3:0/4233187025 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04280f9f80 0x7f04280fcb70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f0418002f40 tx=0x7f0418006ee0).stop 2023-11-17T05:25:20.638 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.635+0000 7f042cb7d700 1 -- 172.21.15.3:0/4233187025 shutdown_connections 2023-11-17T05:25:20.638 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.635+0000 7f042cb7d700 1 --2- 172.21.15.3:0/4233187025 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7f040c02fe60 0x7f040c032320 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:20.638 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.635+0000 7f042cb7d700 1 --2- 172.21.15.3:0/4233187025 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04280f9f80 0x7f04280fcb70 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:20.638 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.635+0000 7f042cb7d700 1 -- 172.21.15.3:0/4233187025 >> 172.21.15.3:0/4233187025 conn(0x7f04280f59e0 msgr2=0x7f04280f76c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:20.638 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.636+0000 7f042cb7d700 1 -- 172.21.15.3:0/4233187025 shutdown_connections 2023-11-17T05:25:20.638 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:20.636+0000 7f042cb7d700 1 -- 172.21.15.3:0/4233187025 wait complete. 2023-11-17T05:25:20.992 INFO:teuthology.orchestra.run.smithi003.stderr:Enabling the dashboard module... 2023-11-17T05:25:20.993 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptcuja8tf:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpk7o1jie2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2023-11-17T05:25:21.372 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.370+0000 7ffb365b7700 1 Processor -- start 2023-11-17T05:25:21.372 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.370+0000 7ffb365b7700 1 -- start start 2023-11-17T05:25:21.372 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.370+0000 7ffb365b7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb30071cb0 0x7ffb30070250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:21.372 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.370+0000 7ffb365b7700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb300720d0 con 0x7ffb30071cb0 2023-11-17T05:25:21.373 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.371+0000 7ffb355b5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb30071cb0 0x7ffb30070250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:21.373 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.371+0000 7ffb355b5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb30071cb0 0x7ffb30070250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55396/0 (socket says 172.21.15.3:55396) 2023-11-17T05:25:21.374 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.371+0000 7ffb355b5700 1 -- 172.21.15.3:0/3793247634 learned_addr learned my addr 172.21.15.3:0/3793247634 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:25:21.374 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.371+0000 7ffb355b5700 1 -- 172.21.15.3:0/3793247634 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb30070740 con 0x7ffb30071cb0 2023-11-17T05:25:21.374 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.371+0000 7ffb355b5700 1 --2- 172.21.15.3:0/3793247634 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb30071cb0 0x7ffb30070250 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7ffb2400b5f0 tx=0x7ffb24006ee0).ready entity=mon.0 client_cookie=177af4e634dd17de server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:21.374 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.371+0000 7ffb23fff700 1 -- 172.21.15.3:0/3793247634 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb2400d600 con 0x7ffb30071cb0 2023-11-17T05:25:21.374 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.371+0000 7ffb23fff700 1 -- 172.21.15.3:0/3793247634 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ffb2400dbd0 con 0x7ffb30071cb0 2023-11-17T05:25:21.374 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.371+0000 7ffb23fff700 1 -- 172.21.15.3:0/3793247634 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb24011c00 con 0x7ffb30071cb0 2023-11-17T05:25:21.374 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.372+0000 7ffb365b7700 1 -- 172.21.15.3:0/3793247634 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb30071cb0 msgr2=0x7ffb30070250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:21.374 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.372+0000 7ffb365b7700 1 --2- 172.21.15.3:0/3793247634 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb30071cb0 0x7ffb30070250 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7ffb2400b5f0 tx=0x7ffb24006ee0).stop 2023-11-17T05:25:21.374 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.372+0000 7ffb365b7700 1 -- 172.21.15.3:0/3793247634 shutdown_connections 2023-11-17T05:25:21.374 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.372+0000 7ffb365b7700 1 --2- 172.21.15.3:0/3793247634 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb30071cb0 0x7ffb30070250 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:21.374 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.372+0000 7ffb365b7700 1 -- 172.21.15.3:0/3793247634 >> 172.21.15.3:0/3793247634 conn(0x7ffb300f71e0 msgr2=0x7ffb300f9600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:21.376 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.373+0000 7ffb365b7700 1 -- 172.21.15.3:0/3793247634 shutdown_connections 2023-11-17T05:25:21.376 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.373+0000 7ffb365b7700 1 -- 172.21.15.3:0/3793247634 wait complete. 2023-11-17T05:25:21.376 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.373+0000 7ffb365b7700 1 Processor -- start 2023-11-17T05:25:21.377 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.373+0000 7ffb365b7700 1 -- start start 2023-11-17T05:25:21.377 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.373+0000 7ffb365b7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb30071cb0 0x7ffb3010c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:21.377 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.373+0000 7ffb365b7700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb3010ced0 con 0x7ffb30071cb0 2023-11-17T05:25:21.377 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.373+0000 7ffb355b5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb30071cb0 0x7ffb3010c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:21.377 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.374+0000 7ffb355b5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb30071cb0 0x7ffb3010c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55406/0 (socket says 172.21.15.3:55406) 2023-11-17T05:25:21.377 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.374+0000 7ffb355b5700 1 -- 172.21.15.3:0/1125902663 learned_addr learned my addr 172.21.15.3:0/1125902663 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:25:21.414 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.374+0000 7ffb355b5700 1 -- 172.21.15.3:0/1125902663 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb2400b200 con 0x7ffb30071cb0 2023-11-17T05:25:21.415 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.374+0000 7ffb355b5700 1 --2- 172.21.15.3:0/1125902663 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb30071cb0 0x7ffb3010c9e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7ffb2400e040 tx=0x7ffb24006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:21.415 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.374+0000 7ffb21ffb700 1 -- 172.21.15.3:0/1125902663 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb2400d9f0 con 0x7ffb30071cb0 2023-11-17T05:25:21.415 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.374+0000 7ffb21ffb700 1 -- 172.21.15.3:0/1125902663 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ffb2401a9f0 con 0x7ffb30071cb0 2023-11-17T05:25:21.415 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.374+0000 7ffb21ffb700 1 -- 172.21.15.3:0/1125902663 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb2401b8c0 con 0x7ffb30071cb0 2023-11-17T05:25:21.415 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.374+0000 7ffb365b7700 1 -- 172.21.15.3:0/1125902663 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb3010d0d0 con 0x7ffb30071cb0 2023-11-17T05:25:21.416 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.375+0000 7ffb365b7700 1 -- 172.21.15.3:0/1125902663 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb3010d4e0 con 0x7ffb30071cb0 2023-11-17T05:25:21.416 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.375+0000 7ffb21ffb700 1 -- 172.21.15.3:0/1125902663 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7ffb2400f070 con 0x7ffb30071cb0 2023-11-17T05:25:21.416 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.376+0000 7ffb21ffb700 1 --2- 172.21.15.3:0/1125902663 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7ffb1802fe60 0x7ffb18032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:21.416 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.376+0000 7ffb21ffb700 1 -- 172.21.15.3:0/1125902663 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7ffb24041440 con 0x7ffb30071cb0 2023-11-17T05:25:21.417 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.376+0000 7ffb365b7700 1 -- 172.21.15.3:0/1125902663 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb30045ad0 con 0x7ffb30071cb0 2023-11-17T05:25:21.417 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.376+0000 7ffb34db4700 1 --2- 172.21.15.3:0/1125902663 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7ffb1802fe60 0x7ffb18032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:21.417 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.376+0000 7ffb34db4700 1 --2- 172.21.15.3:0/1125902663 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7ffb1802fe60 0x7ffb18032320 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7ffb2c0096a0 tx=0x7ffb2c006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:21.417 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.380+0000 7ffb21ffb700 1 -- 172.21.15.3:0/1125902663 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffb24016070 con 0x7ffb30071cb0 2023-11-17T05:25:21.588 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.586+0000 7ffb365b7700 1 -- 172.21.15.3:0/1125902663 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7ffb30101b90 con 0x7ffb30071cb0 2023-11-17T05:25:21.634 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.633+0000 7ffb21ffb700 1 -- 172.21.15.3:0/1125902663 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7ffb24022d80 con 0x7ffb30071cb0 2023-11-17T05:25:21.636 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.634+0000 7ffb21ffb700 1 -- 172.21.15.3:0/1125902663 <== mon.0 v2:172.21.15.3:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v13) v1 ==== 88+0+0 (secure 0 0 0) 0x7ffb2402ee30 con 0x7ffb30071cb0 2023-11-17T05:25:21.639 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.637+0000 7ffb365b7700 1 -- 172.21.15.3:0/1125902663 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7ffb1802fe60 msgr2=0x7ffb18032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:21.639 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.637+0000 7ffb365b7700 1 --2- 172.21.15.3:0/1125902663 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7ffb1802fe60 0x7ffb18032320 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7ffb2c0096a0 tx=0x7ffb2c006b40).stop 2023-11-17T05:25:21.639 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.637+0000 7ffb365b7700 1 -- 172.21.15.3:0/1125902663 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb30071cb0 msgr2=0x7ffb3010c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:21.639 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.637+0000 7ffb365b7700 1 --2- 172.21.15.3:0/1125902663 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb30071cb0 0x7ffb3010c9e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7ffb2400e040 tx=0x7ffb24006ee0).stop 2023-11-17T05:25:21.639 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.637+0000 7ffb365b7700 1 -- 172.21.15.3:0/1125902663 shutdown_connections 2023-11-17T05:25:21.640 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.637+0000 7ffb365b7700 1 --2- 172.21.15.3:0/1125902663 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7ffb1802fe60 0x7ffb18032320 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:21.640 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.637+0000 7ffb365b7700 1 --2- 172.21.15.3:0/1125902663 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffb30071cb0 0x7ffb3010c9e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:21.640 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.637+0000 7ffb365b7700 1 -- 172.21.15.3:0/1125902663 >> 172.21.15.3:0/1125902663 conn(0x7ffb300f71e0 msgr2=0x7ffb300f8d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:21.640 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.638+0000 7ffb365b7700 1 -- 172.21.15.3:0/1125902663 shutdown_connections 2023-11-17T05:25:21.640 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:21.638+0000 7ffb365b7700 1 -- 172.21.15.3:0/1125902663 wait complete. 2023-11-17T05:25:22.012 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptcuja8tf:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpk7o1jie2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2023-11-17T05:25:22.015 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:21 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:20.635001+0000 mon.smithi003 (mon.0) 84 : audit [INF] from='client.? 172.21.15.3:0/4233187025' entity='client.admin' 2023-11-17T05:25:22.015 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:21 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:21.588408+0000 mon.smithi003 (mon.0) 85 : audit [INF] from='client.? 172.21.15.3:0/1125902663' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-11-17T05:25:22.538 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.536+0000 7f014c6c9700 1 Processor -- start 2023-11-17T05:25:22.538 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.536+0000 7f014c6c9700 1 -- start start 2023-11-17T05:25:22.538 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.536+0000 7f014c6c9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f014406fc50 0x7f0144100c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:22.538 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.536+0000 7f014c6c9700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01440fc6a0 con 0x7f014406fc50 2023-11-17T05:25:22.539 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.537+0000 7f014a465700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f014406fc50 0x7f0144100c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:22.539 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.537+0000 7f014a465700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f014406fc50 0x7f0144100c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55438/0 (socket says 172.21.15.3:55438) 2023-11-17T05:25:22.539 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.537+0000 7f014a465700 1 -- 172.21.15.3:0/3513930543 learned_addr learned my addr 172.21.15.3:0/3513930543 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:25:22.539 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.537+0000 7f014a465700 1 -- 172.21.15.3:0/3513930543 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0144101110 con 0x7f014406fc50 2023-11-17T05:25:22.540 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.538+0000 7f014a465700 1 --2- 172.21.15.3:0/3513930543 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f014406fc50 0x7f0144100c20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f014000b5f0 tx=0x7f0140006ee0).ready entity=mon.0 client_cookie=8ef157eb51c34abe server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:22.540 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.538+0000 7f0149463700 1 -- 172.21.15.3:0/3513930543 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f014000d600 con 0x7f014406fc50 2023-11-17T05:25:22.540 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.538+0000 7f0149463700 1 -- 172.21.15.3:0/3513930543 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f014000dbd0 con 0x7f014406fc50 2023-11-17T05:25:22.540 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.538+0000 7f0149463700 1 -- 172.21.15.3:0/3513930543 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0140011c00 con 0x7f014406fc50 2023-11-17T05:25:22.540 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.539+0000 7f014c6c9700 1 -- 172.21.15.3:0/3513930543 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f014406fc50 msgr2=0x7f0144100c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:22.541 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.539+0000 7f014c6c9700 1 --2- 172.21.15.3:0/3513930543 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f014406fc50 0x7f0144100c20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f014000b5f0 tx=0x7f0140006ee0).stop 2023-11-17T05:25:22.541 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.539+0000 7f014c6c9700 1 -- 172.21.15.3:0/3513930543 shutdown_connections 2023-11-17T05:25:22.541 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.539+0000 7f014c6c9700 1 --2- 172.21.15.3:0/3513930543 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f014406fc50 0x7f0144100c20 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:22.541 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.539+0000 7f014c6c9700 1 -- 172.21.15.3:0/3513930543 >> 172.21.15.3:0/3513930543 conn(0x7f01440f6280 msgr2=0x7f01440f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:22.541 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.539+0000 7f014c6c9700 1 -- 172.21.15.3:0/3513930543 shutdown_connections 2023-11-17T05:25:22.541 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.539+0000 7f014c6c9700 1 -- 172.21.15.3:0/3513930543 wait complete. 2023-11-17T05:25:22.543 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.539+0000 7f014c6c9700 1 Processor -- start 2023-11-17T05:25:22.543 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.539+0000 7f014c6c9700 1 -- start start 2023-11-17T05:25:22.543 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.540+0000 7f014c6c9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f014406fc50 0x7f014410cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:22.543 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.540+0000 7f014c6c9700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f014410d0c0 con 0x7f014406fc50 2023-11-17T05:25:22.543 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.540+0000 7f014a465700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f014406fc50 0x7f014410cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:22.543 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.540+0000 7f014a465700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f014406fc50 0x7f014410cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55452/0 (socket says 172.21.15.3:55452) 2023-11-17T05:25:22.543 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.540+0000 7f014a465700 1 -- 172.21.15.3:0/197701154 learned_addr learned my addr 172.21.15.3:0/197701154 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:25:22.543 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.540+0000 7f014a465700 1 -- 172.21.15.3:0/197701154 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f014000b200 con 0x7f014406fc50 2023-11-17T05:25:22.543 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.540+0000 7f014a465700 1 --2- 172.21.15.3:0/197701154 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f014406fc50 0x7f014410cbd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f014000b1d0 tx=0x7f0140006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:22.543 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.541+0000 7f0132ffd700 1 -- 172.21.15.3:0/197701154 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0140011690 con 0x7f014406fc50 2023-11-17T05:25:22.545 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.541+0000 7f0132ffd700 1 -- 172.21.15.3:0/197701154 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f014001b880 con 0x7f014406fc50 2023-11-17T05:25:22.545 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.541+0000 7f014c6c9700 1 -- 172.21.15.3:0/197701154 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f014410d2c0 con 0x7f014406fc50 2023-11-17T05:25:22.545 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.541+0000 7f0132ffd700 1 -- 172.21.15.3:0/197701154 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f014001a650 con 0x7f014406fc50 2023-11-17T05:25:22.545 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.541+0000 7f014c6c9700 1 -- 172.21.15.3:0/197701154 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f014410d6d0 con 0x7f014406fc50 2023-11-17T05:25:22.545 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.542+0000 7f0132ffd700 1 -- 172.21.15.3:0/197701154 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f014000f070 con 0x7f014406fc50 2023-11-17T05:25:22.545 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.542+0000 7f0132ffd700 1 --2- 172.21.15.3:0/197701154 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7f012c02ff00 0x7f012c0323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:22.545 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.542+0000 7f0132ffd700 1 -- 172.21.15.3:0/197701154 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f0140041170 con 0x7f014406fc50 2023-11-17T05:25:22.545 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.542+0000 7f0149c64700 1 -- 172.21.15.3:0/197701154 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7f012c02ff00 msgr2=0x7f012c0323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/2830354403 2023-11-17T05:25:22.545 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.542+0000 7f0149c64700 1 --2- 172.21.15.3:0/197701154 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7f012c02ff00 0x7f012c0323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-17T05:25:22.545 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.542+0000 7f014c6c9700 1 -- 172.21.15.3:0/197701154 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0144059100 con 0x7f014406fc50 2023-11-17T05:25:22.548 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.547+0000 7f0132ffd700 1 -- 172.21.15.3:0/197701154 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0140016070 con 0x7f014406fc50 2023-11-17T05:25:22.744 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.743+0000 7f0149c64700 1 -- 172.21.15.3:0/197701154 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7f012c02ff00 msgr2=0x7f012c0323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/2830354403 2023-11-17T05:25:22.745 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.743+0000 7f0149c64700 1 --2- 172.21.15.3:0/197701154 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7f012c02ff00 0x7f012c0323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-17T05:25:22.802 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.800+0000 7f014c6c9700 1 -- 172.21.15.3:0/197701154 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f0144106af0 con 0x7f014406fc50 2023-11-17T05:25:22.803 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.801+0000 7f0132ffd700 1 -- 172.21.15.3:0/197701154 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v13) v1 ==== 56+0+104 (secure 0 0 0) 0x7f01400224f0 con 0x7f014406fc50 2023-11-17T05:25:22.803 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: { 2023-11-17T05:25:22.803 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "epoch": 13, 2023-11-17T05:25:22.803 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "available": true, 2023-11-17T05:25:22.803 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "active_name": "smithi003.eddcuc", 2023-11-17T05:25:22.804 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "num_standby": 0 2023-11-17T05:25:22.804 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: } 2023-11-17T05:25:22.805 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.802+0000 7f014c6c9700 1 -- 172.21.15.3:0/197701154 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7f012c02ff00 msgr2=0x7f012c0323c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-17T05:25:22.805 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.802+0000 7f014c6c9700 1 --2- 172.21.15.3:0/197701154 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7f012c02ff00 0x7f012c0323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-17T05:25:22.805 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.802+0000 7f014c6c9700 1 -- 172.21.15.3:0/197701154 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f014406fc50 msgr2=0x7f014410cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:22.805 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.802+0000 7f014c6c9700 1 --2- 172.21.15.3:0/197701154 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f014406fc50 0x7f014410cbd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f014000b1d0 tx=0x7f0140006ee0).stop 2023-11-17T05:25:22.805 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.803+0000 7f014c6c9700 1 -- 172.21.15.3:0/197701154 shutdown_connections 2023-11-17T05:25:22.805 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.803+0000 7f014c6c9700 1 --2- 172.21.15.3:0/197701154 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7f012c02ff00 0x7f012c0323c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-17T05:25:22.805 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.803+0000 7f014c6c9700 1 --2- 172.21.15.3:0/197701154 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f014406fc50 0x7f014410cbd0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:22.806 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.803+0000 7f014c6c9700 1 -- 172.21.15.3:0/197701154 >> 172.21.15.3:0/197701154 conn(0x7f01440f6280 msgr2=0x7f01440f7e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:22.806 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.803+0000 7f014c6c9700 1 -- 172.21.15.3:0/197701154 shutdown_connections 2023-11-17T05:25:22.806 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:22.803+0000 7f014c6c9700 1 -- 172.21.15.3:0/197701154 wait complete. 2023-11-17T05:25:22.961 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:22 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:21.636195+0000 mon.smithi003 (mon.0) 86 : audit [INF] from='client.? 172.21.15.3:0/1125902663' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-11-17T05:25:23.021 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:22 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:25:21.636256+0000 mon.smithi003 (mon.0) 87 : cluster [DBG] mgrmap e13: smithi003.eddcuc(active, since 6s) 2023-11-17T05:25:23.284 INFO:teuthology.orchestra.run.smithi003.stderr:Waiting for the mgr to restart... 2023-11-17T05:25:23.285 INFO:teuthology.orchestra.run.smithi003.stderr:Waiting for mgr epoch 13... 2023-11-17T05:25:23.285 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptcuja8tf:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpk7o1jie2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2023-11-17T05:25:23.684 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:23.682+0000 7fcf9becf700 1 Processor -- start 2023-11-17T05:25:23.696 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:23.682+0000 7fcf9becf700 1 -- start start 2023-11-17T05:25:23.696 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:23.682+0000 7fcf9becf700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf9406fc70 0x7fcf94100c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:23.696 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:23.682+0000 7fcf9becf700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf940fc680 con 0x7fcf9406fc70 2023-11-17T05:25:23.696 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:23.683+0000 7fcf99c6b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf9406fc70 0x7fcf94100c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:23.696 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:23.683+0000 7fcf99c6b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf9406fc70 0x7fcf94100c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55466/0 (socket says 172.21.15.3:55466) 2023-11-17T05:25:23.697 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:23.683+0000 7fcf99c6b700 1 -- 172.21.15.3:0/1804748318 learned_addr learned my addr 172.21.15.3:0/1804748318 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:25:23.697 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:23.683+0000 7fcf99c6b700 1 -- 172.21.15.3:0/1804748318 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf941010f0 con 0x7fcf9406fc70 2023-11-17T05:25:23.697 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:23.683+0000 7fcf99c6b700 1 --2- 172.21.15.3:0/1804748318 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf9406fc70 0x7fcf94100c00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fcf9000b5f0 tx=0x7fcf90006ee0).ready entity=mon.0 client_cookie=7b4647e1191af261 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:23.697 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:23.683+0000 7fcf98c69700 1 -- 172.21.15.3:0/1804748318 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf9000d600 con 0x7fcf9406fc70 2023-11-17T05:25:23.697 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:23.683+0000 7fcf98c69700 1 -- 172.21.15.3:0/1804748318 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fcf9000dbd0 con 0x7fcf9406fc70 2023-11-17T05:25:23.697 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:23.683+0000 7fcf98c69700 1 -- 172.21.15.3:0/1804748318 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf90011c00 con 0x7fcf9406fc70 2023-11-17T05:25:23.697 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:23.684+0000 7fcf9becf700 1 -- 172.21.15.3:0/1804748318 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf9406fc70 msgr2=0x7fcf94100c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:23.697 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:23.684+0000 7fcf9becf700 1 --2- 172.21.15.3:0/1804748318 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf9406fc70 0x7fcf94100c00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fcf9000b5f0 tx=0x7fcf90006ee0).stop 2023-11-17T05:25:23.697 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:23.684+0000 7fcf9becf700 1 -- 172.21.15.3:0/1804748318 shutdown_connections 2023-11-17T05:25:23.697 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:23.684+0000 7fcf9becf700 1 --2- 172.21.15.3:0/1804748318 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf9406fc70 0x7fcf94100c00 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:23.697 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:23.684+0000 7fcf9becf700 1 -- 172.21.15.3:0/1804748318 >> 172.21.15.3:0/1804748318 conn(0x7fcf940f6280 msgr2=0x7fcf940f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:23.697 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:23.684+0000 7fcf9becf700 1 -- 172.21.15.3:0/1804748318 shutdown_connections 2023-11-17T05:25:23.697 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:23.684+0000 7fcf9becf700 1 -- 172.21.15.3:0/1804748318 wait complete. 2023-11-17T05:25:23.698 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:23.685+0000 7fcf9becf700 1 Processor -- start 2023-11-17T05:25:23.698 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:23.685+0000 7fcf9becf700 1 -- start start 2023-11-17T05:25:23.698 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:23.685+0000 7fcf9becf700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf9406fc70 0x7fcf9410cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:23.698 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:23.685+0000 7fcf9becf700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9410d090 con 0x7fcf9406fc70 2023-11-17T05:25:23.698 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:23.685+0000 7fcf99c6b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf9406fc70 0x7fcf9410cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:23.698 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:23.685+0000 7fcf99c6b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf9406fc70 0x7fcf9410cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55474/0 (socket says 172.21.15.3:55474) 2023-11-17T05:25:23.698 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:23.685+0000 7fcf99c6b700 1 -- 172.21.15.3:0/14640177 learned_addr learned my addr 172.21.15.3:0/14640177 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:25:23.698 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:23.686+0000 7fcf99c6b700 1 -- 172.21.15.3:0/14640177 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf9000b200 con 0x7fcf9406fc70 2023-11-17T05:25:23.698 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:23.686+0000 7fcf99c6b700 1 --2- 172.21.15.3:0/14640177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf9406fc70 0x7fcf9410cba0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fcf9000e040 tx=0x7fcf90006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:23.698 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:23.686+0000 7fcf827fc700 1 -- 172.21.15.3:0/14640177 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf9001b410 con 0x7fcf9406fc70 2023-11-17T05:25:23.698 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:23.686+0000 7fcf827fc700 1 -- 172.21.15.3:0/14640177 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fcf9001b9e0 con 0x7fcf9406fc70 2023-11-17T05:25:23.729 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:23.686+0000 7fcf9becf700 1 -- 172.21.15.3:0/14640177 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf9410d290 con 0x7fcf9406fc70 2023-11-17T05:25:23.788 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:23.686+0000 7fcf827fc700 1 -- 172.21.15.3:0/14640177 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf9001a770 con 0x7fcf9406fc70 2023-11-17T05:25:23.874 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:23.686+0000 7fcf9becf700 1 -- 172.21.15.3:0/14640177 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf9410d6a0 con 0x7fcf9406fc70 2023-11-17T05:25:23.874 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:23.687+0000 7fcf827fc700 1 -- 172.21.15.3:0/14640177 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fcf9000f070 con 0x7fcf9406fc70 2023-11-17T05:25:23.874 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:23.687+0000 7fcf827fc700 1 --2- 172.21.15.3:0/14640177 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7fcf7c02feb0 0x7fcf7c032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:23.874 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:23.687+0000 7fcf9becf700 1 -- 172.21.15.3:0/14640177 --> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcf94059100 con 0x7fcf7c02feb0 2023-11-17T05:25:23.874 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:23.687+0000 7fcf827fc700 1 -- 172.21.15.3:0/14640177 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fcf90041200 con 0x7fcf9406fc70 2023-11-17T05:25:23.874 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:23.687+0000 7fcf9946a700 1 -- 172.21.15.3:0/14640177 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7fcf7c02feb0 msgr2=0x7fcf7c032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/2830354403 2023-11-17T05:25:23.874 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:23.687+0000 7fcf9946a700 1 --2- 172.21.15.3:0/14640177 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7fcf7c02feb0 0x7fcf7c032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-17T05:25:23.889 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:23.888+0000 7fcf9946a700 1 -- 172.21.15.3:0/14640177 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7fcf7c02feb0 msgr2=0x7fcf7c032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/2830354403 2023-11-17T05:25:23.889 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:23.888+0000 7fcf9946a700 1 --2- 172.21.15.3:0/14640177 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7fcf7c02feb0 0x7fcf7c032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-17T05:25:24.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:23 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:22.802510+0000 mon.smithi003 (mon.0) 88 : audit [DBG] from='client.? 172.21.15.3:0/197701154' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-11-17T05:25:24.290 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:24.289+0000 7fcf9946a700 1 -- 172.21.15.3:0/14640177 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7fcf7c02feb0 msgr2=0x7fcf7c032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/2830354403 2023-11-17T05:25:24.291 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:24.289+0000 7fcf9946a700 1 --2- 172.21.15.3:0/14640177 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7fcf7c02feb0 0x7fcf7c032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-11-17T05:25:25.091 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:25.090+0000 7fcf9946a700 1 -- 172.21.15.3:0/14640177 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7fcf7c02feb0 msgr2=0x7fcf7c032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/2830354403 2023-11-17T05:25:25.091 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:25.090+0000 7fcf9946a700 1 --2- 172.21.15.3:0/14640177 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7fcf7c02feb0 0x7fcf7c032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-11-17T05:25:26.693 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:26.691+0000 7fcf9946a700 1 -- 172.21.15.3:0/14640177 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7fcf7c02feb0 msgr2=0x7fcf7c032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/2830354403 2023-11-17T05:25:26.693 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:26.691+0000 7fcf9946a700 1 --2- 172.21.15.3:0/14640177 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7fcf7c02feb0 0x7fcf7c032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-11-17T05:25:27.386 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:27.384+0000 7fcf827fc700 1 -- 172.21.15.3:0/14640177 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mgrmap(e 14) v1 ==== 35997+0+0 (secure 0 0 0) 0x7fcf90024790 con 0x7fcf9406fc70 2023-11-17T05:25:27.386 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:27.384+0000 7fcf827fc700 1 -- 172.21.15.3:0/14640177 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7fcf7c02feb0 msgr2=0x7fcf7c032370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-17T05:25:27.386 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:27.384+0000 7fcf827fc700 1 --2- 172.21.15.3:0/14640177 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7fcf7c02feb0 0x7fcf7c032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-17T05:25:27.650 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:27 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:25:27.333287+0000 mon.smithi003 (mon.0) 89 : cluster [INF] Active manager daemon smithi003.eddcuc restarted 2023-11-17T05:25:27.650 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:27 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:25:27.333733+0000 mon.smithi003 (mon.0) 90 : cluster [INF] Activating manager daemon smithi003.eddcuc 2023-11-17T05:25:27.651 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:27 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:25:27.335661+0000 mon.smithi003 (mon.0) 91 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-11-17T05:25:28.395 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:28.391+0000 7fcf827fc700 1 -- 172.21.15.3:0/14640177 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7fcf90019300 con 0x7fcf9406fc70 2023-11-17T05:25:28.396 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:28.392+0000 7fcf827fc700 1 --2- 172.21.15.3:0/14640177 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fcf7c033990 0x7fcf7c035d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:28.396 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:28.392+0000 7fcf827fc700 1 -- 172.21.15.3:0/14640177 --> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcf94059100 con 0x7fcf7c033990 2023-11-17T05:25:28.396 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:28.393+0000 7fcf9946a700 1 --2- 172.21.15.3:0/14640177 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fcf7c033990 0x7fcf7c035d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:28.396 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:28.394+0000 7fcf9946a700 1 --2- 172.21.15.3:0/14640177 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fcf7c033990 0x7fcf7c035d80 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fcf84006fd0 tx=0x7fcf84006cf0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:28.396 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:28.394+0000 7fcf827fc700 1 -- 172.21.15.3:0/14640177 <== mgr.14164 v2:172.21.15.3:6800/17534166 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7fcf94059100 con 0x7fcf7c033990 2023-11-17T05:25:28.410 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:28.401+0000 7fcf9becf700 1 -- 172.21.15.3:0/14640177 --> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fcf94045ad0 con 0x7fcf7c033990 2023-11-17T05:25:28.411 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:28.401+0000 7fcf827fc700 1 -- 172.21.15.3:0/14640177 <== mgr.14164 v2:172.21.15.3:6800/17534166 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fcf94045ad0 con 0x7fcf7c033990 2023-11-17T05:25:28.411 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: { 2023-11-17T05:25:28.411 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "mgrmap_epoch": 15, 2023-11-17T05:25:28.412 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: "initialized": true 2023-11-17T05:25:28.413 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: } 2023-11-17T05:25:28.413 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:28.402+0000 7fcf9becf700 1 -- 172.21.15.3:0/14640177 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fcf7c033990 msgr2=0x7fcf7c035d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:28.413 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:28.402+0000 7fcf9becf700 1 --2- 172.21.15.3:0/14640177 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fcf7c033990 0x7fcf7c035d80 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fcf84006fd0 tx=0x7fcf84006cf0).stop 2023-11-17T05:25:28.414 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:28.402+0000 7fcf9becf700 1 -- 172.21.15.3:0/14640177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf9406fc70 msgr2=0x7fcf9410cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:28.414 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:28.402+0000 7fcf9becf700 1 --2- 172.21.15.3:0/14640177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf9406fc70 0x7fcf9410cba0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fcf9000e040 tx=0x7fcf90006ee0).stop 2023-11-17T05:25:28.415 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:28.402+0000 7fcf9becf700 1 -- 172.21.15.3:0/14640177 shutdown_connections 2023-11-17T05:25:28.415 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:28.402+0000 7fcf9becf700 1 --2- 172.21.15.3:0/14640177 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fcf7c033990 0x7fcf7c035d80 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:28.415 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:28.402+0000 7fcf9becf700 1 --2- 172.21.15.3:0/14640177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf9406fc70 0x7fcf9410cba0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:28.416 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:28.402+0000 7fcf9becf700 1 --2- 172.21.15.3:0/14640177 >> [v2:172.21.15.3:6800/2830354403,v1:172.21.15.3:6801/2830354403] conn(0x7fcf7c02feb0 0x7fcf7c032370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-17T05:25:28.416 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:28.402+0000 7fcf9becf700 1 -- 172.21.15.3:0/14640177 >> 172.21.15.3:0/14640177 conn(0x7fcf940f6280 msgr2=0x7fcf940f7e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:28.416 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:28.402+0000 7fcf9becf700 1 -- 172.21.15.3:0/14640177 shutdown_connections 2023-11-17T05:25:28.417 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:28.402+0000 7fcf9becf700 1 -- 172.21.15.3:0/14640177 wait complete. 2023-11-17T05:25:28.700 INFO:teuthology.orchestra.run.smithi003.stderr:mgr epoch 13 is available 2023-11-17T05:25:28.700 INFO:teuthology.orchestra.run.smithi003.stderr:Generating a dashboard self-signed certificate... 2023-11-17T05:25:28.700 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptcuja8tf:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpk7o1jie2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2023-11-17T05:25:28.703 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:25:27.386439+0000 mon.smithi003 (mon.0) 92 : cluster [DBG] mgrmap e14: smithi003.eddcuc(active, starting, since 0.0528004s) 2023-11-17T05:25:28.703 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:27.389523+0000 mon.smithi003 (mon.0) 93 : audit [DBG] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' cmd=[{"prefix": "mon metadata", "id": "smithi003"}]: dispatch 2023-11-17T05:25:28.703 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:27.389826+0000 mon.smithi003 (mon.0) 94 : audit [DBG] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' cmd=[{"prefix": "mgr metadata", "who": "smithi003.eddcuc", "id": "smithi003.eddcuc"}]: dispatch 2023-11-17T05:25:28.703 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:27.390117+0000 mon.smithi003 (mon.0) 95 : audit [DBG] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-17T05:25:28.703 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:27.390270+0000 mon.smithi003 (mon.0) 96 : audit [DBG] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-17T05:25:28.703 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:27.390429+0000 mon.smithi003 (mon.0) 97 : audit [DBG] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-17T05:25:28.703 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:25:27.404251+0000 mon.smithi003 (mon.0) 98 : cluster [INF] Manager daemon smithi003.eddcuc is now available 2023-11-17T05:25:28.704 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:27.516656+0000 mon.smithi003 (mon.0) 99 : audit [DBG] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:25:28.704 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:27.519420+0000 mon.smithi003 (mon.0) 100 : audit [DBG] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:25:28.704 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:27.543778+0000 mon.smithi003 (mon.0) 101 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.eddcuc/mirror_snapshot_schedule"}]: dispatch 2023-11-17T05:25:28.704 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:27.547831+0000 mon.smithi003 (mon.0) 102 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.eddcuc/trash_purge_schedule"}]: dispatch 2023-11-17T05:25:29.094 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.090+0000 7f3508a98700 1 Processor -- start 2023-11-17T05:25:29.094 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.090+0000 7f3508a98700 1 -- start start 2023-11-17T05:25:29.094 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.091+0000 7f3508a98700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3504071cb0 0x7f3504070250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:29.094 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.091+0000 7f3508a98700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35040720d0 con 0x7f3504071cb0 2023-11-17T05:25:29.094 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.091+0000 7f350259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3504071cb0 0x7f3504070250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:29.094 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.091+0000 7f350259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3504071cb0 0x7f3504070250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52110/0 (socket says 172.21.15.3:52110) 2023-11-17T05:25:29.094 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.091+0000 7f350259c700 1 -- 172.21.15.3:0/505524070 learned_addr learned my addr 172.21.15.3:0/505524070 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:25:29.094 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.091+0000 7f350259c700 1 -- 172.21.15.3:0/505524070 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3504070740 con 0x7f3504071cb0 2023-11-17T05:25:29.094 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.092+0000 7f350259c700 1 --2- 172.21.15.3:0/505524070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3504071cb0 0x7f3504070250 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f34f400b5f0 tx=0x7f34f4006ee0).ready entity=mon.0 client_cookie=14f71103fd204e50 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:29.094 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.092+0000 7f350159a700 1 -- 172.21.15.3:0/505524070 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34f400d600 con 0x7f3504071cb0 2023-11-17T05:25:29.095 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.092+0000 7f350159a700 1 -- 172.21.15.3:0/505524070 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f34f400dbd0 con 0x7f3504071cb0 2023-11-17T05:25:29.095 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.092+0000 7f350159a700 1 -- 172.21.15.3:0/505524070 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34f4011c00 con 0x7f3504071cb0 2023-11-17T05:25:29.095 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.092+0000 7f3508a98700 1 -- 172.21.15.3:0/505524070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3504071cb0 msgr2=0x7f3504070250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:29.097 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.092+0000 7f3508a98700 1 --2- 172.21.15.3:0/505524070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3504071cb0 0x7f3504070250 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f34f400b5f0 tx=0x7f34f4006ee0).stop 2023-11-17T05:25:29.097 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.093+0000 7f3508a98700 1 -- 172.21.15.3:0/505524070 shutdown_connections 2023-11-17T05:25:29.097 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.093+0000 7f3508a98700 1 --2- 172.21.15.3:0/505524070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3504071cb0 0x7f3504070250 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:29.098 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.093+0000 7f3508a98700 1 -- 172.21.15.3:0/505524070 >> 172.21.15.3:0/505524070 conn(0x7f35040f73e0 msgr2=0x7f35040f9800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:29.098 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.093+0000 7f3508a98700 1 -- 172.21.15.3:0/505524070 shutdown_connections 2023-11-17T05:25:29.098 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.093+0000 7f3508a98700 1 -- 172.21.15.3:0/505524070 wait complete. 2023-11-17T05:25:29.098 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.093+0000 7f3508a98700 1 Processor -- start 2023-11-17T05:25:29.098 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.093+0000 7f3508a98700 1 -- start start 2023-11-17T05:25:29.098 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.093+0000 7f3508a98700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3504071cb0 0x7f350410cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:29.098 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.093+0000 7f3508a98700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f350410d0a0 con 0x7f3504071cb0 2023-11-17T05:25:29.098 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.094+0000 7f350259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3504071cb0 0x7f350410cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:29.098 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.094+0000 7f350259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3504071cb0 0x7f350410cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52114/0 (socket says 172.21.15.3:52114) 2023-11-17T05:25:29.098 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.094+0000 7f350259c700 1 -- 172.21.15.3:0/549890255 learned_addr learned my addr 172.21.15.3:0/549890255 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:25:29.098 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.094+0000 7f350259c700 1 -- 172.21.15.3:0/549890255 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f400b200 con 0x7f3504071cb0 2023-11-17T05:25:29.098 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.094+0000 7f350259c700 1 --2- 172.21.15.3:0/549890255 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3504071cb0 0x7f350410cbb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f34f400e040 tx=0x7f34f4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:29.098 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.094+0000 7f34faffd700 1 -- 172.21.15.3:0/549890255 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34f400d9f0 con 0x7f3504071cb0 2023-11-17T05:25:29.099 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.094+0000 7f34faffd700 1 -- 172.21.15.3:0/549890255 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f34f401a9f0 con 0x7f3504071cb0 2023-11-17T05:25:29.099 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.094+0000 7f3508a98700 1 -- 172.21.15.3:0/549890255 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f350410d2a0 con 0x7f3504071cb0 2023-11-17T05:25:29.099 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.094+0000 7f34faffd700 1 -- 172.21.15.3:0/549890255 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34f401b8c0 con 0x7f3504071cb0 2023-11-17T05:25:29.099 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.094+0000 7f3508a98700 1 -- 172.21.15.3:0/549890255 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f350410d6b0 con 0x7f3504071cb0 2023-11-17T05:25:29.099 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.095+0000 7f34faffd700 1 -- 172.21.15.3:0/549890255 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f34f400f070 con 0x7f3504071cb0 2023-11-17T05:25:29.099 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.095+0000 7f34faffd700 1 --2- 172.21.15.3:0/549890255 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f34e402fdc0 0x7f34e4032280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:29.099 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.095+0000 7f34faffd700 1 -- 172.21.15.3:0/549890255 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f34f401ab60 con 0x7f3504071cb0 2023-11-17T05:25:29.099 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.095+0000 7f3501d9b700 1 --2- 172.21.15.3:0/549890255 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f34e402fdc0 0x7f34e4032280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:29.099 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.095+0000 7f3508a98700 1 -- 172.21.15.3:0/549890255 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3504059100 con 0x7f3504071cb0 2023-11-17T05:25:29.099 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.096+0000 7f3501d9b700 1 --2- 172.21.15.3:0/549890255 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f34e402fdc0 0x7f34e4032280 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f34ec0096a0 tx=0x7f34ec006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:29.101 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.100+0000 7f34faffd700 1 -- 172.21.15.3:0/549890255 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f34f4016070 con 0x7f3504071cb0 2023-11-17T05:25:29.272 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.270+0000 7f3508a98700 1 -- 172.21.15.3:0/549890255 --> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f3504106a30 con 0x7f34e402fdc0 2023-11-17T05:25:29.326 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.324+0000 7f34faffd700 1 -- 172.21.15.3:0/549890255 <== mgr.14164 v2:172.21.15.3:6800/17534166 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f3504106a30 con 0x7f34e402fdc0 2023-11-17T05:25:29.326 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: Self-signed certificate created 2023-11-17T05:25:29.331 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.326+0000 7f3508a98700 1 -- 172.21.15.3:0/549890255 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f34e402fdc0 msgr2=0x7f34e4032280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:29.331 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.326+0000 7f3508a98700 1 --2- 172.21.15.3:0/549890255 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f34e402fdc0 0x7f34e4032280 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f34ec0096a0 tx=0x7f34ec006b40).stop 2023-11-17T05:25:29.331 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.326+0000 7f3508a98700 1 -- 172.21.15.3:0/549890255 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3504071cb0 msgr2=0x7f350410cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:29.331 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.326+0000 7f3508a98700 1 --2- 172.21.15.3:0/549890255 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3504071cb0 0x7f350410cbb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f34f400e040 tx=0x7f34f4006ee0).stop 2023-11-17T05:25:29.331 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.327+0000 7f3508a98700 1 -- 172.21.15.3:0/549890255 shutdown_connections 2023-11-17T05:25:29.331 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.327+0000 7f3508a98700 1 --2- 172.21.15.3:0/549890255 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f34e402fdc0 0x7f34e4032280 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:29.331 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.327+0000 7f3508a98700 1 --2- 172.21.15.3:0/549890255 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3504071cb0 0x7f350410cbb0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:29.331 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.327+0000 7f3508a98700 1 -- 172.21.15.3:0/549890255 >> 172.21.15.3:0/549890255 conn(0x7f35040f73e0 msgr2=0x7f35040f8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:29.331 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.327+0000 7f3508a98700 1 -- 172.21.15.3:0/549890255 shutdown_connections 2023-11-17T05:25:29.332 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:29.327+0000 7f3508a98700 1 -- 172.21.15.3:0/549890255 wait complete. 2023-11-17T05:25:29.611 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:29 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:28.395740+0000 mgr.smithi003.eddcuc (mgr.14164) 1 : audit [DBG] from='client.14168 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-11-17T05:25:29.611 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:29 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:25:28.395839+0000 mon.smithi003 (mon.0) 103 : cluster [DBG] mgrmap e15: smithi003.eddcuc(active, since 1.06219s) 2023-11-17T05:25:29.611 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:29 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:28.403008+0000 mgr.smithi003.eddcuc (mgr.14164) 2 : audit [DBG] from='client.14168 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-11-17T05:25:29.611 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:29 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:29.323261+0000 mon.smithi003 (mon.0) 104 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' 2023-11-17T05:25:29.611 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:29 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:29.326047+0000 mon.smithi003 (mon.0) 105 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' 2023-11-17T05:25:29.691 INFO:teuthology.orchestra.run.smithi003.stderr:Creating initial admin user... 2023-11-17T05:25:29.692 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptcuja8tf:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpk7o1jie2:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpj2uant2s:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2023-11-17T05:25:30.091 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.083+0000 7f2f17db6700 1 Processor -- start 2023-11-17T05:25:30.091 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.084+0000 7f2f17db6700 1 -- start start 2023-11-17T05:25:30.091 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.084+0000 7f2f17db6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f100fc210 0x7f2f100fc630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:30.091 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.084+0000 7f2f17db6700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f100fcb20 con 0x7f2f100fc210 2023-11-17T05:25:30.091 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.084+0000 7f2f15b52700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f100fc210 0x7f2f100fc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:30.091 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.084+0000 7f2f15b52700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f100fc210 0x7f2f100fc630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52116/0 (socket says 172.21.15.3:52116) 2023-11-17T05:25:30.091 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.084+0000 7f2f15b52700 1 -- 172.21.15.3:0/3656554028 learned_addr learned my addr 172.21.15.3:0/3656554028 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:25:30.091 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.084+0000 7f2f15b52700 1 -- 172.21.15.3:0/3656554028 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f100fcc60 con 0x7f2f100fc210 2023-11-17T05:25:30.091 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.084+0000 7f2f15b52700 1 --2- 172.21.15.3:0/3656554028 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f100fc210 0x7f2f100fc630 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f2f0c00b5f0 tx=0x7f2f0c006ee0).ready entity=mon.0 client_cookie=dd555ceca93d9708 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:30.091 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.085+0000 7f2f14b50700 1 -- 172.21.15.3:0/3656554028 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f0c00d600 con 0x7f2f100fc210 2023-11-17T05:25:30.092 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.085+0000 7f2f14b50700 1 -- 172.21.15.3:0/3656554028 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f2f0c00dbd0 con 0x7f2f100fc210 2023-11-17T05:25:30.092 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.085+0000 7f2f14b50700 1 -- 172.21.15.3:0/3656554028 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f0c011c00 con 0x7f2f100fc210 2023-11-17T05:25:30.092 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.085+0000 7f2f17db6700 1 -- 172.21.15.3:0/3656554028 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f100fc210 msgr2=0x7f2f100fc630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:30.092 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.085+0000 7f2f17db6700 1 --2- 172.21.15.3:0/3656554028 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f100fc210 0x7f2f100fc630 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f2f0c00b5f0 tx=0x7f2f0c006ee0).stop 2023-11-17T05:25:30.092 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.085+0000 7f2f17db6700 1 -- 172.21.15.3:0/3656554028 shutdown_connections 2023-11-17T05:25:30.092 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.085+0000 7f2f17db6700 1 --2- 172.21.15.3:0/3656554028 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f100fc210 0x7f2f100fc630 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:30.092 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.085+0000 7f2f17db6700 1 -- 172.21.15.3:0/3656554028 >> 172.21.15.3:0/3656554028 conn(0x7f2f100f5a80 msgr2=0x7f2f100f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:30.092 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.085+0000 7f2f17db6700 1 -- 172.21.15.3:0/3656554028 shutdown_connections 2023-11-17T05:25:30.092 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.086+0000 7f2f17db6700 1 -- 172.21.15.3:0/3656554028 wait complete. 2023-11-17T05:25:30.092 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.086+0000 7f2f17db6700 1 Processor -- start 2023-11-17T05:25:30.092 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.086+0000 7f2f17db6700 1 -- start start 2023-11-17T05:25:30.092 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.086+0000 7f2f17db6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f100fc210 0x7f2f1010cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:30.092 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.086+0000 7f2f17db6700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1010d0f0 con 0x7f2f100fc210 2023-11-17T05:25:30.093 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.086+0000 7f2f15b52700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f100fc210 0x7f2f1010cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:30.093 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.087+0000 7f2f15b52700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f100fc210 0x7f2f1010cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52122/0 (socket says 172.21.15.3:52122) 2023-11-17T05:25:30.093 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.087+0000 7f2f15b52700 1 -- 172.21.15.3:0/2331007705 learned_addr learned my addr 172.21.15.3:0/2331007705 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:25:30.093 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.087+0000 7f2f15b52700 1 -- 172.21.15.3:0/2331007705 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f0c00b200 con 0x7f2f100fc210 2023-11-17T05:25:30.093 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.087+0000 7f2f15b52700 1 --2- 172.21.15.3:0/2331007705 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f100fc210 0x7f2f1010cc00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f2f0c00e040 tx=0x7f2f0c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:30.093 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.087+0000 7f2efe7fc700 1 -- 172.21.15.3:0/2331007705 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f0c01b410 con 0x7f2f100fc210 2023-11-17T05:25:30.093 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.087+0000 7f2efe7fc700 1 -- 172.21.15.3:0/2331007705 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f2f0c01b9e0 con 0x7f2f100fc210 2023-11-17T05:25:30.094 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.087+0000 7f2f17db6700 1 -- 172.21.15.3:0/2331007705 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f1010d2f0 con 0x7f2f100fc210 2023-11-17T05:25:30.095 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.087+0000 7f2efe7fc700 1 -- 172.21.15.3:0/2331007705 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f0c01a770 con 0x7f2f100fc210 2023-11-17T05:25:30.095 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.087+0000 7f2f17db6700 1 -- 172.21.15.3:0/2331007705 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f1010d700 con 0x7f2f100fc210 2023-11-17T05:25:30.095 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.088+0000 7f2efe7fc700 1 -- 172.21.15.3:0/2331007705 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f2f0c00f070 con 0x7f2f100fc210 2023-11-17T05:25:30.095 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.088+0000 7f2f17db6700 1 -- 172.21.15.3:0/2331007705 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f10045ad0 con 0x7f2f100fc210 2023-11-17T05:25:30.096 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.088+0000 7f2efe7fc700 1 --2- 172.21.15.3:0/2331007705 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f2ef802fd50 0x7f2ef8032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:30.096 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.089+0000 7f2efe7fc700 1 -- 172.21.15.3:0/2331007705 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2f0c041150 con 0x7f2f100fc210 2023-11-17T05:25:30.096 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.089+0000 7f2f15351700 1 --2- 172.21.15.3:0/2331007705 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f2ef802fd50 0x7f2ef8032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:30.096 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.089+0000 7f2f15351700 1 --2- 172.21.15.3:0/2331007705 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f2ef802fd50 0x7f2ef8032210 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f2f000096a0 tx=0x7f2f00006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:30.096 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.093+0000 7f2efe7fc700 1 -- 172.21.15.3:0/2331007705 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2f0c024410 con 0x7f2f100fc210 2023-11-17T05:25:30.275 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.273+0000 7f2f17db6700 1 -- 172.21.15.3:0/2331007705 --> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f2f10106bc0 con 0x7f2ef802fd50 2023-11-17T05:25:30.327 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.325+0000 7f2efe7fc700 1 -- 172.21.15.3:0/2331007705 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f2f0c010dd0 con 0x7f2f100fc210 2023-11-17T05:25:30.527 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.525+0000 7f2efe7fc700 1 -- 172.21.15.3:0/2331007705 <== mgr.14164 v2:172.21.15.3:6800/17534166 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f2f10106bc0 con 0x7f2ef802fd50 2023-11-17T05:25:30.527 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$kIBPeB7b3meQIV4LrSIKSOPpY/0Ofr/wEG0gAwdf.qtZna.sWXKyK", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1700198730, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-11-17T05:25:30.529 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.527+0000 7f2f17db6700 1 -- 172.21.15.3:0/2331007705 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f2ef802fd50 msgr2=0x7f2ef8032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:30.529 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.527+0000 7f2f17db6700 1 --2- 172.21.15.3:0/2331007705 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f2ef802fd50 0x7f2ef8032210 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f2f000096a0 tx=0x7f2f00006b40).stop 2023-11-17T05:25:30.529 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.527+0000 7f2f17db6700 1 -- 172.21.15.3:0/2331007705 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f100fc210 msgr2=0x7f2f1010cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:30.529 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.527+0000 7f2f17db6700 1 --2- 172.21.15.3:0/2331007705 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f100fc210 0x7f2f1010cc00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f2f0c00e040 tx=0x7f2f0c006ee0).stop 2023-11-17T05:25:30.530 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.527+0000 7f2f17db6700 1 -- 172.21.15.3:0/2331007705 shutdown_connections 2023-11-17T05:25:30.530 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.527+0000 7f2f17db6700 1 --2- 172.21.15.3:0/2331007705 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f2ef802fd50 0x7f2ef8032210 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:30.530 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.527+0000 7f2f17db6700 1 --2- 172.21.15.3:0/2331007705 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f100fc210 0x7f2f1010cc00 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:30.530 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.527+0000 7f2f17db6700 1 -- 172.21.15.3:0/2331007705 >> 172.21.15.3:0/2331007705 conn(0x7f2f100f5a80 msgr2=0x7f2f100f7640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:30.530 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.528+0000 7f2f17db6700 1 -- 172.21.15.3:0/2331007705 shutdown_connections 2023-11-17T05:25:30.530 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:30.528+0000 7f2f17db6700 1 -- 172.21.15.3:0/2331007705 wait complete. 2023-11-17T05:25:30.698 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:30 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:29.272796+0000 mgr.smithi003.eddcuc (mgr.14164) 3 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:25:30.698 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:30 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:25:30.328560+0000 mon.smithi003 (mon.0) 106 : cluster [DBG] mgrmap e16: smithi003.eddcuc(active, since 2s) 2023-11-17T05:25:30.874 INFO:teuthology.orchestra.run.smithi003.stderr:Fetching dashboard port number... 2023-11-17T05:25:30.874 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmptcuja8tf:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpk7o1jie2:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2023-11-17T05:25:31.261 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.259+0000 7f58f76a0700 1 Processor -- start 2023-11-17T05:25:31.261 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.259+0000 7f58f76a0700 1 -- start start 2023-11-17T05:25:31.261 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.260+0000 7f58f76a0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f58f006fcd0 0x7f58f00fe7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:31.264 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.260+0000 7f58f76a0700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f00fed20 con 0x7f58f006fcd0 2023-11-17T05:25:31.264 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.260+0000 7f58f543c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f58f006fcd0 0x7f58f00fe7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:31.264 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.260+0000 7f58f543c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f58f006fcd0 0x7f58f00fe7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52136/0 (socket says 172.21.15.3:52136) 2023-11-17T05:25:31.264 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.260+0000 7f58f543c700 1 -- 172.21.15.3:0/2446830422 learned_addr learned my addr 172.21.15.3:0/2446830422 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:25:31.264 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.260+0000 7f58f543c700 1 -- 172.21.15.3:0/2446830422 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58f00fee60 con 0x7f58f006fcd0 2023-11-17T05:25:31.264 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.260+0000 7f58f543c700 1 --2- 172.21.15.3:0/2446830422 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f58f006fcd0 0x7f58f00fe7a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f58e400b5f0 tx=0x7f58e4006ee0).ready entity=mon.0 client_cookie=307039f66e1a99bd server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:31.264 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.261+0000 7f58e3fff700 1 -- 172.21.15.3:0/2446830422 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58e400d600 con 0x7f58f006fcd0 2023-11-17T05:25:31.264 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.261+0000 7f58e3fff700 1 -- 172.21.15.3:0/2446830422 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f58e400dbd0 con 0x7f58f006fcd0 2023-11-17T05:25:31.264 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.261+0000 7f58e3fff700 1 -- 172.21.15.3:0/2446830422 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58e4011c00 con 0x7f58f006fcd0 2023-11-17T05:25:31.265 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.261+0000 7f58f76a0700 1 -- 172.21.15.3:0/2446830422 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f58f006fcd0 msgr2=0x7f58f00fe7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:31.265 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.261+0000 7f58f76a0700 1 --2- 172.21.15.3:0/2446830422 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f58f006fcd0 0x7f58f00fe7a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f58e400b5f0 tx=0x7f58e4006ee0).stop 2023-11-17T05:25:31.265 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.262+0000 7f58f76a0700 1 -- 172.21.15.3:0/2446830422 shutdown_connections 2023-11-17T05:25:31.265 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.262+0000 7f58f76a0700 1 --2- 172.21.15.3:0/2446830422 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f58f006fcd0 0x7f58f00fe7a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:31.265 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.262+0000 7f58f76a0700 1 -- 172.21.15.3:0/2446830422 >> 172.21.15.3:0/2446830422 conn(0x7f58f00f6280 msgr2=0x7f58f00f86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:31.265 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.262+0000 7f58f76a0700 1 -- 172.21.15.3:0/2446830422 shutdown_connections 2023-11-17T05:25:31.265 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.262+0000 7f58f76a0700 1 -- 172.21.15.3:0/2446830422 wait complete. 2023-11-17T05:25:31.265 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.262+0000 7f58f76a0700 1 Processor -- start 2023-11-17T05:25:31.265 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.262+0000 7f58f76a0700 1 -- start start 2023-11-17T05:25:31.265 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.262+0000 7f58f76a0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f58f006fcd0 0x7f58f00fc220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:31.265 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.262+0000 7f58f76a0700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f00fa6a0 con 0x7f58f006fcd0 2023-11-17T05:25:31.269 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.263+0000 7f58f543c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f58f006fcd0 0x7f58f00fc220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:31.269 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.263+0000 7f58f543c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f58f006fcd0 0x7f58f00fc220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52144/0 (socket says 172.21.15.3:52144) 2023-11-17T05:25:31.269 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.263+0000 7f58f543c700 1 -- 172.21.15.3:0/81259246 learned_addr learned my addr 172.21.15.3:0/81259246 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:25:31.270 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.263+0000 7f58f543c700 1 -- 172.21.15.3:0/81259246 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e400b200 con 0x7f58f006fcd0 2023-11-17T05:25:31.270 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.263+0000 7f58f543c700 1 --2- 172.21.15.3:0/81259246 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f58f006fcd0 0x7f58f00fc220 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f58e400e040 tx=0x7f58e4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:31.270 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.263+0000 7f58e1ffb700 1 -- 172.21.15.3:0/81259246 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58e401b410 con 0x7f58f006fcd0 2023-11-17T05:25:31.270 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.263+0000 7f58e1ffb700 1 -- 172.21.15.3:0/81259246 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f58e401b9e0 con 0x7f58f006fcd0 2023-11-17T05:25:31.270 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.263+0000 7f58f76a0700 1 -- 172.21.15.3:0/81259246 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58f00fa840 con 0x7f58f006fcd0 2023-11-17T05:25:31.271 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.263+0000 7f58e1ffb700 1 -- 172.21.15.3:0/81259246 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58e401a770 con 0x7f58f006fcd0 2023-11-17T05:25:31.271 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.264+0000 7f58f76a0700 1 -- 172.21.15.3:0/81259246 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58f00fac50 con 0x7f58f006fcd0 2023-11-17T05:25:31.271 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.265+0000 7f58e1ffb700 1 -- 172.21.15.3:0/81259246 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f58e400f070 con 0x7f58f006fcd0 2023-11-17T05:25:31.271 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.265+0000 7f58e1ffb700 1 --2- 172.21.15.3:0/81259246 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f58d802ff00 0x7f58d80323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:31.271 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.265+0000 7f58f76a0700 1 -- 172.21.15.3:0/81259246 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58f0045ad0 con 0x7f58f006fcd0 2023-11-17T05:25:31.272 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.265+0000 7f58e1ffb700 1 -- 172.21.15.3:0/81259246 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f58e4041240 con 0x7f58f006fcd0 2023-11-17T05:25:31.272 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.265+0000 7f58f4c3b700 1 --2- 172.21.15.3:0/81259246 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f58d802ff00 0x7f58d80323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:31.272 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.265+0000 7f58f4c3b700 1 --2- 172.21.15.3:0/81259246 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f58d802ff00 0x7f58d80323c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f58ec0096a0 tx=0x7f58ec006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:31.272 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.269+0000 7f58e1ffb700 1 -- 172.21.15.3:0/81259246 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f58e4016070 con 0x7f58f006fcd0 2023-11-17T05:25:31.437 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.436+0000 7f58f76a0700 1 -- 172.21.15.3:0/81259246 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f58f00ffc90 con 0x7f58f006fcd0 2023-11-17T05:25:31.438 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.436+0000 7f58e1ffb700 1 -- 172.21.15.3:0/81259246 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v9) v1 ==== 112+0+5 (secure 0 0 0) 0x7f58e4010580 con 0x7f58f006fcd0 2023-11-17T05:25:31.438 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 8443 2023-11-17T05:25:31.440 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.438+0000 7f58f76a0700 1 -- 172.21.15.3:0/81259246 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f58d802ff00 msgr2=0x7f58d80323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:31.440 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.438+0000 7f58f76a0700 1 --2- 172.21.15.3:0/81259246 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f58d802ff00 0x7f58d80323c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f58ec0096a0 tx=0x7f58ec006b40).stop 2023-11-17T05:25:31.441 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.438+0000 7f58f76a0700 1 -- 172.21.15.3:0/81259246 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f58f006fcd0 msgr2=0x7f58f00fc220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:31.441 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.438+0000 7f58f76a0700 1 --2- 172.21.15.3:0/81259246 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f58f006fcd0 0x7f58f00fc220 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f58e400e040 tx=0x7f58e4006ee0).stop 2023-11-17T05:25:31.441 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.438+0000 7f58f76a0700 1 -- 172.21.15.3:0/81259246 shutdown_connections 2023-11-17T05:25:31.441 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.438+0000 7f58f76a0700 1 --2- 172.21.15.3:0/81259246 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f58d802ff00 0x7f58d80323c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:31.441 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.438+0000 7f58f76a0700 1 --2- 172.21.15.3:0/81259246 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f58f006fcd0 0x7f58f00fc220 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:31.441 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.438+0000 7f58f76a0700 1 -- 172.21.15.3:0/81259246 >> 172.21.15.3:0/81259246 conn(0x7f58f00f6280 msgr2=0x7f58f00f7e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:31.441 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.438+0000 7f58f76a0700 1 -- 172.21.15.3:0/81259246 shutdown_connections 2023-11-17T05:25:31.441 INFO:teuthology.orchestra.run.smithi003.stderr:/usr/bin/ceph: 2023-11-17T05:25:31.438+0000 7f58f76a0700 1 -- 172.21.15.3:0/81259246 wait complete. 2023-11-17T05:25:31.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:31 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:30.275525+0000 mgr.smithi003.eddcuc (mgr.14164) 4 : audit [DBG] from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:25:31.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:31 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:30.527072+0000 mon.smithi003 (mon.0) 107 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' 2023-11-17T05:25:31.771 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:31 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:31.437767+0000 mon.smithi003 (mon.0) 108 : audit [DBG] from='client.? 172.21.15.3:0/81259246' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-11-17T05:25:31.808 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl is-enabled firewalld.service 2023-11-17T05:25:31.819 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: disabled 2023-11-17T05:25:31.820 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: systemctl is-active firewalld.service 2023-11-17T05:25:31.829 INFO:teuthology.orchestra.run.smithi003.stderr:systemctl: inactive 2023-11-17T05:25:31.830 INFO:teuthology.orchestra.run.smithi003.stderr:firewalld.service is not enabled 2023-11-17T05:25:31.830 INFO:teuthology.orchestra.run.smithi003.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2023-11-17T05:25:31.832 INFO:teuthology.orchestra.run.smithi003.stderr:Ceph Dashboard is now available at: 2023-11-17T05:25:31.832 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-11-17T05:25:31.832 INFO:teuthology.orchestra.run.smithi003.stderr: URL: https://smithi003.front.sepia.ceph.com:8443/ 2023-11-17T05:25:31.832 INFO:teuthology.orchestra.run.smithi003.stderr: User: admin 2023-11-17T05:25:31.832 INFO:teuthology.orchestra.run.smithi003.stderr: Password: 2n327yqnan 2023-11-17T05:25:31.833 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-11-17T05:25:31.833 INFO:teuthology.orchestra.run.smithi003.stderr:You can access the Ceph CLI with: 2023-11-17T05:25:31.833 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-11-17T05:25:31.833 INFO:teuthology.orchestra.run.smithi003.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-11-17T05:25:31.833 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-11-17T05:25:31.834 INFO:teuthology.orchestra.run.smithi003.stderr:Please consider enabling telemetry to help improve Ceph: 2023-11-17T05:25:31.834 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-11-17T05:25:31.834 INFO:teuthology.orchestra.run.smithi003.stderr: ceph telemetry on 2023-11-17T05:25:31.834 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-11-17T05:25:31.834 INFO:teuthology.orchestra.run.smithi003.stderr:For more information see: 2023-11-17T05:25:31.834 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-11-17T05:25:31.834 INFO:teuthology.orchestra.run.smithi003.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2023-11-17T05:25:31.834 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-11-17T05:25:31.834 INFO:teuthology.orchestra.run.smithi003.stderr:Bootstrap complete. 2023-11-17T05:25:31.834 INFO:teuthology.orchestra.run.smithi003.stderr:Releasing lock 140593299061224 on /run/cephadm/851feb1e-8509-11ee-8dbe-212e2dc638e7.lock 2023-11-17T05:25:31.835 INFO:teuthology.orchestra.run.smithi003.stderr:Lock 140593299061224 released on /run/cephadm/851feb1e-8509-11ee-8dbe-212e2dc638e7.lock 2023-11-17T05:25:31.874 INFO:tasks.cephadm:Fetching config... 2023-11-17T05:25:31.874 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-11-17T05:25:31.875 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-11-17T05:25:31.896 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-11-17T05:25:31.896 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-11-17T05:25:31.896 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-11-17T05:25:31.960 INFO:tasks.cephadm:Fetching mon keyring... 2023-11-17T05:25:31.960 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-11-17T05:25:31.960 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/mon.smithi003/keyring of=/dev/stdout 2023-11-17T05:25:32.035 INFO:tasks.cephadm:Fetching pub ssh key... 2023-11-17T05:25:32.036 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2023-11-17T05:25:32.036 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-11-17T05:25:32.099 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-11-17T05:25:32.099 DEBUG:teuthology.orchestra.run.smithi003:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDrzTHMzFX2NWVwNhuoufWlyf/uJpnw9PgXmv2GlMM71C50xs9YU4jzcVV3I6GU9YnqHwEjMLwjwA1H0VePmw+83pyfhzq7WbqCIvAZUXNHnMKD272bIQ8jLVM+2piyZOCo+tvmpcvncbTV34nouTQ0+IDcHOzDHNXq8KOmU3t9e/mGS+YHU3upQ/C9a8t2JQlxFfIBjYjxSRa7znX+gFikk2eABQ3Xst5jpMhuJWrTAOnA9Mlq9MhVyr8NXJmdRmZ5tuu6Lzi6Za31q3/0Gtq8rhpty7SwYN6BFY1achyEZUOI6gN4GyHT7sD/pRJGw3IAOhjTBaQ3ygjq10I9/5Q9R4pgaS2QTBbG6WRGwRuhYg1h0YwBWQ8uvNOBByYTU+RsjNU+KDi0fnlVYhj4qQkeBN7QggLcob8maIDJv92STaKZnoD/ZaoUispkCxJGlN3xq4WxK0J8O1XGQ7V5d6Fz8N14p4C1iMmMPsP+Q5Y89PLknYcPq2k3eBwJu1FA0wk= ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-11-17T05:25:32.188 INFO:teuthology.orchestra.run.smithi003.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDrzTHMzFX2NWVwNhuoufWlyf/uJpnw9PgXmv2GlMM71C50xs9YU4jzcVV3I6GU9YnqHwEjMLwjwA1H0VePmw+83pyfhzq7WbqCIvAZUXNHnMKD272bIQ8jLVM+2piyZOCo+tvmpcvncbTV34nouTQ0+IDcHOzDHNXq8KOmU3t9e/mGS+YHU3upQ/C9a8t2JQlxFfIBjYjxSRa7znX+gFikk2eABQ3Xst5jpMhuJWrTAOnA9Mlq9MhVyr8NXJmdRmZ5tuu6Lzi6Za31q3/0Gtq8rhpty7SwYN6BFY1achyEZUOI6gN4GyHT7sD/pRJGw3IAOhjTBaQ3ygjq10I9/5Q9R4pgaS2QTBbG6WRGwRuhYg1h0YwBWQ8uvNOBByYTU+RsjNU+KDi0fnlVYhj4qQkeBN7QggLcob8maIDJv92STaKZnoD/ZaoUispkCxJGlN3xq4WxK0J8O1XGQ7V5d6Fz8N14p4C1iMmMPsP+Q5Y89PLknYcPq2k3eBwJu1FA0wk= ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7 2023-11-17T05:25:32.201 DEBUG:teuthology.orchestra.run.smithi160:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDrzTHMzFX2NWVwNhuoufWlyf/uJpnw9PgXmv2GlMM71C50xs9YU4jzcVV3I6GU9YnqHwEjMLwjwA1H0VePmw+83pyfhzq7WbqCIvAZUXNHnMKD272bIQ8jLVM+2piyZOCo+tvmpcvncbTV34nouTQ0+IDcHOzDHNXq8KOmU3t9e/mGS+YHU3upQ/C9a8t2JQlxFfIBjYjxSRa7znX+gFikk2eABQ3Xst5jpMhuJWrTAOnA9Mlq9MhVyr8NXJmdRmZ5tuu6Lzi6Za31q3/0Gtq8rhpty7SwYN6BFY1achyEZUOI6gN4GyHT7sD/pRJGw3IAOhjTBaQ3ygjq10I9/5Q9R4pgaS2QTBbG6WRGwRuhYg1h0YwBWQ8uvNOBByYTU+RsjNU+KDi0fnlVYhj4qQkeBN7QggLcob8maIDJv92STaKZnoD/ZaoUispkCxJGlN3xq4WxK0J8O1XGQ7V5d6Fz8N14p4C1iMmMPsP+Q5Y89PLknYcPq2k3eBwJu1FA0wk= ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-11-17T05:25:32.248 INFO:teuthology.orchestra.run.smithi160.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDrzTHMzFX2NWVwNhuoufWlyf/uJpnw9PgXmv2GlMM71C50xs9YU4jzcVV3I6GU9YnqHwEjMLwjwA1H0VePmw+83pyfhzq7WbqCIvAZUXNHnMKD272bIQ8jLVM+2piyZOCo+tvmpcvncbTV34nouTQ0+IDcHOzDHNXq8KOmU3t9e/mGS+YHU3upQ/C9a8t2JQlxFfIBjYjxSRa7znX+gFikk2eABQ3Xst5jpMhuJWrTAOnA9Mlq9MhVyr8NXJmdRmZ5tuu6Lzi6Za31q3/0Gtq8rhpty7SwYN6BFY1achyEZUOI6gN4GyHT7sD/pRJGw3IAOhjTBaQ3ygjq10I9/5Q9R4pgaS2QTBbG6WRGwRuhYg1h0YwBWQ8uvNOBByYTU+RsjNU+KDi0fnlVYhj4qQkeBN7QggLcob8maIDJv92STaKZnoD/ZaoUispkCxJGlN3xq4WxK0J8O1XGQ7V5d6Fz8N14p4C1iMmMPsP+Q5Y89PLknYcPq2k3eBwJu1FA0wk= ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7 2023-11-17T05:25:32.260 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-11-17T05:25:33.457 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:33.455+0000 7fa6557ff700 1 -- 172.21.15.3:0/1431531680 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa650060280 msgr2=0x7fa6500f8ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:33.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:33.455+0000 7fa6557ff700 1 --2- 172.21.15.3:0/1431531680 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa650060280 0x7fa6500f8ea0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fa638009fb0 tx=0x7fa638009b30).stop 2023-11-17T05:25:33.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:33.455+0000 7fa6557ff700 1 -- 172.21.15.3:0/1431531680 shutdown_connections 2023-11-17T05:25:33.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:33.455+0000 7fa6557ff700 1 --2- 172.21.15.3:0/1431531680 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa650060280 0x7fa6500f8ea0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:33.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:33.455+0000 7fa6557ff700 1 -- 172.21.15.3:0/1431531680 >> 172.21.15.3:0/1431531680 conn(0x7fa6500f0920 msgr2=0x7fa6500f2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:33.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:33.455+0000 7fa6557ff700 1 -- 172.21.15.3:0/1431531680 shutdown_connections 2023-11-17T05:25:33.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:33.455+0000 7fa6557ff700 1 -- 172.21.15.3:0/1431531680 wait complete. 2023-11-17T05:25:33.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:33.456+0000 7fa6557ff700 1 Processor -- start 2023-11-17T05:25:33.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:33.456+0000 7fa6557ff700 1 -- start start 2023-11-17T05:25:33.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:33.456+0000 7fa6557ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa650060280 0x7fa6501064c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:33.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:33.456+0000 7fa6557ff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6501069b0 con 0x7fa650060280 2023-11-17T05:25:33.458 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:33.456+0000 7fa64effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa650060280 0x7fa6501064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:33.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:33.456+0000 7fa64effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa650060280 0x7fa6501064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52172/0 (socket says 172.21.15.3:52172) 2023-11-17T05:25:33.459 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:33.456+0000 7fa64effd700 1 -- 172.21.15.3:0/3838812907 learned_addr learned my addr 172.21.15.3:0/3838812907 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:25:33.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:33.457+0000 7fa64effd700 1 -- 172.21.15.3:0/3838812907 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa638005040 con 0x7fa650060280 2023-11-17T05:25:33.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:33.457+0000 7fa64effd700 1 --2- 172.21.15.3:0/3838812907 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa650060280 0x7fa6501064c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fa638002040 tx=0x7fa63800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:33.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:33.457+0000 7fa6477fe700 1 -- 172.21.15.3:0/3838812907 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6380095f0 con 0x7fa650060280 2023-11-17T05:25:33.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:33.457+0000 7fa6557ff700 1 -- 172.21.15.3:0/3838812907 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa650106bb0 con 0x7fa650060280 2023-11-17T05:25:33.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:33.457+0000 7fa6477fe700 1 -- 172.21.15.3:0/3838812907 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa638007e30 con 0x7fa650060280 2023-11-17T05:25:33.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:33.457+0000 7fa6477fe700 1 -- 172.21.15.3:0/3838812907 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa638014d20 con 0x7fa650060280 2023-11-17T05:25:33.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:33.457+0000 7fa6557ff700 1 -- 172.21.15.3:0/3838812907 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa650107040 con 0x7fa650060280 2023-11-17T05:25:33.460 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:33.458+0000 7fa6477fe700 1 -- 172.21.15.3:0/3838812907 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa638025070 con 0x7fa650060280 2023-11-17T05:25:33.461 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:33.458+0000 7fa6557ff700 1 -- 172.21.15.3:0/3838812907 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa650045ad0 con 0x7fa650060280 2023-11-17T05:25:33.461 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:33.458+0000 7fa6477fe700 1 --2- 172.21.15.3:0/3838812907 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa63c02ff60 0x7fa63c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:33.461 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:33.458+0000 7fa6477fe700 1 -- 172.21.15.3:0/3838812907 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa638040970 con 0x7fa650060280 2023-11-17T05:25:33.461 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:33.458+0000 7fa64e7fc700 1 --2- 172.21.15.3:0/3838812907 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa63c02ff60 0x7fa63c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:33.461 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:33.459+0000 7fa64e7fc700 1 --2- 172.21.15.3:0/3838812907 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa63c02ff60 0x7fa63c032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fa6400096a0 tx=0x7fa640006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:33.464 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:33.463+0000 7fa6477fe700 1 -- 172.21.15.3:0/3838812907 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa63800f4e0 con 0x7fa650060280 2023-11-17T05:25:33.630 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:33.629+0000 7fa6557ff700 1 -- 172.21.15.3:0/3838812907 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fa650107eb0 con 0x7fa650060280 2023-11-17T05:25:33.634 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:33.633+0000 7fa6477fe700 1 -- 172.21.15.3:0/3838812907 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v10)=0 v10) v1 ==== 125+0+0 (secure 0 0 0) 0x7fa63800f300 con 0x7fa650060280 2023-11-17T05:25:33.636 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:33.635+0000 7fa6557ff700 1 -- 172.21.15.3:0/3838812907 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa63c02ff60 msgr2=0x7fa63c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:33.637 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:33.635+0000 7fa6557ff700 1 --2- 172.21.15.3:0/3838812907 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa63c02ff60 0x7fa63c032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fa6400096a0 tx=0x7fa640006b40).stop 2023-11-17T05:25:33.637 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:33.635+0000 7fa6557ff700 1 -- 172.21.15.3:0/3838812907 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa650060280 msgr2=0x7fa6501064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:33.637 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:33.635+0000 7fa6557ff700 1 --2- 172.21.15.3:0/3838812907 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa650060280 0x7fa6501064c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fa638002040 tx=0x7fa63800b040).stop 2023-11-17T05:25:33.637 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:33.635+0000 7fa6557ff700 1 -- 172.21.15.3:0/3838812907 shutdown_connections 2023-11-17T05:25:33.637 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:33.635+0000 7fa6557ff700 1 --2- 172.21.15.3:0/3838812907 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa63c02ff60 0x7fa63c032420 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:33.637 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:33.635+0000 7fa6557ff700 1 --2- 172.21.15.3:0/3838812907 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa650060280 0x7fa6501064c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:33.637 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:33.635+0000 7fa6557ff700 1 -- 172.21.15.3:0/3838812907 >> 172.21.15.3:0/3838812907 conn(0x7fa6500f0920 msgr2=0x7fa6500f1400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:33.637 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:33.635+0000 7fa6557ff700 1 -- 172.21.15.3:0/3838812907 shutdown_connections 2023-11-17T05:25:33.637 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:33.635+0000 7fa6557ff700 1 -- 172.21.15.3:0/3838812907 wait complete. 2023-11-17T05:25:34.026 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi160 2023-11-17T05:25:34.026 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-11-17T05:25:34.026 DEBUG:teuthology.orchestra.run.smithi160:> dd of=/etc/ceph/ceph.conf 2023-11-17T05:25:34.048 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2023-11-17T05:25:34.048 DEBUG:teuthology.orchestra.run.smithi160:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-11-17T05:25:34.110 INFO:tasks.cephadm:Adding host smithi160 to orchestrator... 2023-11-17T05:25:34.111 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph orch host add smithi160 2023-11-17T05:25:34.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:34 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:25:33.411802+0000 mon.smithi003 (mon.0) 109 : cluster [DBG] mgrmap e17: smithi003.eddcuc(active, since 6s) 2023-11-17T05:25:34.790 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:34 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:33.634882+0000 mon.smithi003 (mon.0) 110 : audit [INF] from='client.? 172.21.15.3:0/3838812907' entity='client.admin' 2023-11-17T05:25:36.520 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:36 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:35.452189+0000 mon.smithi003 (mon.0) 111 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' 2023-11-17T05:25:36.520 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:36 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:35.699817+0000 mon.smithi003 (mon.0) 112 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' 2023-11-17T05:25:36.520 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:36 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:36.405499+0000 mon.smithi003 (mon.0) 113 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' 2023-11-17T05:25:40.271 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:39 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:38.970905+0000 mon.smithi003 (mon.0) 114 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' 2023-11-17T05:25:40.271 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:39 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:39.274022+0000 mon.smithi003 (mon.0) 115 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' 2023-11-17T05:25:40.271 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:39 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:39.276362+0000 mon.smithi003 (mon.0) 116 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' 2023-11-17T05:25:40.271 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:39 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cephadm 2023-11-17T05:25:39.282830+0000 mgr.smithi003.eddcuc (mgr.14164) 5 : cephadm [INF] Deploying daemon alertmanager.smithi003 on smithi003 2023-11-17T05:25:48.771 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:48 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:25:47.390783+0000 mgr.smithi003.eddcuc (mgr.14164) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:25:50.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:49 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:25:49.391042+0000 mgr.smithi003.eddcuc (mgr.14164) 7 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:25:52.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:52 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:25:51.391310+0000 mgr.smithi003.eddcuc (mgr.14164) 8 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:25:54.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:54 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:25:53.391648+0000 mgr.smithi003.eddcuc (mgr.14164) 9 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:25:54.874 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:54.871+0000 7f5893a40700 1 -- 172.21.15.160:0/844060391 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f588c0f6510 msgr2=0x7f588c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:54.874 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:54.871+0000 7f5893a40700 1 --2- 172.21.15.160:0/844060391 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f588c0f6510 0x7f588c0f6930 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f587c009fb0 tx=0x7f587c009b30).stop 2023-11-17T05:25:54.874 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:54.871+0000 7f5893a40700 1 -- 172.21.15.160:0/844060391 shutdown_connections 2023-11-17T05:25:54.874 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:54.871+0000 7f5893a40700 1 --2- 172.21.15.160:0/844060391 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f588c0f6510 0x7f588c0f6930 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:54.874 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:54.871+0000 7f5893a40700 1 -- 172.21.15.160:0/844060391 >> 172.21.15.160:0/844060391 conn(0x7f588c0f1c70 msgr2=0x7f588c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:54.875 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:54.871+0000 7f5893a40700 1 -- 172.21.15.160:0/844060391 shutdown_connections 2023-11-17T05:25:54.875 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:54.871+0000 7f5893a40700 1 -- 172.21.15.160:0/844060391 wait complete. 2023-11-17T05:25:54.875 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:54.872+0000 7f5893a40700 1 Processor -- start 2023-11-17T05:25:54.875 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:54.872+0000 7f5893a40700 1 -- start start 2023-11-17T05:25:54.875 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:54.872+0000 7f5893a40700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f588c0f6510 0x7f588c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:54.875 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:54.872+0000 7f5893a40700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f588c106c20 con 0x7f588c0f6510 2023-11-17T05:25:54.875 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:54.872+0000 7f58917dc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f588c0f6510 0x7f588c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:54.875 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:54.873+0000 7f58917dc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f588c0f6510 0x7f588c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:52038/0 (socket says 172.21.15.160:52038) 2023-11-17T05:25:54.875 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:54.873+0000 7f58917dc700 1 -- 172.21.15.160:0/1918074040 learned_addr learned my addr 172.21.15.160:0/1918074040 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:25:54.875 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:54.873+0000 7f58917dc700 1 -- 172.21.15.160:0/1918074040 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f587c005040 con 0x7f588c0f6510 2023-11-17T05:25:54.875 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:54.873+0000 7f58917dc700 1 --2- 172.21.15.160:0/1918074040 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f588c0f6510 0x7f588c106730 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f587c013510 tx=0x7f587c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:54.875 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:54.873+0000 7f5881ffb700 1 -- 172.21.15.160:0/1918074040 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f587c009320 con 0x7f588c0f6510 2023-11-17T05:25:54.876 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:54.873+0000 7f5881ffb700 1 -- 172.21.15.160:0/1918074040 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f587c013f00 con 0x7f588c0f6510 2023-11-17T05:25:54.876 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:54.873+0000 7f5893a40700 1 -- 172.21.15.160:0/1918074040 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f588c106e20 con 0x7f588c0f6510 2023-11-17T05:25:54.876 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:54.873+0000 7f5881ffb700 1 -- 172.21.15.160:0/1918074040 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f587c014d90 con 0x7f588c0f6510 2023-11-17T05:25:54.876 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:54.873+0000 7f5893a40700 1 -- 172.21.15.160:0/1918074040 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f588c1072b0 con 0x7f588c0f6510 2023-11-17T05:25:54.877 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:54.874+0000 7f5881ffb700 1 -- 172.21.15.160:0/1918074040 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f587c0263a0 con 0x7f588c0f6510 2023-11-17T05:25:54.877 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:54.874+0000 7f5881ffb700 1 --2- 172.21.15.160:0/1918074040 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f587802ff60 0x7f5878032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:54.877 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:54.874+0000 7f5881ffb700 1 -- 172.21.15.160:0/1918074040 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f587c041f10 con 0x7f588c0f6510 2023-11-17T05:25:54.877 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:54.874+0000 7f5893a40700 1 -- 172.21.15.160:0/1918074040 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f588c059100 con 0x7f588c0f6510 2023-11-17T05:25:54.877 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:54.875+0000 7f5890fdb700 1 --2- 172.21.15.160:0/1918074040 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f587802ff60 0x7f5878032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:54.877 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:54.875+0000 7f5890fdb700 1 --2- 172.21.15.160:0/1918074040 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f587802ff60 0x7f5878032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f58880096a0 tx=0x7f5888006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:54.880 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:54.879+0000 7f5881ffb700 1 -- 172.21.15.160:0/1918074040 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f587c012920 con 0x7f588c0f6510 2023-11-17T05:25:55.053 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:55.052+0000 7f5893a40700 1 -- 172.21.15.160:0/1918074040 --> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi160", "target": ["mon-mgr", ""]}) v1 -- 0x7f588c0f2e70 con 0x7f587802ff60 2023-11-17T05:25:55.653 INFO:teuthology.orchestra.run.smithi160.stdout:Added host 'smithi160' 2023-11-17T05:25:55.654 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:55.648+0000 7f5881ffb700 1 -- 172.21.15.160:0/1918074040 <== mgr.14164 v2:172.21.15.3:6800/17534166 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f588c0f2e70 con 0x7f587802ff60 2023-11-17T05:25:55.654 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:55.651+0000 7f5893a40700 1 -- 172.21.15.160:0/1918074040 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f587802ff60 msgr2=0x7f5878032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:55.654 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:55.651+0000 7f5893a40700 1 --2- 172.21.15.160:0/1918074040 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f587802ff60 0x7f5878032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f58880096a0 tx=0x7f5888006b40).stop 2023-11-17T05:25:55.654 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:55.651+0000 7f5893a40700 1 -- 172.21.15.160:0/1918074040 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f588c0f6510 msgr2=0x7f588c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:55.654 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:55.651+0000 7f5893a40700 1 --2- 172.21.15.160:0/1918074040 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f588c0f6510 0x7f588c106730 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f587c013510 tx=0x7f587c00b040).stop 2023-11-17T05:25:55.655 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:55.651+0000 7f5893a40700 1 -- 172.21.15.160:0/1918074040 shutdown_connections 2023-11-17T05:25:55.655 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:55.651+0000 7f5893a40700 1 --2- 172.21.15.160:0/1918074040 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f587802ff60 0x7f5878032420 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:55.655 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:55.651+0000 7f5893a40700 1 --2- 172.21.15.160:0/1918074040 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f588c0f6510 0x7f588c106730 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:55.655 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:55.651+0000 7f5893a40700 1 -- 172.21.15.160:0/1918074040 >> 172.21.15.160:0/1918074040 conn(0x7f588c0f1c70 msgr2=0x7f588c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:55.655 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:55.651+0000 7f5893a40700 1 -- 172.21.15.160:0/1918074040 shutdown_connections 2023-11-17T05:25:55.655 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:55.651+0000 7f5893a40700 1 -- 172.21.15.160:0/1918074040 wait complete. 2023-11-17T05:25:55.988 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph orch host ls --format=json 2023-11-17T05:25:57.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:56 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:55.054357+0000 mgr.smithi003.eddcuc (mgr.14164) 10 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi160", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:25:57.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:56 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:25:55.392024+0000 mgr.smithi003.eddcuc (mgr.14164) 11 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:25:57.021 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:56 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:55.648695+0000 mon.smithi003 (mon.0) 117 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' 2023-11-17T05:25:57.021 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:56 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cephadm 2023-11-17T05:25:55.649140+0000 mgr.smithi003.eddcuc (mgr.14164) 12 : cephadm [INF] Added host smithi160 2023-11-17T05:25:57.184 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:57.182+0000 7fe00759e700 1 -- 172.21.15.160:0/2089863349 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe0080f51a0 msgr2=0x7fe0080f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:57.184 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:57.182+0000 7fe00759e700 1 --2- 172.21.15.160:0/2089863349 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe0080f51a0 0x7fe0080f55c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fdffc009fb0 tx=0x7fdffc009b30).stop 2023-11-17T05:25:57.184 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:57.182+0000 7fe00759e700 1 -- 172.21.15.160:0/2089863349 shutdown_connections 2023-11-17T05:25:57.184 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:57.182+0000 7fe00759e700 1 --2- 172.21.15.160:0/2089863349 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe0080f51a0 0x7fe0080f55c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:57.184 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:57.182+0000 7fe00759e700 1 -- 172.21.15.160:0/2089863349 >> 172.21.15.160:0/2089863349 conn(0x7fe0080f09a0 msgr2=0x7fe0080f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:57.184 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:57.182+0000 7fe00759e700 1 -- 172.21.15.160:0/2089863349 shutdown_connections 2023-11-17T05:25:57.184 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:57.182+0000 7fe00759e700 1 -- 172.21.15.160:0/2089863349 wait complete. 2023-11-17T05:25:57.184 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:57.182+0000 7fe00759e700 1 Processor -- start 2023-11-17T05:25:57.184 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:57.182+0000 7fe00759e700 1 -- start start 2023-11-17T05:25:57.184 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:57.183+0000 7fe00759e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe0080f51a0 0x7fe0081086d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:57.185 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:57.183+0000 7fe00759e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe008108bc0 con 0x7fe0080f51a0 2023-11-17T05:25:57.185 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:57.183+0000 7fe00659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe0080f51a0 0x7fe0081086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:57.185 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:57.183+0000 7fe00659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe0080f51a0 0x7fe0081086d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:52068/0 (socket says 172.21.15.160:52068) 2023-11-17T05:25:57.185 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:57.183+0000 7fe00659c700 1 -- 172.21.15.160:0/1403435389 learned_addr learned my addr 172.21.15.160:0/1403435389 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:25:57.185 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:57.183+0000 7fe00659c700 1 -- 172.21.15.160:0/1403435389 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdffc005040 con 0x7fe0080f51a0 2023-11-17T05:25:57.186 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:57.183+0000 7fe00659c700 1 --2- 172.21.15.160:0/1403435389 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe0080f51a0 0x7fe0081086d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fdffc013510 tx=0x7fdffc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:57.186 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:57.183+0000 7fdff6ffd700 1 -- 172.21.15.160:0/1403435389 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdffc009320 con 0x7fe0080f51a0 2023-11-17T05:25:57.186 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:57.183+0000 7fe00759e700 1 -- 172.21.15.160:0/1403435389 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe008108dc0 con 0x7fe0080f51a0 2023-11-17T05:25:57.186 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:57.183+0000 7fdff6ffd700 1 -- 172.21.15.160:0/1403435389 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdffc013f00 con 0x7fe0080f51a0 2023-11-17T05:25:57.186 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:57.183+0000 7fdff6ffd700 1 -- 172.21.15.160:0/1403435389 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdffc014dc0 con 0x7fe0080f51a0 2023-11-17T05:25:57.186 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:57.183+0000 7fe00759e700 1 -- 172.21.15.160:0/1403435389 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe008109250 con 0x7fe0080f51a0 2023-11-17T05:25:57.186 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:57.184+0000 7fe00759e700 1 -- 172.21.15.160:0/1403435389 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe008045ad0 con 0x7fe0080f51a0 2023-11-17T05:25:57.186 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:57.184+0000 7fdff6ffd700 1 -- 172.21.15.160:0/1403435389 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdffc0095f0 con 0x7fe0080f51a0 2023-11-17T05:25:57.186 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:57.184+0000 7fdff6ffd700 1 --2- 172.21.15.160:0/1403435389 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fdff002ffb0 0x7fdff0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:57.187 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:57.184+0000 7fdff6ffd700 1 -- 172.21.15.160:0/1403435389 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdffc0410a0 con 0x7fe0080f51a0 2023-11-17T05:25:57.187 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:57.185+0000 7fe005d9b700 1 --2- 172.21.15.160:0/1403435389 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fdff002ffb0 0x7fdff0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:57.187 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:57.185+0000 7fe005d9b700 1 --2- 172.21.15.160:0/1403435389 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fdff002ffb0 0x7fdff0032470 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fdff80096a0 tx=0x7fdff8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:57.190 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:57.189+0000 7fdff6ffd700 1 -- 172.21.15.160:0/1403435389 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdffc06d2b0 con 0x7fe0080f51a0 2023-11-17T05:25:57.359 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:57.359+0000 7fe00759e700 1 -- 172.21.15.160:0/1403435389 --> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe0080f2790 con 0x7fdff002ffb0 2023-11-17T05:25:57.361 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:57.360+0000 7fdff6ffd700 1 -- 172.21.15.160:0/1403435389 <== mgr.14164 v2:172.21.15.3:6800/17534166 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+155 (secure 0 0 0) 0x7fe0080f2790 con 0x7fdff002ffb0 2023-11-17T05:25:57.361 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:25:57.361 INFO:teuthology.orchestra.run.smithi160.stdout:[{"addr": "172.21.15.3", "hostname": "smithi003", "labels": [], "status": ""}, {"addr": "smithi160", "hostname": "smithi160", "labels": [], "status": ""}] 2023-11-17T05:25:57.363 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:57.362+0000 7fe00759e700 1 -- 172.21.15.160:0/1403435389 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fdff002ffb0 msgr2=0x7fdff0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:57.363 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:57.362+0000 7fe00759e700 1 --2- 172.21.15.160:0/1403435389 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fdff002ffb0 0x7fdff0032470 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fdff80096a0 tx=0x7fdff8006b40).stop 2023-11-17T05:25:57.363 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:57.362+0000 7fe00759e700 1 -- 172.21.15.160:0/1403435389 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe0080f51a0 msgr2=0x7fe0081086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:57.363 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:57.362+0000 7fe00759e700 1 --2- 172.21.15.160:0/1403435389 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe0080f51a0 0x7fe0081086d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fdffc013510 tx=0x7fdffc00b040).stop 2023-11-17T05:25:57.363 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:57.362+0000 7fe00759e700 1 -- 172.21.15.160:0/1403435389 shutdown_connections 2023-11-17T05:25:57.363 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:57.362+0000 7fe00759e700 1 --2- 172.21.15.160:0/1403435389 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fdff002ffb0 0x7fdff0032470 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:57.363 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:57.362+0000 7fe00759e700 1 --2- 172.21.15.160:0/1403435389 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe0080f51a0 0x7fe0081086d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:57.363 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:57.362+0000 7fe00759e700 1 -- 172.21.15.160:0/1403435389 >> 172.21.15.160:0/1403435389 conn(0x7fe0080f09a0 msgr2=0x7fe00805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:57.363 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:57.362+0000 7fe00759e700 1 -- 172.21.15.160:0/1403435389 shutdown_connections 2023-11-17T05:25:57.364 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:25:57.362+0000 7fe00759e700 1 -- 172.21.15.160:0/1403435389 wait complete. 2023-11-17T05:25:57.785 INFO:tasks.cephadm:Setting crush tunables to default 2023-11-17T05:25:57.786 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph osd crush tunables default 2023-11-17T05:25:58.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:58 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:57.360884+0000 mgr.smithi003.eddcuc (mgr.14164) 13 : audit [DBG] from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-17T05:25:58.771 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:58 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:25:57.392294+0000 mgr.smithi003.eddcuc (mgr.14164) 14 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:25:59.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:59.031+0000 7f51c9ee3700 1 -- 172.21.15.3:0/3550884683 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51c40f51a0 msgr2=0x7f51c40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:59.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:59.031+0000 7f51c9ee3700 1 --2- 172.21.15.3:0/3550884683 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51c40f51a0 0x7f51c40f55c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f51b8009fb0 tx=0x7f51b8009b30).stop 2023-11-17T05:25:59.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:59.031+0000 7f51c9ee3700 1 -- 172.21.15.3:0/3550884683 shutdown_connections 2023-11-17T05:25:59.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:59.031+0000 7f51c9ee3700 1 --2- 172.21.15.3:0/3550884683 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51c40f51a0 0x7f51c40f55c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:59.032 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:59.031+0000 7f51c9ee3700 1 -- 172.21.15.3:0/3550884683 >> 172.21.15.3:0/3550884683 conn(0x7f51c40f09a0 msgr2=0x7f51c40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:59.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:59.031+0000 7f51c9ee3700 1 -- 172.21.15.3:0/3550884683 shutdown_connections 2023-11-17T05:25:59.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:59.031+0000 7f51c9ee3700 1 -- 172.21.15.3:0/3550884683 wait complete. 2023-11-17T05:25:59.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:59.032+0000 7f51c9ee3700 1 Processor -- start 2023-11-17T05:25:59.033 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:59.032+0000 7f51c9ee3700 1 -- start start 2023-11-17T05:25:59.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:59.032+0000 7f51c9ee3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51c4060200 0x7f51c4109340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:59.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:59.032+0000 7f51c9ee3700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c4109830 con 0x7f51c4060200 2023-11-17T05:25:59.034 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:59.032+0000 7f51c8ee1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51c4060200 0x7f51c4109340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:59.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:59.032+0000 7f51c8ee1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51c4060200 0x7f51c4109340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:37042/0 (socket says 172.21.15.3:37042) 2023-11-17T05:25:59.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:59.032+0000 7f51c8ee1700 1 -- 172.21.15.3:0/2046986006 learned_addr learned my addr 172.21.15.3:0/2046986006 (peer_addr_for_me v2:172.21.15.3:0/0) 2023-11-17T05:25:59.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:59.032+0000 7f51c8ee1700 1 -- 172.21.15.3:0/2046986006 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51b8005040 con 0x7f51c4060200 2023-11-17T05:25:59.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:59.033+0000 7f51c8ee1700 1 --2- 172.21.15.3:0/2046986006 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51c4060200 0x7f51c4109340 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f51b8009f80 tx=0x7f51b8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:59.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:59.033+0000 7f51c17fa700 1 -- 172.21.15.3:0/2046986006 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f51b80136c0 con 0x7f51c4060200 2023-11-17T05:25:59.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:59.033+0000 7f51c17fa700 1 -- 172.21.15.3:0/2046986006 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f51b80095f0 con 0x7f51c4060200 2023-11-17T05:25:59.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:59.033+0000 7f51c17fa700 1 -- 172.21.15.3:0/2046986006 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f51b8014420 con 0x7f51c4060200 2023-11-17T05:25:59.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:59.033+0000 7f51c9ee3700 1 -- 172.21.15.3:0/2046986006 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51c4109970 con 0x7f51c4060200 2023-11-17T05:25:59.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:59.033+0000 7f51c9ee3700 1 -- 172.21.15.3:0/2046986006 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51c4109da0 con 0x7f51c4060200 2023-11-17T05:25:59.035 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:59.034+0000 7f51c9ee3700 1 -- 172.21.15.3:0/2046986006 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51c4045ad0 con 0x7f51c4060200 2023-11-17T05:25:59.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:59.034+0000 7f51c17fa700 1 -- 172.21.15.3:0/2046986006 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f51b8025060 con 0x7f51c4060200 2023-11-17T05:25:59.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:59.035+0000 7f51c17fa700 1 --2- 172.21.15.3:0/2046986006 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f51b402ff60 0x7f51b4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:25:59.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:59.035+0000 7f51c17fa700 1 -- 172.21.15.3:0/2046986006 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f51b8041000 con 0x7f51c4060200 2023-11-17T05:25:59.036 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:59.035+0000 7f51c3fff700 1 --2- 172.21.15.3:0/2046986006 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f51b402ff60 0x7f51b4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:25:59.037 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:59.035+0000 7f51c3fff700 1 --2- 172.21.15.3:0/2046986006 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f51b402ff60 0x7f51b4032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f51b00096a0 tx=0x7f51b0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:25:59.039 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:59.039+0000 7f51c17fa700 1 -- 172.21.15.3:0/2046986006 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f51b801c950 con 0x7f51c4060200 2023-11-17T05:25:59.209 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:59.209+0000 7f51c9ee3700 1 -- 172.21.15.3:0/2046986006 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f51c40facd0 con 0x7f51c4060200 2023-11-17T05:25:59.915 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:59.914+0000 7f51c17fa700 1 -- 172.21.15.3:0/2046986006 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7f51b801cb60 con 0x7f51c4060200 2023-11-17T05:25:59.917 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:59.917+0000 7f51c9ee3700 1 -- 172.21.15.3:0/2046986006 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f51b402ff60 msgr2=0x7f51b4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:59.918 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:59.917+0000 7f51c9ee3700 1 --2- 172.21.15.3:0/2046986006 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f51b402ff60 0x7f51b4032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f51b00096a0 tx=0x7f51b0006b40).stop 2023-11-17T05:25:59.918 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:59.917+0000 7f51c9ee3700 1 -- 172.21.15.3:0/2046986006 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51c4060200 msgr2=0x7f51c4109340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:25:59.918 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:59.917+0000 7f51c9ee3700 1 --2- 172.21.15.3:0/2046986006 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51c4060200 0x7f51c4109340 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f51b8009f80 tx=0x7f51b8009b30).stop 2023-11-17T05:25:59.918 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:59.917+0000 7f51c9ee3700 1 -- 172.21.15.3:0/2046986006 shutdown_connections 2023-11-17T05:25:59.918 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:59.917+0000 7f51c9ee3700 1 --2- 172.21.15.3:0/2046986006 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f51b402ff60 0x7f51b4032420 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:59.918 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:59.917+0000 7f51c9ee3700 1 --2- 172.21.15.3:0/2046986006 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f51c4060200 0x7f51c4109340 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:25:59.931 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:59.917+0000 7f51c9ee3700 1 -- 172.21.15.3:0/2046986006 >> 172.21.15.3:0/2046986006 conn(0x7f51c40f09a0 msgr2=0x7f51c40f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:25:59.931 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:59.917+0000 7f51c9ee3700 1 -- 172.21.15.3:0/2046986006 shutdown_connections 2023-11-17T05:25:59.931 INFO:teuthology.orchestra.run.smithi003.stderr:2023-11-17T05:25:59.917+0000 7f51c9ee3700 1 -- 172.21.15.3:0/2046986006 wait complete. 2023-11-17T05:25:59.931 INFO:teuthology.orchestra.run.smithi003.stderr:adjusted tunables profile to default 2023-11-17T05:26:00.231 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:59 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:59.210168+0000 mon.smithi003 (mon.0) 118 : audit [INF] from='client.? 172.21.15.3:0/2046986006' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-11-17T05:26:00.231 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:25:59 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:25:59.392574+0000 mgr.smithi003.eddcuc (mgr.14164) 15 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:26:00.338 INFO:tasks.cephadm:Adding mon.smithi003 on smithi003 2023-11-17T05:26:00.338 INFO:tasks.cephadm:Adding mon.smithi160 on smithi160 2023-11-17T05:26:00.338 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph orch apply mon '2;smithi003:172.21.15.3=smithi003;smithi160:172.21.15.160=smithi160' 2023-11-17T05:26:01.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:26:00 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:25:59.915269+0000 mon.smithi003 (mon.0) 119 : audit [INF] from='client.? 172.21.15.3:0/2046986006' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-11-17T05:26:01.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:26:00 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:25:59.915339+0000 mon.smithi003 (mon.0) 120 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2023-11-17T05:26:01.465 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:01.464+0000 7f88dd599700 1 -- 172.21.15.160:0/4118604816 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88d80f51a0 msgr2=0x7f88d80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:26:01.465 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:01.464+0000 7f88dd599700 1 --2- 172.21.15.160:0/4118604816 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88d80f51a0 0x7f88d80f55c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f88c0009fb0 tx=0x7f88c0009b30).stop 2023-11-17T05:26:01.465 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:01.464+0000 7f88dd599700 1 -- 172.21.15.160:0/4118604816 shutdown_connections 2023-11-17T05:26:01.465 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:01.464+0000 7f88dd599700 1 --2- 172.21.15.160:0/4118604816 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88d80f51a0 0x7f88d80f55c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:26:01.466 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:01.464+0000 7f88dd599700 1 -- 172.21.15.160:0/4118604816 >> 172.21.15.160:0/4118604816 conn(0x7f88d80f09a0 msgr2=0x7f88d80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:26:01.466 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:01.464+0000 7f88dd599700 1 -- 172.21.15.160:0/4118604816 shutdown_connections 2023-11-17T05:26:01.466 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:01.464+0000 7f88dd599700 1 -- 172.21.15.160:0/4118604816 wait complete. 2023-11-17T05:26:01.466 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:01.464+0000 7f88dd599700 1 Processor -- start 2023-11-17T05:26:01.466 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:01.465+0000 7f88dd599700 1 -- start start 2023-11-17T05:26:01.467 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:01.465+0000 7f88dd599700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88d80f51a0 0x7f88d81053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:26:01.468 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:01.465+0000 7f88dd599700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d81058a0 con 0x7f88d80f51a0 2023-11-17T05:26:01.468 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:01.465+0000 7f88d7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88d80f51a0 0x7f88d81053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:26:01.468 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:01.465+0000 7f88d7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88d80f51a0 0x7f88d81053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:33830/0 (socket says 172.21.15.160:33830) 2023-11-17T05:26:01.468 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:01.465+0000 7f88d7fff700 1 -- 172.21.15.160:0/3695560752 learned_addr learned my addr 172.21.15.160:0/3695560752 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:26:01.468 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:01.465+0000 7f88d7fff700 1 -- 172.21.15.160:0/3695560752 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c0005040 con 0x7f88d80f51a0 2023-11-17T05:26:01.468 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:01.465+0000 7f88d7fff700 1 --2- 172.21.15.160:0/3695560752 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88d80f51a0 0x7f88d81053b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f88c0013510 tx=0x7f88c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:26:01.468 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:01.466+0000 7f88d4ff9700 1 -- 172.21.15.160:0/3695560752 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88c0009320 con 0x7f88d80f51a0 2023-11-17T05:26:01.468 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:01.466+0000 7f88dd599700 1 -- 172.21.15.160:0/3695560752 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88d8105aa0 con 0x7f88d80f51a0 2023-11-17T05:26:01.468 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:01.466+0000 7f88d4ff9700 1 -- 172.21.15.160:0/3695560752 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f88c0013f00 con 0x7f88d80f51a0 2023-11-17T05:26:01.468 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:01.466+0000 7f88d4ff9700 1 -- 172.21.15.160:0/3695560752 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88c0014dc0 con 0x7f88d80f51a0 2023-11-17T05:26:01.468 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:01.466+0000 7f88dd599700 1 -- 172.21.15.160:0/3695560752 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88d8105f30 con 0x7f88d80f51a0 2023-11-17T05:26:01.468 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:01.467+0000 7f88d4ff9700 1 -- 172.21.15.160:0/3695560752 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f88c00095f0 con 0x7f88d80f51a0 2023-11-17T05:26:01.468 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:01.467+0000 7f88dd599700 1 -- 172.21.15.160:0/3695560752 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88d8045ad0 con 0x7f88d80f51a0 2023-11-17T05:26:01.469 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:01.467+0000 7f88d4ff9700 1 --2- 172.21.15.160:0/3695560752 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f88c402ff60 0x7f88c4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:26:01.469 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:01.467+0000 7f88d4ff9700 1 -- 172.21.15.160:0/3695560752 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f88c0041040 con 0x7f88d80f51a0 2023-11-17T05:26:01.469 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:01.467+0000 7f88d77fe700 1 --2- 172.21.15.160:0/3695560752 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f88c402ff60 0x7f88c4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:26:01.469 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:01.468+0000 7f88d77fe700 1 --2- 172.21.15.160:0/3695560752 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f88c402ff60 0x7f88c4032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f88c80096a0 tx=0x7f88c8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:26:01.472 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:01.472+0000 7f88d4ff9700 1 -- 172.21.15.160:0/3695560752 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f88c0014420 con 0x7f88d80f51a0 2023-11-17T05:26:01.649 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:01.648+0000 7f88dd599700 1 -- 172.21.15.160:0/3695560752 --> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi003:172.21.15.3=smithi003;smithi160:172.21.15.160=smithi160", "target": ["mon-mgr", ""]}) v1 -- 0x7f88d80f1c20 con 0x7f88c402ff60 2023-11-17T05:26:01.654 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:01.654+0000 7f88d4ff9700 1 -- 172.21.15.160:0/3695560752 <== mgr.14164 v2:172.21.15.3:6800/17534166 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f88d80f1c20 con 0x7f88c402ff60 2023-11-17T05:26:01.655 INFO:teuthology.orchestra.run.smithi160.stdout:Scheduled mon update... 2023-11-17T05:26:01.656 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:01.655+0000 7f88dd599700 1 -- 172.21.15.160:0/3695560752 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f88c402ff60 msgr2=0x7f88c4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:26:01.656 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:01.655+0000 7f88dd599700 1 --2- 172.21.15.160:0/3695560752 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f88c402ff60 0x7f88c4032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f88c80096a0 tx=0x7f88c8006b40).stop 2023-11-17T05:26:01.656 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:01.655+0000 7f88dd599700 1 -- 172.21.15.160:0/3695560752 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88d80f51a0 msgr2=0x7f88d81053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:26:01.656 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:01.655+0000 7f88dd599700 1 --2- 172.21.15.160:0/3695560752 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88d80f51a0 0x7f88d81053b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f88c0013510 tx=0x7f88c000b040).stop 2023-11-17T05:26:01.657 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:01.655+0000 7f88dd599700 1 -- 172.21.15.160:0/3695560752 shutdown_connections 2023-11-17T05:26:01.657 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:01.655+0000 7f88dd599700 1 --2- 172.21.15.160:0/3695560752 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f88c402ff60 0x7f88c4032420 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:26:01.657 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:01.655+0000 7f88dd599700 1 --2- 172.21.15.160:0/3695560752 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88d80f51a0 0x7f88d81053b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:26:01.657 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:01.655+0000 7f88dd599700 1 -- 172.21.15.160:0/3695560752 >> 172.21.15.160:0/3695560752 conn(0x7f88d80f09a0 msgr2=0x7f88d80f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:26:01.657 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:01.655+0000 7f88dd599700 1 -- 172.21.15.160:0/3695560752 shutdown_connections 2023-11-17T05:26:01.657 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:01.655+0000 7f88dd599700 1 -- 172.21.15.160:0/3695560752 wait complete. 2023-11-17T05:26:02.020 DEBUG:teuthology.orchestra.run.smithi160:mon.smithi160> sudo journalctl -f -n 0 -u ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7@mon.smithi160.service 2023-11-17T05:26:02.022 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:26:02.022 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:26:02.050 INFO:journalctl@ceph.mon.smithi160.smithi160.stdout:-- Logs begin at Fri 2023-11-17 05:06:51 UTC. -- 2023-11-17T05:26:03.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:26:02 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:26:01.392889+0000 mgr.smithi003.eddcuc (mgr.14164) 16 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:23.587 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:26:02 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:26:01.650400+0000 mgr.smithi003.eddcuc (mgr.14164) 17 : audit [DBG] from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi003:172.21.15.3=smithi003;smithi160:172.21.15.160=smithi160", "target": ["mon-mgr", ""]}]: dispatch 2023-11-17T05:27:23.587 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:26:02 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cephadm 2023-11-17T05:26:01.651934+0000 mgr.smithi003.eddcuc (mgr.14164) 18 : cephadm [INF] Saving service mon spec with placement smithi003:172.21.15.3=smithi003;smithi160:172.21.15.160=smithi160;count:2 2023-11-17T05:27:23.587 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:26:02 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:26:01.654642+0000 mon.smithi003 (mon.0) 121 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' 2023-11-17T05:27:23.631 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:03.167+0000 7f883e8e8700 1 -- 172.21.15.160:0/2537035701 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8838060280 msgr2=0x7f88380f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:27:23.646 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:03.167+0000 7f883e8e8700 1 --2- 172.21.15.160:0/2537035701 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8838060280 0x7f88380f90f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f882c009fb0 tx=0x7f882c009b30).stop 2023-11-17T05:27:23.662 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:03.167+0000 7f883e8e8700 1 -- 172.21.15.160:0/2537035701 shutdown_connections 2023-11-17T05:27:23.672 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:03.167+0000 7f883e8e8700 1 --2- 172.21.15.160:0/2537035701 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8838060280 0x7f88380f90f0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:27:23.713 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:03.167+0000 7f883e8e8700 1 -- 172.21.15.160:0/2537035701 >> 172.21.15.160:0/2537035701 conn(0x7f88380f0b70 msgr2=0x7f88380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:27:23.713 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:03.167+0000 7f883e8e8700 1 -- 172.21.15.160:0/2537035701 shutdown_connections 2023-11-17T05:27:23.713 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:03.167+0000 7f883e8e8700 1 -- 172.21.15.160:0/2537035701 wait complete. 2023-11-17T05:27:23.713 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:03.168+0000 7f883e8e8700 1 Processor -- start 2023-11-17T05:27:23.713 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:03.168+0000 7f883e8e8700 1 -- start start 2023-11-17T05:27:23.713 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:03.168+0000 7f883e8e8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8838060280 0x7f88381066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:27:23.713 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:03.168+0000 7f883e8e8700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8838106bd0 con 0x7f8838060280 2023-11-17T05:27:23.713 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:03.168+0000 7f8837fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8838060280 0x7f88381066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:27:23.713 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:03.168+0000 7f8837fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8838060280 0x7f88381066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:33844/0 (socket says 172.21.15.160:33844) 2023-11-17T05:27:23.713 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:03.168+0000 7f8837fff700 1 -- 172.21.15.160:0/2067486534 learned_addr learned my addr 172.21.15.160:0/2067486534 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:27:23.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:03.168+0000 7f8837fff700 1 -- 172.21.15.160:0/2067486534 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f882c005040 con 0x7f8838060280 2023-11-17T05:27:23.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:03.169+0000 7f8837fff700 1 --2- 172.21.15.160:0/2067486534 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8838060280 0x7f88381066e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f882c013510 tx=0x7f882c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:27:23.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:03.169+0000 7f8834ff9700 1 -- 172.21.15.160:0/2067486534 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f882c009320 con 0x7f8838060280 2023-11-17T05:27:23.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:03.169+0000 7f883e8e8700 1 -- 172.21.15.160:0/2067486534 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8838106dd0 con 0x7f8838060280 2023-11-17T05:27:23.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:03.169+0000 7f8834ff9700 1 -- 172.21.15.160:0/2067486534 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f882c013f00 con 0x7f8838060280 2023-11-17T05:27:23.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:03.169+0000 7f8834ff9700 1 -- 172.21.15.160:0/2067486534 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f882c014dc0 con 0x7f8838060280 2023-11-17T05:27:23.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:03.169+0000 7f883e8e8700 1 -- 172.21.15.160:0/2067486534 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8838107260 con 0x7f8838060280 2023-11-17T05:27:23.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:03.170+0000 7f8834ff9700 1 -- 172.21.15.160:0/2067486534 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f882c01d3d0 con 0x7f8838060280 2023-11-17T05:27:23.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:03.170+0000 7f883e8e8700 1 -- 172.21.15.160:0/2067486534 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8838045ad0 con 0x7f8838060280 2023-11-17T05:27:23.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:03.170+0000 7f8834ff9700 1 --2- 172.21.15.160:0/2067486534 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f882002ff60 0x7f8820032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:27:23.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:03.170+0000 7f8834ff9700 1 -- 172.21.15.160:0/2067486534 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f882c0413c0 con 0x7f8838060280 2023-11-17T05:27:23.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:03.170+0000 7f88377fe700 1 --2- 172.21.15.160:0/2067486534 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f882002ff60 0x7f8820032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:27:23.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:03.170+0000 7f88377fe700 1 --2- 172.21.15.160:0/2067486534 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f882002ff60 0x7f8820032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f88280096a0 tx=0x7f8828006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:27:23.777 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:03.175+0000 7f8834ff9700 1 -- 172.21.15.160:0/2067486534 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f882c01ccb0 con 0x7f8838060280 2023-11-17T05:27:23.777 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:03.380+0000 7f883e8e8700 1 -- 172.21.15.160:0/2067486534 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8838108110 con 0x7f8838060280 2023-11-17T05:27:23.777 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:03.380+0000 7f8834ff9700 1 -- 172.21.15.160:0/2067486534 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f882c01cad0 con 0x7f8838060280 2023-11-17T05:27:23.778 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:27:23.778 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:27:23.781 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:03.382+0000 7f883e8e8700 1 -- 172.21.15.160:0/2067486534 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f882002ff60 msgr2=0x7f8820032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:27:23.843 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:03.382+0000 7f883e8e8700 1 --2- 172.21.15.160:0/2067486534 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f882002ff60 0x7f8820032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f88280096a0 tx=0x7f8828006b40).stop 2023-11-17T05:27:23.843 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:03.382+0000 7f883e8e8700 1 -- 172.21.15.160:0/2067486534 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8838060280 msgr2=0x7f88381066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:27:23.843 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:03.382+0000 7f883e8e8700 1 --2- 172.21.15.160:0/2067486534 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8838060280 0x7f88381066e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f882c013510 tx=0x7f882c00b040).stop 2023-11-17T05:27:23.843 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:03.382+0000 7f883e8e8700 1 -- 172.21.15.160:0/2067486534 shutdown_connections 2023-11-17T05:27:23.843 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:03.382+0000 7f883e8e8700 1 --2- 172.21.15.160:0/2067486534 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f882002ff60 0x7f8820032420 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:27:23.843 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:03.382+0000 7f883e8e8700 1 --2- 172.21.15.160:0/2067486534 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8838060280 0x7f88381066e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:27:23.843 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:03.382+0000 7f883e8e8700 1 -- 172.21.15.160:0/2067486534 >> 172.21.15.160:0/2067486534 conn(0x7f88380f0b70 msgr2=0x7f88380f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:27:23.843 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:03.382+0000 7f883e8e8700 1 -- 172.21.15.160:0/2067486534 shutdown_connections 2023-11-17T05:27:23.843 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:26:03.382+0000 7f883e8e8700 1 -- 172.21.15.160:0/2067486534 wait complete. 2023-11-17T05:27:23.844 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:27:23.845 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:26:03 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:26:03.381465+0000 mon.smithi003 (mon.0) 122 : audit [DBG] from='client.? 172.21.15.160:0/2067486534' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:27:23.845 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:26:04 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:26:03.393145+0000 mgr.smithi003.eddcuc (mgr.14164) 19 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:23.845 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:26:06 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:26:05.393387+0000 mgr.smithi003.eddcuc (mgr.14164) 20 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:23.845 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:26:08 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:26:07.393656+0000 mgr.smithi003.eddcuc (mgr.14164) 21 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:23.845 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:26:09 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:26:09.394003+0000 mgr.smithi003.eddcuc (mgr.14164) 22 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:23.845 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:26:12 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:26:11.394263+0000 mgr.smithi003.eddcuc (mgr.14164) 23 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:23.845 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:26:14 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:26:13.394556+0000 mgr.smithi003.eddcuc (mgr.14164) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:23.845 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:26:16 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:26:15.394875+0000 mgr.smithi003.eddcuc (mgr.14164) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:23.845 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:26:18 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:26:17.395164+0000 mgr.smithi003.eddcuc (mgr.14164) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:23.845 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:26:19 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:26:19.395527+0000 mgr.smithi003.eddcuc (mgr.14164) 27 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:23.846 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:26:22 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:26:21.395816+0000 mgr.smithi003.eddcuc (mgr.14164) 28 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:23.846 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:26:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:26:23.396150+0000 mgr.smithi003.eddcuc (mgr.14164) 29 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:23.846 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:26:26 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:26:25.396461+0000 mgr.smithi003.eddcuc (mgr.14164) 30 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:23.846 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:26:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:26:27.396827+0000 mgr.smithi003.eddcuc (mgr.14164) 31 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:23.883 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:26:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:26:27.546211+0000 mon.smithi003 (mon.0) 123 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.eddcuc/mirror_snapshot_schedule"}]: dispatch 2023-11-17T05:27:23.883 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:26:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:26:27.550302+0000 mon.smithi003 (mon.0) 124 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.eddcuc/trash_purge_schedule"}]: dispatch 2023-11-17T05:27:23.883 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:26:29 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:26:29.397142+0000 mgr.smithi003.eddcuc (mgr.14164) 32 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:23.883 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:26:32 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:26:31.397470+0000 mgr.smithi003.eddcuc (mgr.14164) 33 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:23.883 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:26:34 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:26:33.397775+0000 mgr.smithi003.eddcuc (mgr.14164) 34 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:23.883 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:26:36 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:26:35.398097+0000 mgr.smithi003.eddcuc (mgr.14164) 35 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:23.883 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:26:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:26:37.398456+0000 mgr.smithi003.eddcuc (mgr.14164) 36 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:23.884 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:26:39 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:26:39.398827+0000 mgr.smithi003.eddcuc (mgr.14164) 37 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:23.884 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:26:42 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:26:41.399175+0000 mgr.smithi003.eddcuc (mgr.14164) 38 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:23.884 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:26:44 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:26:43.399457+0000 mgr.smithi003.eddcuc (mgr.14164) 39 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:23.884 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:26:46 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:26:45.399782+0000 mgr.smithi003.eddcuc (mgr.14164) 40 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:23.884 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:26:48 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:26:47.400146+0000 mgr.smithi003.eddcuc (mgr.14164) 41 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:23.884 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:26:49 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:26:49.400413+0000 mgr.smithi003.eddcuc (mgr.14164) 42 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:23.884 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:26:52 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:26:51.400648+0000 mgr.smithi003.eddcuc (mgr.14164) 43 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:23.884 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:26:54 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:26:53.400890+0000 mgr.smithi003.eddcuc (mgr.14164) 44 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:23.884 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:26:56 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:26:55.401220+0000 mgr.smithi003.eddcuc (mgr.14164) 45 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:23.884 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:26:58 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:26:57.401651+0000 mgr.smithi003.eddcuc (mgr.14164) 46 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:23.884 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:26:59 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:26:59.402022+0000 mgr.smithi003.eddcuc (mgr.14164) 47 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:23.884 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:27:02 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:27:01.402371+0000 mgr.smithi003.eddcuc (mgr.14164) 48 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:23.884 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:27:04 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:27:03.402728+0000 mgr.smithi003.eddcuc (mgr.14164) 49 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:23.885 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:27:06 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:27:05.403019+0000 mgr.smithi003.eddcuc (mgr.14164) 50 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:23.885 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:27:08 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:27:07.403415+0000 mgr.smithi003.eddcuc (mgr.14164) 51 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:23.885 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:27:09 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:27:09.403857+0000 mgr.smithi003.eddcuc (mgr.14164) 52 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:23.885 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:27:12 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:27:11.404243+0000 mgr.smithi003.eddcuc (mgr.14164) 53 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:23.885 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:27:14 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:27:13.404652+0000 mgr.smithi003.eddcuc (mgr.14164) 54 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:23.885 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:27:16 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:27:15.405053+0000 mgr.smithi003.eddcuc (mgr.14164) 55 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:23.885 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:27:18 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:27:17.405462+0000 mgr.smithi003.eddcuc (mgr.14164) 56 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:23.885 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:27:19 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:27:19.405801+0000 mgr.smithi003.eddcuc (mgr.14164) 57 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:23.885 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:27:22 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:27:21.406066+0000 mgr.smithi003.eddcuc (mgr.14164) 58 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:24.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:27:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:27:23.406328+0000 mgr.smithi003.eddcuc (mgr.14164) 59 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:24.845 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:27:24.845 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:27:25.991 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:25.990+0000 7f7fbc8af700 1 -- 172.21.15.160:0/2811951368 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fb80f6510 msgr2=0x7f7fb80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:27:25.991 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:25.990+0000 7f7fbc8af700 1 --2- 172.21.15.160:0/2811951368 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fb80f6510 0x7f7fb80f6930 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f7fa8009fb0 tx=0x7f7fa8009b30).stop 2023-11-17T05:27:25.992 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:25.990+0000 7f7fbc8af700 1 -- 172.21.15.160:0/2811951368 shutdown_connections 2023-11-17T05:27:25.992 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:25.990+0000 7f7fbc8af700 1 --2- 172.21.15.160:0/2811951368 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fb80f6510 0x7f7fb80f6930 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:27:25.992 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:25.990+0000 7f7fbc8af700 1 -- 172.21.15.160:0/2811951368 >> 172.21.15.160:0/2811951368 conn(0x7f7fb80f1c70 msgr2=0x7f7fb80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:27:25.992 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:25.990+0000 7f7fbc8af700 1 -- 172.21.15.160:0/2811951368 shutdown_connections 2023-11-17T05:27:25.992 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:25.990+0000 7f7fbc8af700 1 -- 172.21.15.160:0/2811951368 wait complete. 2023-11-17T05:27:25.992 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:25.991+0000 7f7fbc8af700 1 Processor -- start 2023-11-17T05:27:25.992 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:25.991+0000 7f7fbc8af700 1 -- start start 2023-11-17T05:27:25.992 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:25.991+0000 7f7fbc8af700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fb80f6510 0x7f7fb8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:27:25.992 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:25.991+0000 7f7fbc8af700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb8106c20 con 0x7f7fb80f6510 2023-11-17T05:27:25.994 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:25.991+0000 7f7fb659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fb80f6510 0x7f7fb8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:27:25.994 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:25.991+0000 7f7fb659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fb80f6510 0x7f7fb8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:48170/0 (socket says 172.21.15.160:48170) 2023-11-17T05:27:25.994 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:25.991+0000 7f7fb659c700 1 -- 172.21.15.160:0/3734597871 learned_addr learned my addr 172.21.15.160:0/3734597871 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:27:25.994 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:25.991+0000 7f7fb659c700 1 -- 172.21.15.160:0/3734597871 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa8005040 con 0x7f7fb80f6510 2023-11-17T05:27:25.994 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:25.991+0000 7f7fb659c700 1 --2- 172.21.15.160:0/3734597871 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fb80f6510 0x7f7fb8106730 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f7fa8013510 tx=0x7f7fa800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:27:25.994 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:25.992+0000 7f7fa6ffd700 1 -- 172.21.15.160:0/3734597871 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7fa8009320 con 0x7f7fb80f6510 2023-11-17T05:27:25.994 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:25.992+0000 7f7fbc8af700 1 -- 172.21.15.160:0/3734597871 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fb8106e20 con 0x7f7fb80f6510 2023-11-17T05:27:25.994 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:25.992+0000 7f7fa6ffd700 1 -- 172.21.15.160:0/3734597871 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7fa8013f00 con 0x7f7fb80f6510 2023-11-17T05:27:25.995 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:25.992+0000 7f7fa6ffd700 1 -- 172.21.15.160:0/3734597871 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7fa8014d90 con 0x7f7fb80f6510 2023-11-17T05:27:25.995 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:25.992+0000 7f7fbc8af700 1 -- 172.21.15.160:0/3734597871 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fb81072b0 con 0x7f7fb80f6510 2023-11-17T05:27:25.995 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:25.992+0000 7f7fa6ffd700 1 -- 172.21.15.160:0/3734597871 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7fa8014420 con 0x7f7fb80f6510 2023-11-17T05:27:25.995 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:25.992+0000 7f7fbc8af700 1 -- 172.21.15.160:0/3734597871 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fb8045ad0 con 0x7f7fb80f6510 2023-11-17T05:27:25.995 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:25.993+0000 7f7fa6ffd700 1 --2- 172.21.15.160:0/3734597871 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f7fa002ff60 0x7f7fa0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:27:25.995 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:25.993+0000 7f7fa6ffd700 1 -- 172.21.15.160:0/3734597871 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7fa80410c0 con 0x7f7fb80f6510 2023-11-17T05:27:25.995 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:25.993+0000 7f7fb5d9b700 1 --2- 172.21.15.160:0/3734597871 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f7fa002ff60 0x7f7fa0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:27:25.995 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:25.993+0000 7f7fb5d9b700 1 --2- 172.21.15.160:0/3734597871 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f7fa002ff60 0x7f7fa0032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f7fac0096a0 tx=0x7f7fac006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:27:25.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:25.998+0000 7f7fa6ffd700 1 -- 172.21.15.160:0/3734597871 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7fa80095f0 con 0x7f7fb80f6510 2023-11-17T05:27:26.205 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:26.204+0000 7f7fbc8af700 1 -- 172.21.15.160:0/3734597871 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7fb80fbf60 con 0x7f7fb80f6510 2023-11-17T05:27:26.206 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:26.205+0000 7f7fa6ffd700 1 -- 172.21.15.160:0/3734597871 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f7fa801db70 con 0x7f7fb80f6510 2023-11-17T05:27:26.206 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:27:26.206 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:27:26.207 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:26.206+0000 7f7fbc8af700 1 -- 172.21.15.160:0/3734597871 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f7fa002ff60 msgr2=0x7f7fa0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:27:26.208 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:26.206+0000 7f7fbc8af700 1 --2- 172.21.15.160:0/3734597871 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f7fa002ff60 0x7f7fa0032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f7fac0096a0 tx=0x7f7fac006b40).stop 2023-11-17T05:27:26.208 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:26.206+0000 7f7fbc8af700 1 -- 172.21.15.160:0/3734597871 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fb80f6510 msgr2=0x7f7fb8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:27:26.208 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:26.206+0000 7f7fbc8af700 1 --2- 172.21.15.160:0/3734597871 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fb80f6510 0x7f7fb8106730 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f7fa8013510 tx=0x7f7fa800b040).stop 2023-11-17T05:27:26.208 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:26.206+0000 7f7fbc8af700 1 -- 172.21.15.160:0/3734597871 shutdown_connections 2023-11-17T05:27:26.208 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:26.206+0000 7f7fbc8af700 1 --2- 172.21.15.160:0/3734597871 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f7fa002ff60 0x7f7fa0032420 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:27:26.208 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:26.206+0000 7f7fbc8af700 1 --2- 172.21.15.160:0/3734597871 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7fb80f6510 0x7f7fb8106730 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:27:26.208 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:26.206+0000 7f7fbc8af700 1 -- 172.21.15.160:0/3734597871 >> 172.21.15.160:0/3734597871 conn(0x7f7fb80f1c70 msgr2=0x7f7fb80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:27:26.208 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:26.206+0000 7f7fbc8af700 1 -- 172.21.15.160:0/3734597871 shutdown_connections 2023-11-17T05:27:26.208 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:26.206+0000 7f7fbc8af700 1 -- 172.21.15.160:0/3734597871 wait complete. 2023-11-17T05:27:26.208 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:27:26.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:27:26 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:27:25.406589+0000 mgr.smithi003.eddcuc (mgr.14164) 60 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:26.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:27:26 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:27:26.205762+0000 mon.smithi003 (mon.0) 125 : audit [DBG] from='client.? 172.21.15.160:0/3734597871' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:27:27.582 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:27:27.582 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:27:28.773 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:27:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:27:27.407016+0000 mgr.smithi003.eddcuc (mgr.14164) 61 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:28.774 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:27:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:27:27.547442+0000 mon.smithi003 (mon.0) 126 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.eddcuc/mirror_snapshot_schedule"}]: dispatch 2023-11-17T05:27:28.774 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:27:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:27:27.551314+0000 mon.smithi003 (mon.0) 127 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.eddcuc/trash_purge_schedule"}]: dispatch 2023-11-17T05:27:28.783 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:28.781+0000 7fd389919700 1 -- 172.21.15.160:0/1043955533 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3840f62c0 msgr2=0x7fd3840f66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:27:28.783 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:28.781+0000 7fd389919700 1 --2- 172.21.15.160:0/1043955533 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3840f62c0 0x7fd3840f66e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fd374009fb0 tx=0x7fd374009b30).stop 2023-11-17T05:27:28.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:28.782+0000 7fd389919700 1 -- 172.21.15.160:0/1043955533 shutdown_connections 2023-11-17T05:27:28.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:28.782+0000 7fd389919700 1 --2- 172.21.15.160:0/1043955533 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3840f62c0 0x7fd3840f66e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:27:28.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:28.782+0000 7fd389919700 1 -- 172.21.15.160:0/1043955533 >> 172.21.15.160:0/1043955533 conn(0x7fd3840f1a20 msgr2=0x7fd3840f3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:27:28.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:28.782+0000 7fd389919700 1 -- 172.21.15.160:0/1043955533 shutdown_connections 2023-11-17T05:27:28.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:28.782+0000 7fd389919700 1 -- 172.21.15.160:0/1043955533 wait complete. 2023-11-17T05:27:28.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:28.782+0000 7fd389919700 1 Processor -- start 2023-11-17T05:27:28.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:28.782+0000 7fd389919700 1 -- start start 2023-11-17T05:27:28.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:28.783+0000 7fd389919700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3840f62c0 0x7fd384106580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:27:28.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:28.783+0000 7fd389919700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd384106a70 con 0x7fd3840f62c0 2023-11-17T05:27:28.786 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:28.783+0000 7fd382ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3840f62c0 0x7fd384106580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:27:28.787 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:28.783+0000 7fd382ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3840f62c0 0x7fd384106580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:48194/0 (socket says 172.21.15.160:48194) 2023-11-17T05:27:28.787 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:28.783+0000 7fd382ffd700 1 -- 172.21.15.160:0/2772998683 learned_addr learned my addr 172.21.15.160:0/2772998683 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:27:28.787 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:28.783+0000 7fd382ffd700 1 -- 172.21.15.160:0/2772998683 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd374005040 con 0x7fd3840f62c0 2023-11-17T05:27:28.787 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:28.783+0000 7fd382ffd700 1 --2- 172.21.15.160:0/2772998683 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3840f62c0 0x7fd384106580 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fd374013510 tx=0x7fd37400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:27:28.787 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:28.784+0000 7fd36b7fe700 1 -- 172.21.15.160:0/2772998683 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd374009320 con 0x7fd3840f62c0 2023-11-17T05:27:28.788 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:28.784+0000 7fd36b7fe700 1 -- 172.21.15.160:0/2772998683 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd374013f00 con 0x7fd3840f62c0 2023-11-17T05:27:28.788 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:28.784+0000 7fd389919700 1 -- 172.21.15.160:0/2772998683 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd384106c70 con 0x7fd3840f62c0 2023-11-17T05:27:28.788 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:28.784+0000 7fd36b7fe700 1 -- 172.21.15.160:0/2772998683 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd374014d90 con 0x7fd3840f62c0 2023-11-17T05:27:28.788 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:28.784+0000 7fd389919700 1 -- 172.21.15.160:0/2772998683 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd384107100 con 0x7fd3840f62c0 2023-11-17T05:27:28.788 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:28.784+0000 7fd36b7fe700 1 -- 172.21.15.160:0/2772998683 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd374014420 con 0x7fd3840f62c0 2023-11-17T05:27:28.788 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:28.784+0000 7fd389919700 1 -- 172.21.15.160:0/2772998683 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd384045ad0 con 0x7fd3840f62c0 2023-11-17T05:27:28.789 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:28.785+0000 7fd36b7fe700 1 --2- 172.21.15.160:0/2772998683 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fd36c02ff60 0x7fd36c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:27:28.789 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:28.785+0000 7fd36b7fe700 1 -- 172.21.15.160:0/2772998683 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd3740410c0 con 0x7fd3840f62c0 2023-11-17T05:27:28.789 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:28.785+0000 7fd3827fc700 1 --2- 172.21.15.160:0/2772998683 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fd36c02ff60 0x7fd36c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:27:28.789 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:28.785+0000 7fd3827fc700 1 --2- 172.21.15.160:0/2772998683 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fd36c02ff60 0x7fd36c032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fd3780096a0 tx=0x7fd378006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:27:28.790 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:28.789+0000 7fd36b7fe700 1 -- 172.21.15.160:0/2772998683 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd3740095f0 con 0x7fd3840f62c0 2023-11-17T05:27:28.992 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:28.991+0000 7fd389919700 1 -- 172.21.15.160:0/2772998683 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd384107f00 con 0x7fd3840f62c0 2023-11-17T05:27:28.993 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:28.992+0000 7fd36b7fe700 1 -- 172.21.15.160:0/2772998683 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fd37401db70 con 0x7fd3840f62c0 2023-11-17T05:27:28.993 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:27:28.994 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:27:28.995 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:28.994+0000 7fd389919700 1 -- 172.21.15.160:0/2772998683 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fd36c02ff60 msgr2=0x7fd36c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:27:28.995 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:28.994+0000 7fd389919700 1 --2- 172.21.15.160:0/2772998683 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fd36c02ff60 0x7fd36c032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fd3780096a0 tx=0x7fd378006b40).stop 2023-11-17T05:27:28.995 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:28.994+0000 7fd389919700 1 -- 172.21.15.160:0/2772998683 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3840f62c0 msgr2=0x7fd384106580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:27:28.995 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:28.994+0000 7fd389919700 1 --2- 172.21.15.160:0/2772998683 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3840f62c0 0x7fd384106580 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fd374013510 tx=0x7fd37400b040).stop 2023-11-17T05:27:28.995 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:28.994+0000 7fd389919700 1 -- 172.21.15.160:0/2772998683 shutdown_connections 2023-11-17T05:27:28.995 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:28.994+0000 7fd389919700 1 --2- 172.21.15.160:0/2772998683 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fd36c02ff60 0x7fd36c032420 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:27:28.995 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:28.994+0000 7fd389919700 1 --2- 172.21.15.160:0/2772998683 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3840f62c0 0x7fd384106580 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:27:28.995 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:28.994+0000 7fd389919700 1 -- 172.21.15.160:0/2772998683 >> 172.21.15.160:0/2772998683 conn(0x7fd3840f1a20 msgr2=0x7fd3840f2580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:27:28.995 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:28.994+0000 7fd389919700 1 -- 172.21.15.160:0/2772998683 shutdown_connections 2023-11-17T05:27:28.996 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:28.994+0000 7fd389919700 1 -- 172.21.15.160:0/2772998683 wait complete. 2023-11-17T05:27:28.996 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:27:30.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:27:29 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:27:28.993385+0000 mon.smithi003 (mon.0) 128 : audit [DBG] from='client.? 172.21.15.160:0/2772998683' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:27:30.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:27:29 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:27:29.407537+0000 mgr.smithi003.eddcuc (mgr.14164) 62 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:30.395 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:27:30.395 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:27:31.548 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:31.544+0000 7f159458c700 1 -- 172.21.15.160:0/3812013805 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f158c0f6510 msgr2=0x7f158c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:27:31.549 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:31.544+0000 7f159458c700 1 --2- 172.21.15.160:0/3812013805 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f158c0f6510 0x7f158c0f6930 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f157c009fb0 tx=0x7f157c009b30).stop 2023-11-17T05:27:31.549 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:31.544+0000 7f159458c700 1 -- 172.21.15.160:0/3812013805 shutdown_connections 2023-11-17T05:27:31.549 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:31.544+0000 7f159458c700 1 --2- 172.21.15.160:0/3812013805 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f158c0f6510 0x7f158c0f6930 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:27:31.549 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:31.544+0000 7f159458c700 1 -- 172.21.15.160:0/3812013805 >> 172.21.15.160:0/3812013805 conn(0x7f158c0f1c70 msgr2=0x7f158c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:27:31.549 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:31.544+0000 7f159458c700 1 -- 172.21.15.160:0/3812013805 shutdown_connections 2023-11-17T05:27:31.549 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:31.544+0000 7f159458c700 1 -- 172.21.15.160:0/3812013805 wait complete. 2023-11-17T05:27:31.549 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:31.545+0000 7f159458c700 1 Processor -- start 2023-11-17T05:27:31.549 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:31.545+0000 7f159458c700 1 -- start start 2023-11-17T05:27:31.549 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:31.545+0000 7f159458c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f158c0f6510 0x7f158c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:27:31.549 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:31.545+0000 7f159458c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f158c106c20 con 0x7f158c0f6510 2023-11-17T05:27:31.549 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:31.545+0000 7f1592328700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f158c0f6510 0x7f158c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:27:31.549 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:31.545+0000 7f1592328700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f158c0f6510 0x7f158c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:36626/0 (socket says 172.21.15.160:36626) 2023-11-17T05:27:31.550 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:31.545+0000 7f1592328700 1 -- 172.21.15.160:0/10436203 learned_addr learned my addr 172.21.15.160:0/10436203 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:27:31.550 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:31.546+0000 7f1592328700 1 -- 172.21.15.160:0/10436203 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f157c005040 con 0x7f158c0f6510 2023-11-17T05:27:31.550 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:31.546+0000 7f1592328700 1 --2- 172.21.15.160:0/10436203 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f158c0f6510 0x7f158c106730 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f157c014000 tx=0x7f157c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:27:31.550 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:31.546+0000 7f1582ffd700 1 -- 172.21.15.160:0/10436203 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f157c0095f0 con 0x7f158c0f6510 2023-11-17T05:27:31.550 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:31.546+0000 7f1582ffd700 1 -- 172.21.15.160:0/10436203 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f157c007e70 con 0x7f158c0f6510 2023-11-17T05:27:31.550 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:31.546+0000 7f159458c700 1 -- 172.21.15.160:0/10436203 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f158c106e20 con 0x7f158c0f6510 2023-11-17T05:27:31.550 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:31.546+0000 7f1582ffd700 1 -- 172.21.15.160:0/10436203 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f157c014ea0 con 0x7f158c0f6510 2023-11-17T05:27:31.550 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:31.546+0000 7f159458c700 1 -- 172.21.15.160:0/10436203 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f158c1072b0 con 0x7f158c0f6510 2023-11-17T05:27:31.550 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:31.547+0000 7f1582ffd700 1 -- 172.21.15.160:0/10436203 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f157c025070 con 0x7f158c0f6510 2023-11-17T05:27:31.550 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:31.547+0000 7f159458c700 1 -- 172.21.15.160:0/10436203 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f158c045ad0 con 0x7f158c0f6510 2023-11-17T05:27:31.550 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:31.547+0000 7f1582ffd700 1 --2- 172.21.15.160:0/10436203 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f157802ff60 0x7f1578032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:27:31.550 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:31.547+0000 7f1582ffd700 1 -- 172.21.15.160:0/10436203 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f157c040a30 con 0x7f158c0f6510 2023-11-17T05:27:31.550 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:31.547+0000 7f1591b27700 1 --2- 172.21.15.160:0/10436203 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f157802ff60 0x7f1578032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:27:31.551 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:31.547+0000 7f1591b27700 1 --2- 172.21.15.160:0/10436203 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f157802ff60 0x7f1578032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f15880096a0 tx=0x7f1588006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:27:31.552 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:31.552+0000 7f1582ffd700 1 -- 172.21.15.160:0/10436203 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f157c0079c0 con 0x7f158c0f6510 2023-11-17T05:27:31.759 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:31.758+0000 7f159458c700 1 -- 172.21.15.160:0/10436203 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f158c0fbf60 con 0x7f158c0f6510 2023-11-17T05:27:31.759 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:31.759+0000 7f1582ffd700 1 -- 172.21.15.160:0/10436203 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f157c00f300 con 0x7f158c0f6510 2023-11-17T05:27:31.760 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:27:31.760 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:27:31.762 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:31.760+0000 7f159458c700 1 -- 172.21.15.160:0/10436203 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f157802ff60 msgr2=0x7f1578032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:27:31.762 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:31.760+0000 7f159458c700 1 --2- 172.21.15.160:0/10436203 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f157802ff60 0x7f1578032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f15880096a0 tx=0x7f1588006b40).stop 2023-11-17T05:27:31.762 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:31.760+0000 7f159458c700 1 -- 172.21.15.160:0/10436203 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f158c0f6510 msgr2=0x7f158c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:27:31.762 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:31.760+0000 7f159458c700 1 --2- 172.21.15.160:0/10436203 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f158c0f6510 0x7f158c106730 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f157c014000 tx=0x7f157c00b040).stop 2023-11-17T05:27:31.762 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:31.760+0000 7f159458c700 1 -- 172.21.15.160:0/10436203 shutdown_connections 2023-11-17T05:27:31.762 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:31.760+0000 7f159458c700 1 --2- 172.21.15.160:0/10436203 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f157802ff60 0x7f1578032420 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:27:31.762 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:31.760+0000 7f159458c700 1 --2- 172.21.15.160:0/10436203 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f158c0f6510 0x7f158c106730 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:27:31.762 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:31.760+0000 7f159458c700 1 -- 172.21.15.160:0/10436203 >> 172.21.15.160:0/10436203 conn(0x7f158c0f1c70 msgr2=0x7f158c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:27:31.762 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:31.760+0000 7f159458c700 1 -- 172.21.15.160:0/10436203 shutdown_connections 2023-11-17T05:27:31.762 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:31.760+0000 7f159458c700 1 -- 172.21.15.160:0/10436203 wait complete. 2023-11-17T05:27:31.762 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:27:32.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:27:32 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:27:31.407811+0000 mgr.smithi003.eddcuc (mgr.14164) 63 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:32.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:27:32 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:27:31.759787+0000 mon.smithi003 (mon.0) 129 : audit [DBG] from='client.? 172.21.15.160:0/10436203' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:27:33.091 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:27:33.091 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:27:34.243 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:34.240+0000 7fb584d26700 1 -- 172.21.15.160:0/2030864943 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5800f4740 msgr2=0x7fb5800f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:27:34.243 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:34.240+0000 7fb584d26700 1 --2- 172.21.15.160:0/2030864943 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5800f4740 0x7fb5800f4b60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fb568009fb0 tx=0x7fb568009b30).stop 2023-11-17T05:27:34.243 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:34.240+0000 7fb584d26700 1 -- 172.21.15.160:0/2030864943 shutdown_connections 2023-11-17T05:27:34.243 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:34.240+0000 7fb584d26700 1 --2- 172.21.15.160:0/2030864943 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5800f4740 0x7fb5800f4b60 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:27:34.243 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:34.240+0000 7fb584d26700 1 -- 172.21.15.160:0/2030864943 >> 172.21.15.160:0/2030864943 conn(0x7fb5800f0260 msgr2=0x7fb5800f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:27:34.244 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:34.240+0000 7fb584d26700 1 -- 172.21.15.160:0/2030864943 shutdown_connections 2023-11-17T05:27:34.244 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:34.240+0000 7fb584d26700 1 -- 172.21.15.160:0/2030864943 wait complete. 2023-11-17T05:27:34.244 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:34.240+0000 7fb584d26700 1 Processor -- start 2023-11-17T05:27:34.244 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:34.240+0000 7fb584d26700 1 -- start start 2023-11-17T05:27:34.244 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:34.240+0000 7fb584d26700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5800f4740 0x7fb5801066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:27:34.244 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:34.240+0000 7fb584d26700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb580106bc0 con 0x7fb5800f4740 2023-11-17T05:27:34.244 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:34.241+0000 7fb57e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5800f4740 0x7fb5801066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:27:34.244 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:34.241+0000 7fb57e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5800f4740 0x7fb5801066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:36632/0 (socket says 172.21.15.160:36632) 2023-11-17T05:27:34.244 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:34.241+0000 7fb57e59c700 1 -- 172.21.15.160:0/2841758296 learned_addr learned my addr 172.21.15.160:0/2841758296 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:27:34.244 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:34.241+0000 7fb57e59c700 1 -- 172.21.15.160:0/2841758296 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb568005040 con 0x7fb5800f4740 2023-11-17T05:27:34.244 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:34.241+0000 7fb57e59c700 1 --2- 172.21.15.160:0/2841758296 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5800f4740 0x7fb5801066d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fb568013510 tx=0x7fb56800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:27:34.244 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:34.241+0000 7fb576ffd700 1 -- 172.21.15.160:0/2841758296 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb568009320 con 0x7fb5800f4740 2023-11-17T05:27:34.244 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:34.241+0000 7fb584d26700 1 -- 172.21.15.160:0/2841758296 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb580106dc0 con 0x7fb5800f4740 2023-11-17T05:27:34.245 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:34.241+0000 7fb576ffd700 1 -- 172.21.15.160:0/2841758296 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb568013f00 con 0x7fb5800f4740 2023-11-17T05:27:34.245 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:34.241+0000 7fb576ffd700 1 -- 172.21.15.160:0/2841758296 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb568014d90 con 0x7fb5800f4740 2023-11-17T05:27:34.245 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:34.241+0000 7fb584d26700 1 -- 172.21.15.160:0/2841758296 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb580107250 con 0x7fb5800f4740 2023-11-17T05:27:34.245 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:34.242+0000 7fb576ffd700 1 -- 172.21.15.160:0/2841758296 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb5680095f0 con 0x7fb5800f4740 2023-11-17T05:27:34.245 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:34.242+0000 7fb584d26700 1 -- 172.21.15.160:0/2841758296 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb580045ad0 con 0x7fb5800f4740 2023-11-17T05:27:34.245 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:34.242+0000 7fb576ffd700 1 --2- 172.21.15.160:0/2841758296 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fb56c02ff60 0x7fb56c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:27:34.245 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:34.242+0000 7fb576ffd700 1 -- 172.21.15.160:0/2841758296 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb5680410a0 con 0x7fb5800f4740 2023-11-17T05:27:34.245 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:34.242+0000 7fb57dd9b700 1 --2- 172.21.15.160:0/2841758296 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fb56c02ff60 0x7fb56c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:27:34.245 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:34.243+0000 7fb57dd9b700 1 --2- 172.21.15.160:0/2841758296 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fb56c02ff60 0x7fb56c032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fb5700096a0 tx=0x7fb570006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:27:34.248 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:34.247+0000 7fb576ffd700 1 -- 172.21.15.160:0/2841758296 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb56801d7f0 con 0x7fb5800f4740 2023-11-17T05:27:34.455 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:34.454+0000 7fb584d26700 1 -- 172.21.15.160:0/2841758296 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb580108100 con 0x7fb5800f4740 2023-11-17T05:27:34.456 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:34.455+0000 7fb576ffd700 1 -- 172.21.15.160:0/2841758296 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fb568010230 con 0x7fb5800f4740 2023-11-17T05:27:34.456 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:27:34.456 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:27:34.457 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:34.457+0000 7fb584d26700 1 -- 172.21.15.160:0/2841758296 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fb56c02ff60 msgr2=0x7fb56c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:27:34.458 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:34.457+0000 7fb584d26700 1 --2- 172.21.15.160:0/2841758296 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fb56c02ff60 0x7fb56c032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fb5700096a0 tx=0x7fb570006b40).stop 2023-11-17T05:27:34.458 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:34.457+0000 7fb584d26700 1 -- 172.21.15.160:0/2841758296 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5800f4740 msgr2=0x7fb5801066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:27:34.458 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:34.457+0000 7fb584d26700 1 --2- 172.21.15.160:0/2841758296 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5800f4740 0x7fb5801066d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fb568013510 tx=0x7fb56800b040).stop 2023-11-17T05:27:34.458 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:34.457+0000 7fb584d26700 1 -- 172.21.15.160:0/2841758296 shutdown_connections 2023-11-17T05:27:34.458 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:34.457+0000 7fb584d26700 1 --2- 172.21.15.160:0/2841758296 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fb56c02ff60 0x7fb56c032420 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:27:34.458 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:34.457+0000 7fb584d26700 1 --2- 172.21.15.160:0/2841758296 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5800f4740 0x7fb5801066d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:27:34.458 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:34.457+0000 7fb584d26700 1 -- 172.21.15.160:0/2841758296 >> 172.21.15.160:0/2841758296 conn(0x7fb5800f0260 msgr2=0x7fb5800f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:27:34.458 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:34.457+0000 7fb584d26700 1 -- 172.21.15.160:0/2841758296 shutdown_connections 2023-11-17T05:27:34.458 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:34.457+0000 7fb584d26700 1 -- 172.21.15.160:0/2841758296 wait complete. 2023-11-17T05:27:34.459 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:27:34.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:27:34 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:27:33.408114+0000 mgr.smithi003.eddcuc (mgr.14164) 64 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:34.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:27:34 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:27:34.456201+0000 mon.smithi003 (mon.0) 130 : audit [DBG] from='client.? 172.21.15.160:0/2841758296' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:27:35.854 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:27:35.855 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:27:36.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:27:36 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:27:35.408414+0000 mgr.smithi003.eddcuc (mgr.14164) 65 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:37.093 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:37.090+0000 7f5acedd6700 1 -- 172.21.15.160:0/987428461 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ac80f62a0 msgr2=0x7f5ac80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:27:37.094 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:37.090+0000 7f5acedd6700 1 --2- 172.21.15.160:0/987428461 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ac80f62a0 0x7f5ac80f66c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f5ac4009fb0 tx=0x7f5ac4009b30).stop 2023-11-17T05:27:37.094 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:37.090+0000 7f5acedd6700 1 -- 172.21.15.160:0/987428461 shutdown_connections 2023-11-17T05:27:37.094 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:37.090+0000 7f5acedd6700 1 --2- 172.21.15.160:0/987428461 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ac80f62a0 0x7f5ac80f66c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:27:37.094 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:37.090+0000 7f5acedd6700 1 -- 172.21.15.160:0/987428461 >> 172.21.15.160:0/987428461 conn(0x7f5ac80f1aa0 msgr2=0x7f5ac80f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:27:37.095 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:37.090+0000 7f5acedd6700 1 -- 172.21.15.160:0/987428461 shutdown_connections 2023-11-17T05:27:37.095 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:37.090+0000 7f5acedd6700 1 -- 172.21.15.160:0/987428461 wait complete. 2023-11-17T05:27:37.095 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:37.090+0000 7f5acedd6700 1 Processor -- start 2023-11-17T05:27:37.095 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:37.090+0000 7f5acedd6700 1 -- start start 2023-11-17T05:27:37.095 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:37.091+0000 7f5acedd6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ac80f62a0 0x7f5ac81064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:27:37.095 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:37.091+0000 7f5acedd6700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac81069a0 con 0x7f5ac80f62a0 2023-11-17T05:27:37.095 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:37.091+0000 7f5acddd4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ac80f62a0 0x7f5ac81064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:27:37.095 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:37.091+0000 7f5acddd4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ac80f62a0 0x7f5ac81064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:36640/0 (socket says 172.21.15.160:36640) 2023-11-17T05:27:37.096 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:37.091+0000 7f5acddd4700 1 -- 172.21.15.160:0/4047617709 learned_addr learned my addr 172.21.15.160:0/4047617709 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:27:37.096 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:37.091+0000 7f5acddd4700 1 -- 172.21.15.160:0/4047617709 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac4005040 con 0x7f5ac80f62a0 2023-11-17T05:27:37.096 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:37.091+0000 7f5acddd4700 1 --2- 172.21.15.160:0/4047617709 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ac80f62a0 0x7f5ac81064b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f5ac4013510 tx=0x7f5ac400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:27:37.096 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:37.092+0000 7f5abe7fc700 1 -- 172.21.15.160:0/4047617709 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ac4009320 con 0x7f5ac80f62a0 2023-11-17T05:27:37.096 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:37.092+0000 7f5acedd6700 1 -- 172.21.15.160:0/4047617709 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ac8106ba0 con 0x7f5ac80f62a0 2023-11-17T05:27:37.096 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:37.092+0000 7f5abe7fc700 1 -- 172.21.15.160:0/4047617709 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5ac4013f00 con 0x7f5ac80f62a0 2023-11-17T05:27:37.096 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:37.092+0000 7f5abe7fc700 1 -- 172.21.15.160:0/4047617709 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ac4014dc0 con 0x7f5ac80f62a0 2023-11-17T05:27:37.096 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:37.092+0000 7f5acedd6700 1 -- 172.21.15.160:0/4047617709 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ac8107030 con 0x7f5ac80f62a0 2023-11-17T05:27:37.097 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:37.092+0000 7f5abe7fc700 1 -- 172.21.15.160:0/4047617709 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5ac40095f0 con 0x7f5ac80f62a0 2023-11-17T05:27:37.097 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:37.092+0000 7f5acedd6700 1 -- 172.21.15.160:0/4047617709 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ac8045ad0 con 0x7f5ac80f62a0 2023-11-17T05:27:37.097 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:37.093+0000 7f5abe7fc700 1 --2- 172.21.15.160:0/4047617709 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f5ab402ff60 0x7f5ab4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:27:37.097 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:37.093+0000 7f5abe7fc700 1 -- 172.21.15.160:0/4047617709 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5ac4041100 con 0x7f5ac80f62a0 2023-11-17T05:27:37.097 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:37.093+0000 7f5acd5d3700 1 --2- 172.21.15.160:0/4047617709 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f5ab402ff60 0x7f5ab4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:27:37.098 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:37.093+0000 7f5acd5d3700 1 --2- 172.21.15.160:0/4047617709 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f5ab402ff60 0x7f5ab4032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f5ab80096a0 tx=0x7f5ab8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:27:37.098 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:37.097+0000 7f5abe7fc700 1 -- 172.21.15.160:0/4047617709 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5ac4014420 con 0x7f5ac80f62a0 2023-11-17T05:27:37.308 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:37.307+0000 7f5acedd6700 1 -- 172.21.15.160:0/4047617709 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5ac8107ee0 con 0x7f5ac80f62a0 2023-11-17T05:27:37.309 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:37.309+0000 7f5abe7fc700 1 -- 172.21.15.160:0/4047617709 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f5ac4010230 con 0x7f5ac80f62a0 2023-11-17T05:27:37.309 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:27:37.309 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:27:37.311 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:37.309+0000 7f5acedd6700 1 -- 172.21.15.160:0/4047617709 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f5ab402ff60 msgr2=0x7f5ab4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:27:37.311 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:37.309+0000 7f5acedd6700 1 --2- 172.21.15.160:0/4047617709 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f5ab402ff60 0x7f5ab4032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f5ab80096a0 tx=0x7f5ab8006b40).stop 2023-11-17T05:27:37.311 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:37.310+0000 7f5acedd6700 1 -- 172.21.15.160:0/4047617709 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ac80f62a0 msgr2=0x7f5ac81064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:27:37.311 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:37.310+0000 7f5acedd6700 1 --2- 172.21.15.160:0/4047617709 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ac80f62a0 0x7f5ac81064b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f5ac4013510 tx=0x7f5ac400b040).stop 2023-11-17T05:27:37.311 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:37.310+0000 7f5acedd6700 1 -- 172.21.15.160:0/4047617709 shutdown_connections 2023-11-17T05:27:37.311 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:37.310+0000 7f5acedd6700 1 --2- 172.21.15.160:0/4047617709 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f5ab402ff60 0x7f5ab4032420 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:27:37.311 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:37.310+0000 7f5acedd6700 1 --2- 172.21.15.160:0/4047617709 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ac80f62a0 0x7f5ac81064b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:27:37.311 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:37.310+0000 7f5acedd6700 1 -- 172.21.15.160:0/4047617709 >> 172.21.15.160:0/4047617709 conn(0x7f5ac80f1aa0 msgr2=0x7f5ac80f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:27:37.312 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:37.310+0000 7f5acedd6700 1 -- 172.21.15.160:0/4047617709 shutdown_connections 2023-11-17T05:27:37.312 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:37.310+0000 7f5acedd6700 1 -- 172.21.15.160:0/4047617709 wait complete. 2023-11-17T05:27:37.312 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:27:37.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:27:37 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:27:37.309157+0000 mon.smithi003 (mon.0) 131 : audit [DBG] from='client.? 172.21.15.160:0/4047617709' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:27:38.713 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:27:38.714 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:27:38.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:27:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:27:37.408804+0000 mgr.smithi003.eddcuc (mgr.14164) 66 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:39.854 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:39.853+0000 7f1bc94b4700 1 -- 172.21.15.160:0/2760474314 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1bc40f51a0 msgr2=0x7f1bc40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:27:39.854 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:39.853+0000 7f1bc94b4700 1 --2- 172.21.15.160:0/2760474314 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1bc40f51a0 0x7f1bc40f55c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f1bb8009fb0 tx=0x7f1bb8009b30).stop 2023-11-17T05:27:39.854 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:39.853+0000 7f1bc94b4700 1 -- 172.21.15.160:0/2760474314 shutdown_connections 2023-11-17T05:27:39.854 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:39.853+0000 7f1bc94b4700 1 --2- 172.21.15.160:0/2760474314 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1bc40f51a0 0x7f1bc40f55c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:27:39.854 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:39.853+0000 7f1bc94b4700 1 -- 172.21.15.160:0/2760474314 >> 172.21.15.160:0/2760474314 conn(0x7f1bc40f09a0 msgr2=0x7f1bc40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:27:39.855 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:39.853+0000 7f1bc94b4700 1 -- 172.21.15.160:0/2760474314 shutdown_connections 2023-11-17T05:27:39.855 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:39.853+0000 7f1bc94b4700 1 -- 172.21.15.160:0/2760474314 wait complete. 2023-11-17T05:27:39.855 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:39.853+0000 7f1bc94b4700 1 Processor -- start 2023-11-17T05:27:39.855 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:39.853+0000 7f1bc94b4700 1 -- start start 2023-11-17T05:27:39.856 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:39.854+0000 7f1bc94b4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1bc40f51a0 0x7f1bc41053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:27:39.856 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:39.854+0000 7f1bc94b4700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc41058a0 con 0x7f1bc40f51a0 2023-11-17T05:27:39.856 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:39.854+0000 7f1bc3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1bc40f51a0 0x7f1bc41053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:27:39.856 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:39.854+0000 7f1bc3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1bc40f51a0 0x7f1bc41053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:36670/0 (socket says 172.21.15.160:36670) 2023-11-17T05:27:39.857 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:39.854+0000 7f1bc3fff700 1 -- 172.21.15.160:0/2377271901 learned_addr learned my addr 172.21.15.160:0/2377271901 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:27:39.857 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:39.854+0000 7f1bc3fff700 1 -- 172.21.15.160:0/2377271901 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bb8005040 con 0x7f1bc40f51a0 2023-11-17T05:27:39.857 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:39.854+0000 7f1bc3fff700 1 --2- 172.21.15.160:0/2377271901 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1bc40f51a0 0x7f1bc41053b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f1bb8009830 tx=0x7f1bb800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:27:39.857 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:39.854+0000 7f1bc0ff9700 1 -- 172.21.15.160:0/2377271901 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1bb8009320 con 0x7f1bc40f51a0 2023-11-17T05:27:39.857 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:39.854+0000 7f1bc94b4700 1 -- 172.21.15.160:0/2377271901 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bc4105aa0 con 0x7f1bc40f51a0 2023-11-17T05:27:39.857 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:39.854+0000 7f1bc0ff9700 1 -- 172.21.15.160:0/2377271901 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1bb80136c0 con 0x7f1bc40f51a0 2023-11-17T05:27:39.857 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:39.854+0000 7f1bc0ff9700 1 -- 172.21.15.160:0/2377271901 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1bb801ee90 con 0x7f1bc40f51a0 2023-11-17T05:27:39.857 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:39.855+0000 7f1bc94b4700 1 -- 172.21.15.160:0/2377271901 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bc4105f30 con 0x7f1bc40f51a0 2023-11-17T05:27:39.857 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:39.855+0000 7f1bc0ff9700 1 -- 172.21.15.160:0/2377271901 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1bb8014440 con 0x7f1bc40f51a0 2023-11-17T05:27:39.857 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:39.855+0000 7f1bc0ff9700 1 --2- 172.21.15.160:0/2377271901 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f1bac02ff10 0x7f1bac0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:27:39.857 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:39.855+0000 7f1bc0ff9700 1 -- 172.21.15.160:0/2377271901 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1bb8040ef0 con 0x7f1bc40f51a0 2023-11-17T05:27:39.858 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:39.855+0000 7f1bc94b4700 1 -- 172.21.15.160:0/2377271901 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bc4059100 con 0x7f1bc40f51a0 2023-11-17T05:27:39.858 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:39.856+0000 7f1bc37fe700 1 --2- 172.21.15.160:0/2377271901 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f1bac02ff10 0x7f1bac0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:27:39.858 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:39.856+0000 7f1bc37fe700 1 --2- 172.21.15.160:0/2377271901 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f1bac02ff10 0x7f1bac0323d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f1bb40096a0 tx=0x7f1bb4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:27:39.861 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:39.860+0000 7f1bc0ff9700 1 -- 172.21.15.160:0/2377271901 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1bb80273f0 con 0x7f1bc40f51a0 2023-11-17T05:27:40.067 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:40.066+0000 7f1bc94b4700 1 -- 172.21.15.160:0/2377271901 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1bc40fabe0 con 0x7f1bc40f51a0 2023-11-17T05:27:40.068 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:40.067+0000 7f1bc0ff9700 1 -- 172.21.15.160:0/2377271901 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f1bb801c320 con 0x7f1bc40f51a0 2023-11-17T05:27:40.068 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:27:40.068 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:27:40.070 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:40.069+0000 7f1bc94b4700 1 -- 172.21.15.160:0/2377271901 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f1bac02ff10 msgr2=0x7f1bac0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:27:40.070 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:40.069+0000 7f1bc94b4700 1 --2- 172.21.15.160:0/2377271901 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f1bac02ff10 0x7f1bac0323d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f1bb40096a0 tx=0x7f1bb4006b40).stop 2023-11-17T05:27:40.070 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:40.069+0000 7f1bc94b4700 1 -- 172.21.15.160:0/2377271901 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1bc40f51a0 msgr2=0x7f1bc41053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:27:40.070 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:40.069+0000 7f1bc94b4700 1 --2- 172.21.15.160:0/2377271901 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1bc40f51a0 0x7f1bc41053b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f1bb8009830 tx=0x7f1bb800b040).stop 2023-11-17T05:27:40.070 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:40.069+0000 7f1bc94b4700 1 -- 172.21.15.160:0/2377271901 shutdown_connections 2023-11-17T05:27:40.070 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:40.069+0000 7f1bc94b4700 1 --2- 172.21.15.160:0/2377271901 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f1bac02ff10 0x7f1bac0323d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:27:40.070 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:40.069+0000 7f1bc94b4700 1 --2- 172.21.15.160:0/2377271901 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1bc40f51a0 0x7f1bc41053b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:27:40.070 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:40.069+0000 7f1bc94b4700 1 -- 172.21.15.160:0/2377271901 >> 172.21.15.160:0/2377271901 conn(0x7f1bc40f09a0 msgr2=0x7f1bc40f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:27:40.071 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:40.069+0000 7f1bc94b4700 1 -- 172.21.15.160:0/2377271901 shutdown_connections 2023-11-17T05:27:40.071 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:40.069+0000 7f1bc94b4700 1 -- 172.21.15.160:0/2377271901 wait complete. 2023-11-17T05:27:40.071 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:27:40.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:27:39 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:27:39.409188+0000 mgr.smithi003.eddcuc (mgr.14164) 67 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:41.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:27:40 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:27:40.068199+0000 mon.smithi003 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.160:0/2377271901' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:27:41.446 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:27:41.446 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:27:42.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:27:41 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:27:41.409522+0000 mgr.smithi003.eddcuc (mgr.14164) 68 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:42.610 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:42.609+0000 7fcf290e9700 1 -- 172.21.15.160:0/439315230 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf240f6510 msgr2=0x7fcf240f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:27:42.611 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:42.609+0000 7fcf290e9700 1 --2- 172.21.15.160:0/439315230 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf240f6510 0x7fcf240f6930 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fcf18009fb0 tx=0x7fcf18009b30).stop 2023-11-17T05:27:42.611 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:42.609+0000 7fcf290e9700 1 -- 172.21.15.160:0/439315230 shutdown_connections 2023-11-17T05:27:42.611 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:42.609+0000 7fcf290e9700 1 --2- 172.21.15.160:0/439315230 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf240f6510 0x7fcf240f6930 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:27:42.611 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:42.609+0000 7fcf290e9700 1 -- 172.21.15.160:0/439315230 >> 172.21.15.160:0/439315230 conn(0x7fcf240f1c70 msgr2=0x7fcf240f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:27:42.611 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:42.609+0000 7fcf290e9700 1 -- 172.21.15.160:0/439315230 shutdown_connections 2023-11-17T05:27:42.611 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:42.609+0000 7fcf290e9700 1 -- 172.21.15.160:0/439315230 wait complete. 2023-11-17T05:27:42.611 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:42.610+0000 7fcf290e9700 1 Processor -- start 2023-11-17T05:27:42.611 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:42.610+0000 7fcf290e9700 1 -- start start 2023-11-17T05:27:42.611 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:42.610+0000 7fcf290e9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf240f6510 0x7fcf24106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:27:42.611 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:42.610+0000 7fcf290e9700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf24106c20 con 0x7fcf240f6510 2023-11-17T05:27:42.613 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:42.610+0000 7fcf22d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf240f6510 0x7fcf24106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:27:42.613 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:42.610+0000 7fcf22d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf240f6510 0x7fcf24106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:49428/0 (socket says 172.21.15.160:49428) 2023-11-17T05:27:42.613 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:42.610+0000 7fcf22d9d700 1 -- 172.21.15.160:0/2429466540 learned_addr learned my addr 172.21.15.160:0/2429466540 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:27:42.613 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:42.610+0000 7fcf22d9d700 1 -- 172.21.15.160:0/2429466540 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf18005040 con 0x7fcf240f6510 2023-11-17T05:27:42.613 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:42.611+0000 7fcf22d9d700 1 --2- 172.21.15.160:0/2429466540 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf240f6510 0x7fcf24106730 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fcf18013510 tx=0x7fcf1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:27:42.613 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:42.611+0000 7fcf0b7fe700 1 -- 172.21.15.160:0/2429466540 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf18009320 con 0x7fcf240f6510 2023-11-17T05:27:42.613 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:42.611+0000 7fcf290e9700 1 -- 172.21.15.160:0/2429466540 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf24106e20 con 0x7fcf240f6510 2023-11-17T05:27:42.613 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:42.611+0000 7fcf0b7fe700 1 -- 172.21.15.160:0/2429466540 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcf18013f00 con 0x7fcf240f6510 2023-11-17T05:27:42.614 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:42.611+0000 7fcf0b7fe700 1 -- 172.21.15.160:0/2429466540 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf18014d90 con 0x7fcf240f6510 2023-11-17T05:27:42.614 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:42.611+0000 7fcf290e9700 1 -- 172.21.15.160:0/2429466540 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf241072b0 con 0x7fcf240f6510 2023-11-17T05:27:42.614 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:42.611+0000 7fcf0b7fe700 1 -- 172.21.15.160:0/2429466540 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcf18014420 con 0x7fcf240f6510 2023-11-17T05:27:42.614 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:42.612+0000 7fcf290e9700 1 -- 172.21.15.160:0/2429466540 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf24045ad0 con 0x7fcf240f6510 2023-11-17T05:27:42.614 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:42.612+0000 7fcf0b7fe700 1 --2- 172.21.15.160:0/2429466540 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fcf0c02ff60 0x7fcf0c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:27:42.614 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:42.612+0000 7fcf0b7fe700 1 -- 172.21.15.160:0/2429466540 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcf18041080 con 0x7fcf240f6510 2023-11-17T05:27:42.614 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:42.612+0000 7fcf2259c700 1 --2- 172.21.15.160:0/2429466540 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fcf0c02ff60 0x7fcf0c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:27:42.614 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:42.612+0000 7fcf2259c700 1 --2- 172.21.15.160:0/2429466540 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fcf0c02ff60 0x7fcf0c032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fcf140096a0 tx=0x7fcf14006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:27:42.622 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:42.617+0000 7fcf0b7fe700 1 -- 172.21.15.160:0/2429466540 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcf180095f0 con 0x7fcf240f6510 2023-11-17T05:27:42.827 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:42.827+0000 7fcf290e9700 1 -- 172.21.15.160:0/2429466540 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcf240fbf60 con 0x7fcf240f6510 2023-11-17T05:27:42.828 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:42.827+0000 7fcf0b7fe700 1 -- 172.21.15.160:0/2429466540 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fcf1801db70 con 0x7fcf240f6510 2023-11-17T05:27:42.828 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:27:42.828 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:27:42.830 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:42.829+0000 7fcf290e9700 1 -- 172.21.15.160:0/2429466540 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fcf0c02ff60 msgr2=0x7fcf0c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:27:42.830 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:42.829+0000 7fcf290e9700 1 --2- 172.21.15.160:0/2429466540 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fcf0c02ff60 0x7fcf0c032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fcf140096a0 tx=0x7fcf14006b40).stop 2023-11-17T05:27:42.830 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:42.829+0000 7fcf290e9700 1 -- 172.21.15.160:0/2429466540 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf240f6510 msgr2=0x7fcf24106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:27:42.830 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:42.829+0000 7fcf290e9700 1 --2- 172.21.15.160:0/2429466540 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf240f6510 0x7fcf24106730 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fcf18013510 tx=0x7fcf1800b040).stop 2023-11-17T05:27:42.830 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:42.829+0000 7fcf290e9700 1 -- 172.21.15.160:0/2429466540 shutdown_connections 2023-11-17T05:27:42.830 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:42.829+0000 7fcf290e9700 1 --2- 172.21.15.160:0/2429466540 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fcf0c02ff60 0x7fcf0c032420 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:27:42.830 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:42.829+0000 7fcf290e9700 1 --2- 172.21.15.160:0/2429466540 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf240f6510 0x7fcf24106730 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:27:42.830 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:42.829+0000 7fcf290e9700 1 -- 172.21.15.160:0/2429466540 >> 172.21.15.160:0/2429466540 conn(0x7fcf240f1c70 msgr2=0x7fcf240f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:27:42.831 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:42.829+0000 7fcf290e9700 1 -- 172.21.15.160:0/2429466540 shutdown_connections 2023-11-17T05:27:42.831 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:42.829+0000 7fcf290e9700 1 -- 172.21.15.160:0/2429466540 wait complete. 2023-11-17T05:27:42.831 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:27:43.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:27:42 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:27:42.828609+0000 mon.smithi003 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.160:0/2429466540' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:27:44.248 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:27:44.249 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:27:44.272 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:27:43 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:27:43.409899+0000 mgr.smithi003.eddcuc (mgr.14164) 69 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:45.502 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:45.500+0000 7f18aed9b700 1 -- 172.21.15.160:0/1410220840 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f18a80601e0 msgr2=0x7f18a80f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:27:45.502 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:45.500+0000 7f18aed9b700 1 --2- 172.21.15.160:0/1410220840 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f18a80601e0 0x7f18a80f9190 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f189c009fb0 tx=0x7f189c009b30).stop 2023-11-17T05:27:45.502 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:45.500+0000 7f18aed9b700 1 -- 172.21.15.160:0/1410220840 shutdown_connections 2023-11-17T05:27:45.502 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:45.500+0000 7f18aed9b700 1 --2- 172.21.15.160:0/1410220840 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f18a80601e0 0x7f18a80f9190 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:27:45.502 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:45.500+0000 7f18aed9b700 1 -- 172.21.15.160:0/1410220840 >> 172.21.15.160:0/1410220840 conn(0x7f18a80f0b70 msgr2=0x7f18a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:27:45.502 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:45.500+0000 7f18aed9b700 1 -- 172.21.15.160:0/1410220840 shutdown_connections 2023-11-17T05:27:45.502 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:45.500+0000 7f18aed9b700 1 -- 172.21.15.160:0/1410220840 wait complete. 2023-11-17T05:27:45.502 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:45.501+0000 7f18aed9b700 1 Processor -- start 2023-11-17T05:27:45.502 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:45.501+0000 7f18aed9b700 1 -- start start 2023-11-17T05:27:45.502 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:45.501+0000 7f18aed9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f18a80601e0 0x7f18a80f6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:27:45.503 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:45.501+0000 7f18aed9b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a80f5090 con 0x7f18a80601e0 2023-11-17T05:27:45.504 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:45.501+0000 7f18acb37700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f18a80601e0 0x7f18a80f6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:27:45.504 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:45.501+0000 7f18acb37700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f18a80601e0 0x7f18a80f6c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:49450/0 (socket says 172.21.15.160:49450) 2023-11-17T05:27:45.504 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:45.501+0000 7f18acb37700 1 -- 172.21.15.160:0/1008387438 learned_addr learned my addr 172.21.15.160:0/1008387438 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:27:45.504 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:45.501+0000 7f18acb37700 1 -- 172.21.15.160:0/1008387438 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f189c005040 con 0x7f18a80601e0 2023-11-17T05:27:45.504 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:45.502+0000 7f18acb37700 1 --2- 172.21.15.160:0/1008387438 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f18a80601e0 0x7f18a80f6c10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f189c004dd0 tx=0x7f189c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:27:45.504 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:45.502+0000 7f18a57fa700 1 -- 172.21.15.160:0/1008387438 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f189c018070 con 0x7f18a80601e0 2023-11-17T05:27:45.504 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:45.502+0000 7f18aed9b700 1 -- 172.21.15.160:0/1008387438 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18a80f5230 con 0x7f18a80601e0 2023-11-17T05:27:45.505 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:45.502+0000 7f18a57fa700 1 -- 172.21.15.160:0/1008387438 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f189c013410 con 0x7f18a80601e0 2023-11-17T05:27:45.505 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:45.502+0000 7f18a57fa700 1 -- 172.21.15.160:0/1008387438 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f189c01c3c0 con 0x7f18a80601e0 2023-11-17T05:27:45.505 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:45.502+0000 7f18aed9b700 1 -- 172.21.15.160:0/1008387438 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18a80f56c0 con 0x7f18a80601e0 2023-11-17T05:27:45.505 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:45.503+0000 7f18a57fa700 1 -- 172.21.15.160:0/1008387438 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f189c01d420 con 0x7f18a80601e0 2023-11-17T05:27:45.505 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:45.503+0000 7f18aed9b700 1 -- 172.21.15.160:0/1008387438 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18a8059100 con 0x7f18a80601e0 2023-11-17T05:27:45.505 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:45.503+0000 7f18a57fa700 1 --2- 172.21.15.160:0/1008387438 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f189802ff60 0x7f1898032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:27:45.505 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:45.503+0000 7f18a57fa700 1 -- 172.21.15.160:0/1008387438 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f189c041510 con 0x7f18a80601e0 2023-11-17T05:27:45.505 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:45.503+0000 7f18a7fff700 1 --2- 172.21.15.160:0/1008387438 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f189802ff60 0x7f1898032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:27:45.505 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:45.503+0000 7f18a7fff700 1 --2- 172.21.15.160:0/1008387438 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f189802ff60 0x7f1898032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f18940096a0 tx=0x7f1894006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:27:45.511 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:45.508+0000 7f18a57fa700 1 -- 172.21.15.160:0/1008387438 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f189c01d820 con 0x7f18a80601e0 2023-11-17T05:27:45.711 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:45.711+0000 7f18aed9b700 1 -- 172.21.15.160:0/1008387438 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f18a80fbf60 con 0x7f18a80601e0 2023-11-17T05:27:45.712 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:45.711+0000 7f18a57fa700 1 -- 172.21.15.160:0/1008387438 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f189c01b370 con 0x7f18a80601e0 2023-11-17T05:27:45.712 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:27:45.713 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:27:45.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:45.713+0000 7f18aed9b700 1 -- 172.21.15.160:0/1008387438 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f189802ff60 msgr2=0x7f1898032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:27:45.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:45.713+0000 7f18aed9b700 1 --2- 172.21.15.160:0/1008387438 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f189802ff60 0x7f1898032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f18940096a0 tx=0x7f1894006b40).stop 2023-11-17T05:27:45.715 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:45.713+0000 7f18aed9b700 1 -- 172.21.15.160:0/1008387438 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f18a80601e0 msgr2=0x7f18a80f6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:27:45.715 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:45.713+0000 7f18aed9b700 1 --2- 172.21.15.160:0/1008387438 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f18a80601e0 0x7f18a80f6c10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f189c004dd0 tx=0x7f189c009b30).stop 2023-11-17T05:27:45.715 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:45.713+0000 7f18aed9b700 1 -- 172.21.15.160:0/1008387438 shutdown_connections 2023-11-17T05:27:45.715 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:45.713+0000 7f18aed9b700 1 --2- 172.21.15.160:0/1008387438 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f189802ff60 0x7f1898032420 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:27:45.715 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:45.713+0000 7f18aed9b700 1 --2- 172.21.15.160:0/1008387438 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f18a80601e0 0x7f18a80f6c10 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:27:45.715 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:45.713+0000 7f18aed9b700 1 -- 172.21.15.160:0/1008387438 >> 172.21.15.160:0/1008387438 conn(0x7f18a80f0b70 msgr2=0x7f18a80f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:27:45.715 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:45.713+0000 7f18aed9b700 1 -- 172.21.15.160:0/1008387438 shutdown_connections 2023-11-17T05:27:45.715 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:45.713+0000 7f18aed9b700 1 -- 172.21.15.160:0/1008387438 wait complete. 2023-11-17T05:27:45.715 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:27:46.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:27:46 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:27:45.410345+0000 mgr.smithi003.eddcuc (mgr.14164) 70 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:46.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:27:46 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:27:45.712537+0000 mon.smithi003 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.160:0/1008387438' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:27:47.092 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:27:47.092 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:27:48.247 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:48.246+0000 7f9d3be8e700 1 -- 172.21.15.160:0/2491864455 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d340f5410 msgr2=0x7f9d340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:27:48.247 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:48.246+0000 7f9d3be8e700 1 --2- 172.21.15.160:0/2491864455 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d340f5410 0x7f9d340f5830 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f9d24009fb0 tx=0x7f9d24009b30).stop 2023-11-17T05:27:48.247 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:48.246+0000 7f9d3be8e700 1 -- 172.21.15.160:0/2491864455 shutdown_connections 2023-11-17T05:27:48.248 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:48.246+0000 7f9d3be8e700 1 --2- 172.21.15.160:0/2491864455 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d340f5410 0x7f9d340f5830 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:27:48.248 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:48.246+0000 7f9d3be8e700 1 -- 172.21.15.160:0/2491864455 >> 172.21.15.160:0/2491864455 conn(0x7f9d340f0b70 msgr2=0x7f9d340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:27:48.248 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:48.246+0000 7f9d3be8e700 1 -- 172.21.15.160:0/2491864455 shutdown_connections 2023-11-17T05:27:48.248 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:48.246+0000 7f9d3be8e700 1 -- 172.21.15.160:0/2491864455 wait complete. 2023-11-17T05:27:48.248 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:48.246+0000 7f9d3be8e700 1 Processor -- start 2023-11-17T05:27:48.248 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:48.246+0000 7f9d3be8e700 1 -- start start 2023-11-17T05:27:48.248 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:48.246+0000 7f9d3be8e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d340f5410 0x7f9d340fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:27:48.248 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:48.246+0000 7f9d3be8e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d340fb820 con 0x7f9d340f5410 2023-11-17T05:27:48.250 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:48.247+0000 7f9d39c2a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d340f5410 0x7f9d340fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:27:48.250 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:48.247+0000 7f9d39c2a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d340f5410 0x7f9d340fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:49468/0 (socket says 172.21.15.160:49468) 2023-11-17T05:27:48.250 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:48.247+0000 7f9d39c2a700 1 -- 172.21.15.160:0/2929607050 learned_addr learned my addr 172.21.15.160:0/2929607050 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:27:48.250 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:48.247+0000 7f9d39c2a700 1 -- 172.21.15.160:0/2929607050 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d24005040 con 0x7f9d340f5410 2023-11-17T05:27:48.250 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:48.247+0000 7f9d39c2a700 1 --2- 172.21.15.160:0/2929607050 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d340f5410 0x7f9d340fd3a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f9d24013510 tx=0x7f9d2400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:27:48.250 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:48.247+0000 7f9d2a7fc700 1 -- 172.21.15.160:0/2929607050 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d24009320 con 0x7f9d340f5410 2023-11-17T05:27:48.250 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:48.247+0000 7f9d3be8e700 1 -- 172.21.15.160:0/2929607050 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d340fb9c0 con 0x7f9d340f5410 2023-11-17T05:27:48.250 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:48.247+0000 7f9d2a7fc700 1 -- 172.21.15.160:0/2929607050 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9d24013f00 con 0x7f9d340f5410 2023-11-17T05:27:48.250 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:48.247+0000 7f9d2a7fc700 1 -- 172.21.15.160:0/2929607050 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d24014d90 con 0x7f9d340f5410 2023-11-17T05:27:48.250 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:48.247+0000 7f9d3be8e700 1 -- 172.21.15.160:0/2929607050 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d340fbe50 con 0x7f9d340f5410 2023-11-17T05:27:48.250 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:48.248+0000 7f9d2a7fc700 1 -- 172.21.15.160:0/2929607050 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9d240095f0 con 0x7f9d340f5410 2023-11-17T05:27:48.250 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:48.248+0000 7f9d3be8e700 1 -- 172.21.15.160:0/2929607050 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d34059100 con 0x7f9d340f5410 2023-11-17T05:27:48.251 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:48.248+0000 7f9d2a7fc700 1 --2- 172.21.15.160:0/2929607050 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f9d2002ffb0 0x7f9d20032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:27:48.251 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:48.248+0000 7f9d2a7fc700 1 -- 172.21.15.160:0/2929607050 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9d24041050 con 0x7f9d340f5410 2023-11-17T05:27:48.251 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:48.249+0000 7f9d39429700 1 --2- 172.21.15.160:0/2929607050 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f9d2002ffb0 0x7f9d20032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:27:48.251 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:48.249+0000 7f9d39429700 1 --2- 172.21.15.160:0/2929607050 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f9d2002ffb0 0x7f9d20032470 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f9d300096a0 tx=0x7f9d30006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:27:48.254 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:48.253+0000 7f9d2a7fc700 1 -- 172.21.15.160:0/2929607050 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9d2401d7f0 con 0x7f9d340f5410 2023-11-17T05:27:48.468 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:48.459+0000 7f9d3be8e700 1 -- 172.21.15.160:0/2929607050 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9d340fae60 con 0x7f9d340f5410 2023-11-17T05:27:48.468 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:48.460+0000 7f9d2a7fc700 1 -- 172.21.15.160:0/2929607050 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f9d24010230 con 0x7f9d340f5410 2023-11-17T05:27:48.470 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:27:48.470 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:27:48.471 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:48.461+0000 7f9d3be8e700 1 -- 172.21.15.160:0/2929607050 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f9d2002ffb0 msgr2=0x7f9d20032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:27:48.471 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:48.461+0000 7f9d3be8e700 1 --2- 172.21.15.160:0/2929607050 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f9d2002ffb0 0x7f9d20032470 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f9d300096a0 tx=0x7f9d30006b40).stop 2023-11-17T05:27:48.471 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:48.461+0000 7f9d3be8e700 1 -- 172.21.15.160:0/2929607050 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d340f5410 msgr2=0x7f9d340fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:27:48.479 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:48.461+0000 7f9d3be8e700 1 --2- 172.21.15.160:0/2929607050 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d340f5410 0x7f9d340fd3a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f9d24013510 tx=0x7f9d2400b040).stop 2023-11-17T05:27:48.479 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:48.461+0000 7f9d3be8e700 1 -- 172.21.15.160:0/2929607050 shutdown_connections 2023-11-17T05:27:48.479 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:48.461+0000 7f9d3be8e700 1 --2- 172.21.15.160:0/2929607050 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f9d2002ffb0 0x7f9d20032470 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:27:48.480 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:48.461+0000 7f9d3be8e700 1 --2- 172.21.15.160:0/2929607050 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d340f5410 0x7f9d340fd3a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:27:48.480 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:48.461+0000 7f9d3be8e700 1 -- 172.21.15.160:0/2929607050 >> 172.21.15.160:0/2929607050 conn(0x7f9d340f0b70 msgr2=0x7f9d3405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:27:48.480 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:48.461+0000 7f9d3be8e700 1 -- 172.21.15.160:0/2929607050 shutdown_connections 2023-11-17T05:27:48.480 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:48.461+0000 7f9d3be8e700 1 -- 172.21.15.160:0/2929607050 wait complete. 2023-11-17T05:27:48.480 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:27:48.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:27:48 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:27:47.410618+0000 mgr.smithi003.eddcuc (mgr.14164) 71 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:48.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:27:48 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:27:48.460857+0000 mon.smithi003 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.160:0/2929607050' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:27:49.838 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:27:49.839 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:27:50.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:27:49 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:27:49.410918+0000 mgr.smithi003.eddcuc (mgr.14164) 72 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:51.028 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:51.020+0000 7f440b3e4700 1 -- 172.21.15.160:0/3740605318 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44040f51a0 msgr2=0x7f44040f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:27:51.028 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:51.020+0000 7f440b3e4700 1 --2- 172.21.15.160:0/3740605318 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44040f51a0 0x7f44040f55c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f4400009fb0 tx=0x7f4400009b30).stop 2023-11-17T05:27:51.028 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:51.020+0000 7f440b3e4700 1 -- 172.21.15.160:0/3740605318 shutdown_connections 2023-11-17T05:27:51.028 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:51.020+0000 7f440b3e4700 1 --2- 172.21.15.160:0/3740605318 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44040f51a0 0x7f44040f55c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:27:51.028 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:51.020+0000 7f440b3e4700 1 -- 172.21.15.160:0/3740605318 >> 172.21.15.160:0/3740605318 conn(0x7f44040f09a0 msgr2=0x7f44040f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:27:51.028 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:51.020+0000 7f440b3e4700 1 -- 172.21.15.160:0/3740605318 shutdown_connections 2023-11-17T05:27:51.028 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:51.020+0000 7f440b3e4700 1 -- 172.21.15.160:0/3740605318 wait complete. 2023-11-17T05:27:51.029 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:51.020+0000 7f440b3e4700 1 Processor -- start 2023-11-17T05:27:51.029 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:51.020+0000 7f440b3e4700 1 -- start start 2023-11-17T05:27:51.029 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:51.021+0000 7f440b3e4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44040f51a0 0x7f44041064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:27:51.029 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:51.021+0000 7f440b3e4700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44041069d0 con 0x7f44040f51a0 2023-11-17T05:27:51.029 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:51.021+0000 7f440a3e2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44040f51a0 0x7f44041064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:27:51.029 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:51.021+0000 7f440a3e2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44040f51a0 0x7f44041064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:35042/0 (socket says 172.21.15.160:35042) 2023-11-17T05:27:51.029 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:51.021+0000 7f440a3e2700 1 -- 172.21.15.160:0/1002879481 learned_addr learned my addr 172.21.15.160:0/1002879481 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:27:51.029 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:51.021+0000 7f440a3e2700 1 -- 172.21.15.160:0/1002879481 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4400005040 con 0x7f44040f51a0 2023-11-17T05:27:51.029 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:51.021+0000 7f440a3e2700 1 --2- 172.21.15.160:0/1002879481 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44040f51a0 0x7f44041064e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f4400013510 tx=0x7f440000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:27:51.030 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:51.021+0000 7f43f6ffd700 1 -- 172.21.15.160:0/1002879481 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4400009320 con 0x7f44040f51a0 2023-11-17T05:27:51.030 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:51.022+0000 7f43f6ffd700 1 -- 172.21.15.160:0/1002879481 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4400013f00 con 0x7f44040f51a0 2023-11-17T05:27:51.030 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:51.022+0000 7f440b3e4700 1 -- 172.21.15.160:0/1002879481 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4404106bd0 con 0x7f44040f51a0 2023-11-17T05:27:51.031 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:51.022+0000 7f43f6ffd700 1 -- 172.21.15.160:0/1002879481 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4400014dc0 con 0x7f44040f51a0 2023-11-17T05:27:51.033 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:51.022+0000 7f440b3e4700 1 -- 172.21.15.160:0/1002879481 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4404107060 con 0x7f44040f51a0 2023-11-17T05:27:51.036 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:51.022+0000 7f43f6ffd700 1 -- 172.21.15.160:0/1002879481 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4400014420 con 0x7f44040f51a0 2023-11-17T05:27:51.037 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:51.023+0000 7f43f6ffd700 1 --2- 172.21.15.160:0/1002879481 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f43f002ff60 0x7f43f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:27:51.037 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:51.023+0000 7f440b3e4700 1 -- 172.21.15.160:0/1002879481 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4404045ad0 con 0x7f44040f51a0 2023-11-17T05:27:51.037 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:51.023+0000 7f43f6ffd700 1 -- 172.21.15.160:0/1002879481 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4400041030 con 0x7f44040f51a0 2023-11-17T05:27:51.038 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:51.023+0000 7f4409be1700 1 --2- 172.21.15.160:0/1002879481 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f43f002ff60 0x7f43f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:27:51.038 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:51.023+0000 7f4409be1700 1 --2- 172.21.15.160:0/1002879481 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f43f002ff60 0x7f43f0032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f43f80096a0 tx=0x7f43f8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:27:51.041 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:51.028+0000 7f43f6ffd700 1 -- 172.21.15.160:0/1002879481 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f44000095f0 con 0x7f44040f51a0 2023-11-17T05:27:51.231 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:51.230+0000 7f440b3e4700 1 -- 172.21.15.160:0/1002879481 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4404107f10 con 0x7f44040f51a0 2023-11-17T05:27:51.231 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:51.231+0000 7f43f6ffd700 1 -- 172.21.15.160:0/1002879481 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f4400023250 con 0x7f44040f51a0 2023-11-17T05:27:51.232 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:27:51.232 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:27:51.235 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:51.232+0000 7f440b3e4700 1 -- 172.21.15.160:0/1002879481 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f43f002ff60 msgr2=0x7f43f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:27:51.235 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:51.232+0000 7f440b3e4700 1 --2- 172.21.15.160:0/1002879481 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f43f002ff60 0x7f43f0032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f43f80096a0 tx=0x7f43f8006b40).stop 2023-11-17T05:27:51.236 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:51.232+0000 7f440b3e4700 1 -- 172.21.15.160:0/1002879481 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44040f51a0 msgr2=0x7f44041064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:27:51.236 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:51.232+0000 7f440b3e4700 1 --2- 172.21.15.160:0/1002879481 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44040f51a0 0x7f44041064e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f4400013510 tx=0x7f440000b040).stop 2023-11-17T05:27:51.237 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:51.232+0000 7f440b3e4700 1 -- 172.21.15.160:0/1002879481 shutdown_connections 2023-11-17T05:27:51.238 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:51.232+0000 7f440b3e4700 1 --2- 172.21.15.160:0/1002879481 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f43f002ff60 0x7f43f0032420 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:27:51.238 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:51.232+0000 7f440b3e4700 1 --2- 172.21.15.160:0/1002879481 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44040f51a0 0x7f44041064e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:27:51.239 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:51.232+0000 7f440b3e4700 1 -- 172.21.15.160:0/1002879481 >> 172.21.15.160:0/1002879481 conn(0x7f44040f09a0 msgr2=0x7f44040f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:27:51.239 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:51.232+0000 7f440b3e4700 1 -- 172.21.15.160:0/1002879481 shutdown_connections 2023-11-17T05:27:51.240 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:51.232+0000 7f440b3e4700 1 -- 172.21.15.160:0/1002879481 wait complete. 2023-11-17T05:27:51.240 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:27:51.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:27:51 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:27:51.231817+0000 mon.smithi003 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.160:0/1002879481' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:27:52.626 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:27:52.626 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:27:52.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:27:52 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:27:51.411334+0000 mgr.smithi003.eddcuc (mgr.14164) 73 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:53.768 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:53.765+0000 7fe1caeba700 1 -- 172.21.15.160:0/3574769461 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1c40f51a0 msgr2=0x7fe1c40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:27:53.768 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:53.765+0000 7fe1caeba700 1 --2- 172.21.15.160:0/3574769461 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1c40f51a0 0x7fe1c40f55c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fe1b4009fb0 tx=0x7fe1b4009b30).stop 2023-11-17T05:27:53.769 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:53.766+0000 7fe1caeba700 1 -- 172.21.15.160:0/3574769461 shutdown_connections 2023-11-17T05:27:53.769 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:53.766+0000 7fe1caeba700 1 --2- 172.21.15.160:0/3574769461 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1c40f51a0 0x7fe1c40f55c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:27:53.769 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:53.766+0000 7fe1caeba700 1 -- 172.21.15.160:0/3574769461 >> 172.21.15.160:0/3574769461 conn(0x7fe1c40f09a0 msgr2=0x7fe1c40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:27:53.769 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:53.766+0000 7fe1caeba700 1 -- 172.21.15.160:0/3574769461 shutdown_connections 2023-11-17T05:27:53.769 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:53.766+0000 7fe1caeba700 1 -- 172.21.15.160:0/3574769461 wait complete. 2023-11-17T05:27:53.769 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:53.766+0000 7fe1caeba700 1 Processor -- start 2023-11-17T05:27:53.769 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:53.766+0000 7fe1caeba700 1 -- start start 2023-11-17T05:27:53.769 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:53.766+0000 7fe1caeba700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1c40f51a0 0x7fe1c41042e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:27:53.769 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:53.766+0000 7fe1caeba700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c41047d0 con 0x7fe1c40f51a0 2023-11-17T05:27:53.769 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:53.767+0000 7fe1c9eb8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1c40f51a0 0x7fe1c41042e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:27:53.769 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:53.767+0000 7fe1c9eb8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1c40f51a0 0x7fe1c41042e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:35060/0 (socket says 172.21.15.160:35060) 2023-11-17T05:27:53.769 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:53.767+0000 7fe1c9eb8700 1 -- 172.21.15.160:0/611187600 learned_addr learned my addr 172.21.15.160:0/611187600 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:27:53.769 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:53.767+0000 7fe1c9eb8700 1 -- 172.21.15.160:0/611187600 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1b4005040 con 0x7fe1c40f51a0 2023-11-17T05:27:53.770 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:53.767+0000 7fe1c9eb8700 1 --2- 172.21.15.160:0/611187600 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1c40f51a0 0x7fe1c41042e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fe1b4013510 tx=0x7fe1b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:27:53.770 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:53.767+0000 7fe1ba7fc700 1 -- 172.21.15.160:0/611187600 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe1b4009320 con 0x7fe1c40f51a0 2023-11-17T05:27:53.770 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:53.767+0000 7fe1ba7fc700 1 -- 172.21.15.160:0/611187600 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe1b4013f00 con 0x7fe1c40f51a0 2023-11-17T05:27:53.770 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:53.767+0000 7fe1caeba700 1 -- 172.21.15.160:0/611187600 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1c41049d0 con 0x7fe1c40f51a0 2023-11-17T05:27:53.770 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:53.767+0000 7fe1ba7fc700 1 -- 172.21.15.160:0/611187600 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe1b4014dc0 con 0x7fe1c40f51a0 2023-11-17T05:27:53.770 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:53.767+0000 7fe1caeba700 1 -- 172.21.15.160:0/611187600 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1c4104e60 con 0x7fe1c40f51a0 2023-11-17T05:27:53.771 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:53.768+0000 7fe1ba7fc700 1 -- 172.21.15.160:0/611187600 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe1b4014420 con 0x7fe1c40f51a0 2023-11-17T05:27:53.771 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:53.768+0000 7fe1caeba700 1 -- 172.21.15.160:0/611187600 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1c4045ad0 con 0x7fe1c40f51a0 2023-11-17T05:27:53.771 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:53.768+0000 7fe1ba7fc700 1 --2- 172.21.15.160:0/611187600 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe1b00321b0 0x7fe1b0034670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:27:53.771 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:53.768+0000 7fe1ba7fc700 1 -- 172.21.15.160:0/611187600 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe1b4040ff0 con 0x7fe1c40f51a0 2023-11-17T05:27:53.771 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:53.768+0000 7fe1c96b7700 1 --2- 172.21.15.160:0/611187600 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe1b00321b0 0x7fe1b0034670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:27:53.771 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:53.769+0000 7fe1c96b7700 1 --2- 172.21.15.160:0/611187600 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe1b00321b0 0x7fe1b0034670 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fe1c00096a0 tx=0x7fe1c0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:27:53.774 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:53.774+0000 7fe1ba7fc700 1 -- 172.21.15.160:0/611187600 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe1b40095f0 con 0x7fe1c40f51a0 2023-11-17T05:27:53.982 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:53.981+0000 7fe1caeba700 1 -- 172.21.15.160:0/611187600 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe1c4105d10 con 0x7fe1c40f51a0 2023-11-17T05:27:53.983 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:53.982+0000 7fe1ba7fc700 1 -- 172.21.15.160:0/611187600 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fe1b401db70 con 0x7fe1c40f51a0 2023-11-17T05:27:53.983 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:27:53.983 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:27:53.985 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:53.985+0000 7fe1caeba700 1 -- 172.21.15.160:0/611187600 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe1b00321b0 msgr2=0x7fe1b0034670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:27:53.985 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:53.985+0000 7fe1caeba700 1 --2- 172.21.15.160:0/611187600 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe1b00321b0 0x7fe1b0034670 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fe1c00096a0 tx=0x7fe1c0006b40).stop 2023-11-17T05:27:53.985 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:53.985+0000 7fe1caeba700 1 -- 172.21.15.160:0/611187600 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1c40f51a0 msgr2=0x7fe1c41042e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:27:53.985 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:53.985+0000 7fe1caeba700 1 --2- 172.21.15.160:0/611187600 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1c40f51a0 0x7fe1c41042e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fe1b4013510 tx=0x7fe1b400b040).stop 2023-11-17T05:27:53.985 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:53.985+0000 7fe1caeba700 1 -- 172.21.15.160:0/611187600 shutdown_connections 2023-11-17T05:27:53.985 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:53.985+0000 7fe1caeba700 1 --2- 172.21.15.160:0/611187600 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe1b00321b0 0x7fe1b0034670 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:27:53.985 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:53.985+0000 7fe1caeba700 1 --2- 172.21.15.160:0/611187600 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1c40f51a0 0x7fe1c41042e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:27:53.985 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:53.985+0000 7fe1caeba700 1 -- 172.21.15.160:0/611187600 >> 172.21.15.160:0/611187600 conn(0x7fe1c40f09a0 msgr2=0x7fe1c40f13f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:27:53.985 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:53.985+0000 7fe1caeba700 1 -- 172.21.15.160:0/611187600 shutdown_connections 2023-11-17T05:27:53.985 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:53.985+0000 7fe1caeba700 1 -- 172.21.15.160:0/611187600 wait complete. 2023-11-17T05:27:53.986 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:27:54.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:27:54 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:27:53.411662+0000 mgr.smithi003.eddcuc (mgr.14164) 74 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:54.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:27:54 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:27:53.983369+0000 mon.smithi003 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.160:0/611187600' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:27:55.366 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:27:55.366 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:27:56.542 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:56.542+0000 7f9185804700 1 -- 172.21.15.160:0/2824654796 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f91800f51a0 msgr2=0x7f91800f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:27:56.543 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:56.542+0000 7f9185804700 1 --2- 172.21.15.160:0/2824654796 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f91800f51a0 0x7f91800f55c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f9174009fb0 tx=0x7f9174009b30).stop 2023-11-17T05:27:56.543 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:56.542+0000 7f9185804700 1 -- 172.21.15.160:0/2824654796 shutdown_connections 2023-11-17T05:27:56.543 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:56.542+0000 7f9185804700 1 --2- 172.21.15.160:0/2824654796 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f91800f51a0 0x7f91800f55c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:27:56.543 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:56.542+0000 7f9185804700 1 -- 172.21.15.160:0/2824654796 >> 172.21.15.160:0/2824654796 conn(0x7f91800f09a0 msgr2=0x7f91800f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:27:56.543 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:56.542+0000 7f9185804700 1 -- 172.21.15.160:0/2824654796 shutdown_connections 2023-11-17T05:27:56.543 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:56.542+0000 7f9185804700 1 -- 172.21.15.160:0/2824654796 wait complete. 2023-11-17T05:27:56.543 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:56.542+0000 7f9185804700 1 Processor -- start 2023-11-17T05:27:56.543 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:56.542+0000 7f9185804700 1 -- start start 2023-11-17T05:27:56.543 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:56.543+0000 7f9185804700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f91800f51a0 0x7f91801064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:27:56.543 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:56.543+0000 7f9185804700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91801069d0 con 0x7f91800f51a0 2023-11-17T05:27:56.543 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:56.543+0000 7f9184802700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f91800f51a0 0x7f91801064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:27:56.544 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:56.543+0000 7f9184802700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f91800f51a0 0x7f91801064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:35074/0 (socket says 172.21.15.160:35074) 2023-11-17T05:27:56.544 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:56.543+0000 7f9184802700 1 -- 172.21.15.160:0/1476811456 learned_addr learned my addr 172.21.15.160:0/1476811456 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:27:56.545 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:56.543+0000 7f9184802700 1 -- 172.21.15.160:0/1476811456 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9174005040 con 0x7f91800f51a0 2023-11-17T05:27:56.545 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:56.543+0000 7f9184802700 1 --2- 172.21.15.160:0/1476811456 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f91800f51a0 0x7f91801064e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f9174013510 tx=0x7f917400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:27:56.545 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:56.543+0000 7f917d7fa700 1 -- 172.21.15.160:0/1476811456 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9174009320 con 0x7f91800f51a0 2023-11-17T05:27:56.545 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:56.543+0000 7f917d7fa700 1 -- 172.21.15.160:0/1476811456 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9174013f00 con 0x7f91800f51a0 2023-11-17T05:27:56.545 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:56.543+0000 7f9185804700 1 -- 172.21.15.160:0/1476811456 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9180106bd0 con 0x7f91800f51a0 2023-11-17T05:27:56.546 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:56.543+0000 7f917d7fa700 1 -- 172.21.15.160:0/1476811456 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9174014dc0 con 0x7f91800f51a0 2023-11-17T05:27:56.546 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:56.544+0000 7f9185804700 1 -- 172.21.15.160:0/1476811456 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9180107060 con 0x7f91800f51a0 2023-11-17T05:27:56.546 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:56.544+0000 7f917d7fa700 1 -- 172.21.15.160:0/1476811456 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f917401d3d0 con 0x7f91800f51a0 2023-11-17T05:27:56.546 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:56.544+0000 7f917d7fa700 1 --2- 172.21.15.160:0/1476811456 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f917002ff60 0x7f9170032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:27:56.546 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:56.544+0000 7f917d7fa700 1 -- 172.21.15.160:0/1476811456 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9174041120 con 0x7f91800f51a0 2023-11-17T05:27:56.546 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:56.544+0000 7f9185804700 1 -- 172.21.15.160:0/1476811456 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9180045ad0 con 0x7f91800f51a0 2023-11-17T05:27:56.546 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:56.545+0000 7f917ffff700 1 --2- 172.21.15.160:0/1476811456 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f917002ff60 0x7f9170032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:27:56.546 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:56.545+0000 7f917ffff700 1 --2- 172.21.15.160:0/1476811456 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f917002ff60 0x7f9170032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f916c0096a0 tx=0x7f916c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:27:56.549 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:56.550+0000 7f917d7fa700 1 -- 172.21.15.160:0/1476811456 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9174012920 con 0x7f91800f51a0 2023-11-17T05:27:56.756 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:56.756+0000 7f9185804700 1 -- 172.21.15.160:0/1476811456 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f91800facd0 con 0x7f91800f51a0 2023-11-17T05:27:56.756 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:56.757+0000 7f917d7fa700 1 -- 172.21.15.160:0/1476811456 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f917402b020 con 0x7f91800f51a0 2023-11-17T05:27:56.757 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:27:56.757 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:27:56.759 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:56.758+0000 7f9185804700 1 -- 172.21.15.160:0/1476811456 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f917002ff60 msgr2=0x7f9170032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:27:56.759 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:56.758+0000 7f9185804700 1 --2- 172.21.15.160:0/1476811456 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f917002ff60 0x7f9170032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f916c0096a0 tx=0x7f916c006b40).stop 2023-11-17T05:27:56.759 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:56.758+0000 7f9185804700 1 -- 172.21.15.160:0/1476811456 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f91800f51a0 msgr2=0x7f91801064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:27:56.759 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:56.759+0000 7f9185804700 1 --2- 172.21.15.160:0/1476811456 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f91800f51a0 0x7f91801064e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f9174013510 tx=0x7f917400b040).stop 2023-11-17T05:27:56.759 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:56.759+0000 7f9185804700 1 -- 172.21.15.160:0/1476811456 shutdown_connections 2023-11-17T05:27:56.759 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:56.759+0000 7f9185804700 1 --2- 172.21.15.160:0/1476811456 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f917002ff60 0x7f9170032420 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:27:56.759 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:56.759+0000 7f9185804700 1 --2- 172.21.15.160:0/1476811456 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f91800f51a0 0x7f91801064e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:27:56.759 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:56.759+0000 7f9185804700 1 -- 172.21.15.160:0/1476811456 >> 172.21.15.160:0/1476811456 conn(0x7f91800f09a0 msgr2=0x7f918005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:27:56.759 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:56.759+0000 7f9185804700 1 -- 172.21.15.160:0/1476811456 shutdown_connections 2023-11-17T05:27:56.759 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:56.759+0000 7f9185804700 1 -- 172.21.15.160:0/1476811456 wait complete. 2023-11-17T05:27:56.760 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:27:56.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:27:56 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:27:55.412022+0000 mgr.smithi003.eddcuc (mgr.14164) 75 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:57.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:27:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:27:56.756989+0000 mon.smithi003 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.160:0/1476811456' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:27:58.110 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:27:58.110 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:27:58.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:27:58 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:27:57.412360+0000 mgr.smithi003.eddcuc (mgr.14164) 76 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:27:59.346 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:59.345+0000 7f49f758b700 1 -- 172.21.15.160:0/927428982 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49f00f62a0 msgr2=0x7f49f00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:27:59.347 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:59.345+0000 7f49f758b700 1 --2- 172.21.15.160:0/927428982 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49f00f62a0 0x7f49f00f66c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f49e0009fb0 tx=0x7f49e0009b30).stop 2023-11-17T05:27:59.347 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:59.345+0000 7f49f758b700 1 -- 172.21.15.160:0/927428982 shutdown_connections 2023-11-17T05:27:59.347 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:59.345+0000 7f49f758b700 1 --2- 172.21.15.160:0/927428982 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49f00f62a0 0x7f49f00f66c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:27:59.347 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:59.345+0000 7f49f758b700 1 -- 172.21.15.160:0/927428982 >> 172.21.15.160:0/927428982 conn(0x7f49f00f1aa0 msgr2=0x7f49f00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:27:59.347 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:59.345+0000 7f49f758b700 1 -- 172.21.15.160:0/927428982 shutdown_connections 2023-11-17T05:27:59.347 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:59.345+0000 7f49f758b700 1 -- 172.21.15.160:0/927428982 wait complete. 2023-11-17T05:27:59.347 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:59.346+0000 7f49f758b700 1 Processor -- start 2023-11-17T05:27:59.347 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:59.346+0000 7f49f758b700 1 -- start start 2023-11-17T05:27:59.347 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:59.346+0000 7f49f758b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49f00f62a0 0x7f49f01064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:27:59.347 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:59.346+0000 7f49f758b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f01069a0 con 0x7f49f00f62a0 2023-11-17T05:27:59.349 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:59.346+0000 7f49f6589700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49f00f62a0 0x7f49f01064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:27:59.349 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:59.346+0000 7f49f6589700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49f00f62a0 0x7f49f01064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:35102/0 (socket says 172.21.15.160:35102) 2023-11-17T05:27:59.349 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:59.346+0000 7f49f6589700 1 -- 172.21.15.160:0/2531125510 learned_addr learned my addr 172.21.15.160:0/2531125510 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:27:59.349 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:59.346+0000 7f49f6589700 1 -- 172.21.15.160:0/2531125510 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49e0005040 con 0x7f49f00f62a0 2023-11-17T05:27:59.349 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:59.347+0000 7f49f6589700 1 --2- 172.21.15.160:0/2531125510 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49f00f62a0 0x7f49f01064b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f49e0013510 tx=0x7f49e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:27:59.349 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:59.347+0000 7f49e6ffd700 1 -- 172.21.15.160:0/2531125510 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f49e0009320 con 0x7f49f00f62a0 2023-11-17T05:27:59.349 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:59.347+0000 7f49f758b700 1 -- 172.21.15.160:0/2531125510 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49f0106ba0 con 0x7f49f00f62a0 2023-11-17T05:27:59.349 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:59.347+0000 7f49e6ffd700 1 -- 172.21.15.160:0/2531125510 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f49e0013f00 con 0x7f49f00f62a0 2023-11-17T05:27:59.349 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:59.347+0000 7f49e6ffd700 1 -- 172.21.15.160:0/2531125510 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f49e0014dc0 con 0x7f49f00f62a0 2023-11-17T05:27:59.349 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:59.347+0000 7f49f758b700 1 -- 172.21.15.160:0/2531125510 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49f0107030 con 0x7f49f00f62a0 2023-11-17T05:27:59.349 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:59.348+0000 7f49e6ffd700 1 -- 172.21.15.160:0/2531125510 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f49e00095f0 con 0x7f49f00f62a0 2023-11-17T05:27:59.350 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:59.348+0000 7f49f758b700 1 -- 172.21.15.160:0/2531125510 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49f0045ad0 con 0x7f49f00f62a0 2023-11-17T05:27:59.350 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:59.348+0000 7f49e6ffd700 1 --2- 172.21.15.160:0/2531125510 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f49dc02ff60 0x7f49dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:27:59.350 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:59.348+0000 7f49e6ffd700 1 -- 172.21.15.160:0/2531125510 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f49e0041040 con 0x7f49f00f62a0 2023-11-17T05:27:59.350 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:59.348+0000 7f49f5d88700 1 --2- 172.21.15.160:0/2531125510 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f49dc02ff60 0x7f49dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:27:59.350 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:59.349+0000 7f49f5d88700 1 --2- 172.21.15.160:0/2531125510 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f49dc02ff60 0x7f49dc032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f49ec0096a0 tx=0x7f49ec006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:27:59.353 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:59.353+0000 7f49e6ffd700 1 -- 172.21.15.160:0/2531125510 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f49e0014420 con 0x7f49f00f62a0 2023-11-17T05:27:59.557 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:59.556+0000 7f49f758b700 1 -- 172.21.15.160:0/2531125510 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f49f0107ee0 con 0x7f49f00f62a0 2023-11-17T05:27:59.557 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:59.556+0000 7f49e6ffd700 1 -- 172.21.15.160:0/2531125510 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f49e0010230 con 0x7f49f00f62a0 2023-11-17T05:27:59.557 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:27:59.557 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:27:59.559 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:59.558+0000 7f49f758b700 1 -- 172.21.15.160:0/2531125510 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f49dc02ff60 msgr2=0x7f49dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:27:59.559 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:59.558+0000 7f49f758b700 1 --2- 172.21.15.160:0/2531125510 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f49dc02ff60 0x7f49dc032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f49ec0096a0 tx=0x7f49ec006b40).stop 2023-11-17T05:27:59.559 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:59.558+0000 7f49f758b700 1 -- 172.21.15.160:0/2531125510 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49f00f62a0 msgr2=0x7f49f01064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:27:59.559 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:59.558+0000 7f49f758b700 1 --2- 172.21.15.160:0/2531125510 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49f00f62a0 0x7f49f01064b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f49e0013510 tx=0x7f49e000b040).stop 2023-11-17T05:27:59.559 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:59.558+0000 7f49f758b700 1 -- 172.21.15.160:0/2531125510 shutdown_connections 2023-11-17T05:27:59.559 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:59.558+0000 7f49f758b700 1 --2- 172.21.15.160:0/2531125510 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f49dc02ff60 0x7f49dc032420 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:27:59.559 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:59.558+0000 7f49f758b700 1 --2- 172.21.15.160:0/2531125510 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49f00f62a0 0x7f49f01064b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:27:59.560 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:59.558+0000 7f49f758b700 1 -- 172.21.15.160:0/2531125510 >> 172.21.15.160:0/2531125510 conn(0x7f49f00f1aa0 msgr2=0x7f49f00f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:27:59.560 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:59.558+0000 7f49f758b700 1 -- 172.21.15.160:0/2531125510 shutdown_connections 2023-11-17T05:27:59.560 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:27:59.558+0000 7f49f758b700 1 -- 172.21.15.160:0/2531125510 wait complete. 2023-11-17T05:27:59.560 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:28:00.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:27:59 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:27:59.412787+0000 mgr.smithi003.eddcuc (mgr.14164) 77 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:28:00.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:27:59 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:27:59.557554+0000 mon.smithi003 (mon.0) 139 : audit [DBG] from='client.? 172.21.15.160:0/2531125510' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:28:00.944 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:28:00.944 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:28:02.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:02.089+0000 7ff99e0a6700 1 -- 172.21.15.160:0/2369735949 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9980f6510 msgr2=0x7ff9980f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:28:02.149 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:02.089+0000 7ff99e0a6700 1 --2- 172.21.15.160:0/2369735949 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9980f6510 0x7ff9980f6930 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7ff98c009fb0 tx=0x7ff98c009b30).stop 2023-11-17T05:28:02.149 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:02.089+0000 7ff99e0a6700 1 -- 172.21.15.160:0/2369735949 shutdown_connections 2023-11-17T05:28:02.149 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:02.089+0000 7ff99e0a6700 1 --2- 172.21.15.160:0/2369735949 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9980f6510 0x7ff9980f6930 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:28:02.150 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:02.089+0000 7ff99e0a6700 1 -- 172.21.15.160:0/2369735949 >> 172.21.15.160:0/2369735949 conn(0x7ff9980f1c70 msgr2=0x7ff9980f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:28:02.150 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:02.089+0000 7ff99e0a6700 1 -- 172.21.15.160:0/2369735949 shutdown_connections 2023-11-17T05:28:02.150 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:02.089+0000 7ff99e0a6700 1 -- 172.21.15.160:0/2369735949 wait complete. 2023-11-17T05:28:02.150 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:02.090+0000 7ff99e0a6700 1 Processor -- start 2023-11-17T05:28:02.152 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:02.090+0000 7ff99e0a6700 1 -- start start 2023-11-17T05:28:02.152 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:02.090+0000 7ff99e0a6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9980f6510 0x7ff99806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:28:02.152 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:02.090+0000 7ff99e0a6700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff998069e20 con 0x7ff9980f6510 2023-11-17T05:28:02.152 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:02.090+0000 7ff9977fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9980f6510 0x7ff99806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:28:02.152 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:02.090+0000 7ff9977fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9980f6510 0x7ff99806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:55454/0 (socket says 172.21.15.160:55454) 2023-11-17T05:28:02.152 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:02.090+0000 7ff9977fe700 1 -- 172.21.15.160:0/2039334838 learned_addr learned my addr 172.21.15.160:0/2039334838 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:28:02.152 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:02.091+0000 7ff9977fe700 1 -- 172.21.15.160:0/2039334838 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff98c005040 con 0x7ff9980f6510 2023-11-17T05:28:02.152 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:02.091+0000 7ff9977fe700 1 --2- 172.21.15.160:0/2039334838 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9980f6510 0x7ff99806b9a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7ff98c013510 tx=0x7ff98c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:28:02.152 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:02.091+0000 7ff97ffff700 1 -- 172.21.15.160:0/2039334838 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff98c009320 con 0x7ff9980f6510 2023-11-17T05:28:02.152 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:02.091+0000 7ff99e0a6700 1 -- 172.21.15.160:0/2039334838 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff998069fc0 con 0x7ff9980f6510 2023-11-17T05:28:02.152 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:02.091+0000 7ff97ffff700 1 -- 172.21.15.160:0/2039334838 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff98c013f00 con 0x7ff9980f6510 2023-11-17T05:28:02.153 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:02.091+0000 7ff97ffff700 1 -- 172.21.15.160:0/2039334838 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff98c014d90 con 0x7ff9980f6510 2023-11-17T05:28:02.153 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:02.091+0000 7ff99e0a6700 1 -- 172.21.15.160:0/2039334838 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff99806a450 con 0x7ff9980f6510 2023-11-17T05:28:02.153 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:02.092+0000 7ff97ffff700 1 -- 172.21.15.160:0/2039334838 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff98c0095f0 con 0x7ff9980f6510 2023-11-17T05:28:02.153 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:02.092+0000 7ff99e0a6700 1 -- 172.21.15.160:0/2039334838 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff998059100 con 0x7ff9980f6510 2023-11-17T05:28:02.245 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:02.092+0000 7ff97ffff700 1 --2- 172.21.15.160:0/2039334838 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7ff98002ff60 0x7ff980032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:28:02.245 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:02.092+0000 7ff97ffff700 1 -- 172.21.15.160:0/2039334838 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff98c041000 con 0x7ff9980f6510 2023-11-17T05:28:02.245 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:02.092+0000 7ff996ffd700 1 --2- 172.21.15.160:0/2039334838 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7ff98002ff60 0x7ff980032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:28:02.245 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:02.093+0000 7ff996ffd700 1 --2- 172.21.15.160:0/2039334838 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7ff98002ff60 0x7ff980032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7ff9880096a0 tx=0x7ff988006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:28:02.245 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:02.097+0000 7ff97ffff700 1 -- 172.21.15.160:0/2039334838 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff98c014420 con 0x7ff9980f6510 2023-11-17T05:28:02.306 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:02.305+0000 7ff99e0a6700 1 -- 172.21.15.160:0/2039334838 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff9980fbf60 con 0x7ff9980f6510 2023-11-17T05:28:02.306 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:02.306+0000 7ff97ffff700 1 -- 172.21.15.160:0/2039334838 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7ff98c01d7f0 con 0x7ff9980f6510 2023-11-17T05:28:02.307 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:28:02.307 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:28:02.308 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:02.307+0000 7ff99e0a6700 1 -- 172.21.15.160:0/2039334838 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7ff98002ff60 msgr2=0x7ff980032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:28:02.309 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:02.307+0000 7ff99e0a6700 1 --2- 172.21.15.160:0/2039334838 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7ff98002ff60 0x7ff980032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7ff9880096a0 tx=0x7ff988006b40).stop 2023-11-17T05:28:02.309 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:02.307+0000 7ff99e0a6700 1 -- 172.21.15.160:0/2039334838 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9980f6510 msgr2=0x7ff99806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:28:02.309 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:02.307+0000 7ff99e0a6700 1 --2- 172.21.15.160:0/2039334838 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9980f6510 0x7ff99806b9a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7ff98c013510 tx=0x7ff98c00b040).stop 2023-11-17T05:28:02.309 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:02.307+0000 7ff99e0a6700 1 -- 172.21.15.160:0/2039334838 shutdown_connections 2023-11-17T05:28:02.309 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:02.307+0000 7ff99e0a6700 1 --2- 172.21.15.160:0/2039334838 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7ff98002ff60 0x7ff980032420 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:28:02.309 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:02.307+0000 7ff99e0a6700 1 --2- 172.21.15.160:0/2039334838 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9980f6510 0x7ff99806b9a0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:28:02.309 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:02.307+0000 7ff99e0a6700 1 -- 172.21.15.160:0/2039334838 >> 172.21.15.160:0/2039334838 conn(0x7ff9980f1c70 msgr2=0x7ff9980f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:28:02.309 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:02.308+0000 7ff99e0a6700 1 -- 172.21.15.160:0/2039334838 shutdown_connections 2023-11-17T05:28:02.309 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:02.308+0000 7ff99e0a6700 1 -- 172.21.15.160:0/2039334838 wait complete. 2023-11-17T05:28:02.310 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:28:02.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:02 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:28:01.413227+0000 mgr.smithi003.eddcuc (mgr.14164) 78 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:28:02.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:02 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:28:02.306760+0000 mon.smithi003 (mon.0) 140 : audit [DBG] from='client.? 172.21.15.160:0/2039334838' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:28:03.685 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:28:03.686 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:28:04.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:04 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:28:03.413561+0000 mgr.smithi003.eddcuc (mgr.14164) 79 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:28:04.868 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:04.867+0000 7f6dfe3ae700 1 -- 172.21.15.160:0/3824082327 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6df80f4b80 msgr2=0x7f6df80f4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:28:04.869 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:04.867+0000 7f6dfe3ae700 1 --2- 172.21.15.160:0/3824082327 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6df80f4b80 0x7f6df80f4fa0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f6de0004770 tx=0x7f6de000ac40).stop 2023-11-17T05:28:04.869 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:04.867+0000 7f6dfe3ae700 1 -- 172.21.15.160:0/3824082327 shutdown_connections 2023-11-17T05:28:04.869 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:04.867+0000 7f6dfe3ae700 1 --2- 172.21.15.160:0/3824082327 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6df80f4b80 0x7f6df80f4fa0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:28:04.869 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:04.867+0000 7f6dfe3ae700 1 -- 172.21.15.160:0/3824082327 >> 172.21.15.160:0/3824082327 conn(0x7f6df80f02e0 msgr2=0x7f6df80f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:28:04.869 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:04.867+0000 7f6dfe3ae700 1 -- 172.21.15.160:0/3824082327 shutdown_connections 2023-11-17T05:28:04.869 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:04.867+0000 7f6dfe3ae700 1 -- 172.21.15.160:0/3824082327 wait complete. 2023-11-17T05:28:04.869 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:04.868+0000 7f6dfe3ae700 1 Processor -- start 2023-11-17T05:28:04.870 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:04.868+0000 7f6dfe3ae700 1 -- start start 2023-11-17T05:28:04.870 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:04.868+0000 7f6dfe3ae700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6df80f4b80 0x7f6df81066f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:28:04.870 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:04.868+0000 7f6dfe3ae700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df8106be0 con 0x7f6df80f4b80 2023-11-17T05:28:04.871 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:04.868+0000 7f6df7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6df80f4b80 0x7f6df81066f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:28:04.871 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:04.869+0000 7f6df7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6df80f4b80 0x7f6df81066f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:55474/0 (socket says 172.21.15.160:55474) 2023-11-17T05:28:04.871 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:04.869+0000 7f6df7fff700 1 -- 172.21.15.160:0/3133366721 learned_addr learned my addr 172.21.15.160:0/3133366721 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:28:04.871 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:04.869+0000 7f6df7fff700 1 -- 172.21.15.160:0/3133366721 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6de0005040 con 0x7f6df80f4b80 2023-11-17T05:28:04.871 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:04.869+0000 7f6df7fff700 1 --2- 172.21.15.160:0/3133366721 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6df80f4b80 0x7f6df81066f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f6de0004f00 tx=0x7f6de000ac40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:28:04.871 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:04.869+0000 7f6df4ff9700 1 -- 172.21.15.160:0/3133366721 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6de0019070 con 0x7f6df80f4b80 2023-11-17T05:28:04.871 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:04.869+0000 7f6dfe3ae700 1 -- 172.21.15.160:0/3133366721 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6df8106de0 con 0x7f6df80f4b80 2023-11-17T05:28:04.871 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:04.869+0000 7f6df4ff9700 1 -- 172.21.15.160:0/3133366721 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6de0007c60 con 0x7f6df80f4b80 2023-11-17T05:28:04.871 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:04.869+0000 7f6df4ff9700 1 -- 172.21.15.160:0/3133366721 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6de0008340 con 0x7f6df80f4b80 2023-11-17T05:28:04.871 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:04.869+0000 7f6dfe3ae700 1 -- 172.21.15.160:0/3133366721 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6df8107270 con 0x7f6df80f4b80 2023-11-17T05:28:04.872 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:04.870+0000 7f6df4ff9700 1 -- 172.21.15.160:0/3133366721 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6de0008530 con 0x7f6df80f4b80 2023-11-17T05:28:04.872 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:04.870+0000 7f6dfe3ae700 1 -- 172.21.15.160:0/3133366721 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6df8045ad0 con 0x7f6df80f4b80 2023-11-17T05:28:04.872 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:04.870+0000 7f6df4ff9700 1 --2- 172.21.15.160:0/3133366721 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f6de402ffb0 0x7f6de4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:28:04.872 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:04.870+0000 7f6df4ff9700 1 -- 172.21.15.160:0/3133366721 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6de0040b50 con 0x7f6df80f4b80 2023-11-17T05:28:04.872 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:04.871+0000 7f6df77fe700 1 --2- 172.21.15.160:0/3133366721 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f6de402ffb0 0x7f6de4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:28:04.873 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:04.871+0000 7f6df77fe700 1 --2- 172.21.15.160:0/3133366721 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f6de402ffb0 0x7f6de4032470 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f6de80096a0 tx=0x7f6de8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:28:04.876 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:04.875+0000 7f6df4ff9700 1 -- 172.21.15.160:0/3133366721 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6de001c500 con 0x7f6df80f4b80 2023-11-17T05:28:05.081 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:05.080+0000 7f6dfe3ae700 1 -- 172.21.15.160:0/3133366721 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6df8108120 con 0x7f6df80f4b80 2023-11-17T05:28:05.082 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:05.081+0000 7f6df4ff9700 1 -- 172.21.15.160:0/3133366721 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f6de001c320 con 0x7f6df80f4b80 2023-11-17T05:28:05.083 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:28:05.083 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:28:05.084 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:05.083+0000 7f6dfe3ae700 1 -- 172.21.15.160:0/3133366721 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f6de402ffb0 msgr2=0x7f6de4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:28:05.084 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:05.083+0000 7f6dfe3ae700 1 --2- 172.21.15.160:0/3133366721 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f6de402ffb0 0x7f6de4032470 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f6de80096a0 tx=0x7f6de8006b40).stop 2023-11-17T05:28:05.084 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:05.083+0000 7f6dfe3ae700 1 -- 172.21.15.160:0/3133366721 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6df80f4b80 msgr2=0x7f6df81066f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:28:05.084 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:05.083+0000 7f6dfe3ae700 1 --2- 172.21.15.160:0/3133366721 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6df80f4b80 0x7f6df81066f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f6de0004f00 tx=0x7f6de000ac40).stop 2023-11-17T05:28:05.084 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:05.083+0000 7f6dfe3ae700 1 -- 172.21.15.160:0/3133366721 shutdown_connections 2023-11-17T05:28:05.084 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:05.083+0000 7f6dfe3ae700 1 --2- 172.21.15.160:0/3133366721 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f6de402ffb0 0x7f6de4032470 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:28:05.084 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:05.083+0000 7f6dfe3ae700 1 --2- 172.21.15.160:0/3133366721 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6df80f4b80 0x7f6df81066f0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:28:05.084 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:05.083+0000 7f6dfe3ae700 1 -- 172.21.15.160:0/3133366721 >> 172.21.15.160:0/3133366721 conn(0x7f6df80f02e0 msgr2=0x7f6df80f0d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:28:05.085 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:05.083+0000 7f6dfe3ae700 1 -- 172.21.15.160:0/3133366721 shutdown_connections 2023-11-17T05:28:05.085 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:05.083+0000 7f6dfe3ae700 1 -- 172.21.15.160:0/3133366721 wait complete. 2023-11-17T05:28:05.085 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:28:05.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:05 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:28:05.082555+0000 mon.smithi003 (mon.0) 141 : audit [DBG] from='client.? 172.21.15.160:0/3133366721' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:28:06.509 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:28:06.509 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:28:06.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:06 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:28:05.413955+0000 mgr.smithi003.eddcuc (mgr.14164) 80 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:28:07.733 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:07.732+0000 7fc34359e700 1 -- 172.21.15.160:0/2113888753 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3440601e0 msgr2=0x7fc3440f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:28:07.733 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:07.732+0000 7fc34359e700 1 --2- 172.21.15.160:0/2113888753 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3440601e0 0x7fc3440f8f20 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fc334009fb0 tx=0x7fc334009b30).stop 2023-11-17T05:28:07.733 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:07.732+0000 7fc34359e700 1 -- 172.21.15.160:0/2113888753 shutdown_connections 2023-11-17T05:28:07.733 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:07.732+0000 7fc34359e700 1 --2- 172.21.15.160:0/2113888753 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3440601e0 0x7fc3440f8f20 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:28:07.733 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:07.732+0000 7fc34359e700 1 -- 172.21.15.160:0/2113888753 >> 172.21.15.160:0/2113888753 conn(0x7fc3440f09a0 msgr2=0x7fc3440f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:28:07.733 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:07.732+0000 7fc34359e700 1 -- 172.21.15.160:0/2113888753 shutdown_connections 2023-11-17T05:28:07.734 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:07.732+0000 7fc34359e700 1 -- 172.21.15.160:0/2113888753 wait complete. 2023-11-17T05:28:07.734 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:07.732+0000 7fc34359e700 1 Processor -- start 2023-11-17T05:28:07.734 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:07.732+0000 7fc34359e700 1 -- start start 2023-11-17T05:28:07.734 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:07.732+0000 7fc34359e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3440601e0 0x7fc344106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:28:07.734 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:07.732+0000 7fc34359e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3441069f0 con 0x7fc3440601e0 2023-11-17T05:28:07.735 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:07.733+0000 7fc34259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3440601e0 0x7fc344106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:28:07.735 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:07.733+0000 7fc34259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3440601e0 0x7fc344106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:55482/0 (socket says 172.21.15.160:55482) 2023-11-17T05:28:07.735 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:07.733+0000 7fc34259c700 1 -- 172.21.15.160:0/2159906318 learned_addr learned my addr 172.21.15.160:0/2159906318 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:28:07.736 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:07.733+0000 7fc34259c700 1 -- 172.21.15.160:0/2159906318 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc334005040 con 0x7fc3440601e0 2023-11-17T05:28:07.736 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:07.733+0000 7fc34259c700 1 --2- 172.21.15.160:0/2159906318 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3440601e0 0x7fc344106500 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fc334013510 tx=0x7fc33400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:28:07.736 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:07.733+0000 7fc332ffd700 1 -- 172.21.15.160:0/2159906318 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc334009320 con 0x7fc3440601e0 2023-11-17T05:28:07.736 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:07.733+0000 7fc34359e700 1 -- 172.21.15.160:0/2159906318 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc344106bf0 con 0x7fc3440601e0 2023-11-17T05:28:07.736 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:07.733+0000 7fc332ffd700 1 -- 172.21.15.160:0/2159906318 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc334013f00 con 0x7fc3440601e0 2023-11-17T05:28:07.736 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:07.733+0000 7fc332ffd700 1 -- 172.21.15.160:0/2159906318 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc334014d90 con 0x7fc3440601e0 2023-11-17T05:28:07.736 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:07.733+0000 7fc34359e700 1 -- 172.21.15.160:0/2159906318 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc344107080 con 0x7fc3440601e0 2023-11-17T05:28:07.736 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:07.734+0000 7fc332ffd700 1 -- 172.21.15.160:0/2159906318 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc3340095f0 con 0x7fc3440601e0 2023-11-17T05:28:07.736 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:07.734+0000 7fc34359e700 1 -- 172.21.15.160:0/2159906318 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc344045ad0 con 0x7fc3440601e0 2023-11-17T05:28:07.736 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:07.734+0000 7fc332ffd700 1 --2- 172.21.15.160:0/2159906318 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc32c02ff60 0x7fc32c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:28:07.736 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:07.734+0000 7fc332ffd700 1 -- 172.21.15.160:0/2159906318 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc334040fe0 con 0x7fc3440601e0 2023-11-17T05:28:07.736 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:07.734+0000 7fc341d9b700 1 --2- 172.21.15.160:0/2159906318 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc32c02ff60 0x7fc32c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:28:07.737 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:07.735+0000 7fc341d9b700 1 --2- 172.21.15.160:0/2159906318 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc32c02ff60 0x7fc32c032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fc3380096a0 tx=0x7fc338006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:28:07.739 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:07.739+0000 7fc332ffd700 1 -- 172.21.15.160:0/2159906318 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc33401d7f0 con 0x7fc3440601e0 2023-11-17T05:28:07.944 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:07.944+0000 7fc34359e700 1 -- 172.21.15.160:0/2159906318 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc344107f30 con 0x7fc3440601e0 2023-11-17T05:28:07.945 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:07.944+0000 7fc332ffd700 1 -- 172.21.15.160:0/2159906318 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fc334010230 con 0x7fc3440601e0 2023-11-17T05:28:07.945 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:28:07.945 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:28:07.948 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:07.946+0000 7fc34359e700 1 -- 172.21.15.160:0/2159906318 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc32c02ff60 msgr2=0x7fc32c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:28:07.948 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:07.946+0000 7fc34359e700 1 --2- 172.21.15.160:0/2159906318 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc32c02ff60 0x7fc32c032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fc3380096a0 tx=0x7fc338006b40).stop 2023-11-17T05:28:07.948 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:07.946+0000 7fc34359e700 1 -- 172.21.15.160:0/2159906318 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3440601e0 msgr2=0x7fc344106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:28:07.949 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:07.946+0000 7fc34359e700 1 --2- 172.21.15.160:0/2159906318 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3440601e0 0x7fc344106500 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fc334013510 tx=0x7fc33400b040).stop 2023-11-17T05:28:07.949 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:07.946+0000 7fc34359e700 1 -- 172.21.15.160:0/2159906318 shutdown_connections 2023-11-17T05:28:07.950 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:07.946+0000 7fc34359e700 1 --2- 172.21.15.160:0/2159906318 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc32c02ff60 0x7fc32c032420 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:28:07.950 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:07.946+0000 7fc34359e700 1 --2- 172.21.15.160:0/2159906318 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3440601e0 0x7fc344106500 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:28:07.950 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:07.946+0000 7fc34359e700 1 -- 172.21.15.160:0/2159906318 >> 172.21.15.160:0/2159906318 conn(0x7fc3440f09a0 msgr2=0x7fc3440f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:28:07.951 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:07.946+0000 7fc34359e700 1 -- 172.21.15.160:0/2159906318 shutdown_connections 2023-11-17T05:28:07.952 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:07.946+0000 7fc34359e700 1 -- 172.21.15.160:0/2159906318 wait complete. 2023-11-17T05:28:07.952 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:28:08.773 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:08 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:28:07.414341+0000 mgr.smithi003.eddcuc (mgr.14164) 81 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:28:08.774 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:08 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:28:07.945590+0000 mon.smithi003 (mon.0) 142 : audit [DBG] from='client.? 172.21.15.160:0/2159906318' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:28:09.346 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:28:09.346 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:28:10.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:09 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:28:09.414695+0000 mgr.smithi003.eddcuc (mgr.14164) 82 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:28:10.563 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:10.560+0000 7f27168e7700 1 -- 172.21.15.160:0/1884210527 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27100f5410 msgr2=0x7f27100f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:28:10.564 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:10.560+0000 7f27168e7700 1 --2- 172.21.15.160:0/1884210527 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27100f5410 0x7f27100f5830 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f2700009fb0 tx=0x7f2700009b30).stop 2023-11-17T05:28:10.564 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:10.560+0000 7f27168e7700 1 -- 172.21.15.160:0/1884210527 shutdown_connections 2023-11-17T05:28:10.564 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:10.560+0000 7f27168e7700 1 --2- 172.21.15.160:0/1884210527 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27100f5410 0x7f27100f5830 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:28:10.564 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:10.560+0000 7f27168e7700 1 -- 172.21.15.160:0/1884210527 >> 172.21.15.160:0/1884210527 conn(0x7f27100f0b70 msgr2=0x7f27100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:28:10.564 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:10.560+0000 7f27168e7700 1 -- 172.21.15.160:0/1884210527 shutdown_connections 2023-11-17T05:28:10.564 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:10.560+0000 7f27168e7700 1 -- 172.21.15.160:0/1884210527 wait complete. 2023-11-17T05:28:10.564 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:10.561+0000 7f27168e7700 1 Processor -- start 2023-11-17T05:28:10.564 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:10.561+0000 7f27168e7700 1 -- start start 2023-11-17T05:28:10.564 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:10.561+0000 7f27168e7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27100f5410 0x7f27100fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:28:10.564 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:10.561+0000 7f27168e7700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27100fb820 con 0x7f27100f5410 2023-11-17T05:28:10.564 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:10.562+0000 7f270ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27100f5410 0x7f27100fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:28:10.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:10.562+0000 7f270ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27100f5410 0x7f27100fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:39716/0 (socket says 172.21.15.160:39716) 2023-11-17T05:28:10.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:10.562+0000 7f270ffff700 1 -- 172.21.15.160:0/3872936124 learned_addr learned my addr 172.21.15.160:0/3872936124 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:28:10.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:10.562+0000 7f270ffff700 1 -- 172.21.15.160:0/3872936124 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2700005040 con 0x7f27100f5410 2023-11-17T05:28:10.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:10.562+0000 7f270ffff700 1 --2- 172.21.15.160:0/3872936124 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27100f5410 0x7f27100fd3a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f2700013510 tx=0x7f270000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:28:10.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:10.562+0000 7f270cff9700 1 -- 172.21.15.160:0/3872936124 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2700009320 con 0x7f27100f5410 2023-11-17T05:28:10.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:10.562+0000 7f27168e7700 1 -- 172.21.15.160:0/3872936124 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27100fb9c0 con 0x7f27100f5410 2023-11-17T05:28:10.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:10.562+0000 7f270cff9700 1 -- 172.21.15.160:0/3872936124 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2700013f00 con 0x7f27100f5410 2023-11-17T05:28:10.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:10.562+0000 7f270cff9700 1 -- 172.21.15.160:0/3872936124 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2700014d90 con 0x7f27100f5410 2023-11-17T05:28:10.565 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:10.562+0000 7f27168e7700 1 -- 172.21.15.160:0/3872936124 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27100fbe50 con 0x7f27100f5410 2023-11-17T05:28:10.566 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:10.563+0000 7f270cff9700 1 -- 172.21.15.160:0/3872936124 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f27000263a0 con 0x7f27100f5410 2023-11-17T05:28:10.566 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:10.563+0000 7f27168e7700 1 -- 172.21.15.160:0/3872936124 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2710059100 con 0x7f27100f5410 2023-11-17T05:28:10.566 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:10.563+0000 7f270cff9700 1 --2- 172.21.15.160:0/3872936124 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f26f802ff60 0x7f26f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:28:10.566 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:10.563+0000 7f270cff9700 1 -- 172.21.15.160:0/3872936124 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f27000413a0 con 0x7f27100f5410 2023-11-17T05:28:10.566 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:10.564+0000 7f270f7fe700 1 --2- 172.21.15.160:0/3872936124 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f26f802ff60 0x7f26f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:28:10.566 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:10.564+0000 7f270f7fe700 1 --2- 172.21.15.160:0/3872936124 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f26f802ff60 0x7f26f8032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f27040096a0 tx=0x7f2704006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:28:10.569 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:10.569+0000 7f270cff9700 1 -- 172.21.15.160:0/3872936124 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f27000126f0 con 0x7f27100f5410 2023-11-17T05:28:10.776 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:10.775+0000 7f27168e7700 1 -- 172.21.15.160:0/3872936124 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2710045ad0 con 0x7f27100f5410 2023-11-17T05:28:10.776 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:10.776+0000 7f270cff9700 1 -- 172.21.15.160:0/3872936124 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f270000f340 con 0x7f27100f5410 2023-11-17T05:28:10.777 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:28:10.777 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:28:10.778 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:10.777+0000 7f27168e7700 1 -- 172.21.15.160:0/3872936124 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f26f802ff60 msgr2=0x7f26f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:28:10.778 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:10.777+0000 7f27168e7700 1 --2- 172.21.15.160:0/3872936124 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f26f802ff60 0x7f26f8032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f27040096a0 tx=0x7f2704006b40).stop 2023-11-17T05:28:10.778 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:10.777+0000 7f27168e7700 1 -- 172.21.15.160:0/3872936124 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27100f5410 msgr2=0x7f27100fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:28:10.778 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:10.777+0000 7f27168e7700 1 --2- 172.21.15.160:0/3872936124 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27100f5410 0x7f27100fd3a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f2700013510 tx=0x7f270000b040).stop 2023-11-17T05:28:10.778 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:10.777+0000 7f27168e7700 1 -- 172.21.15.160:0/3872936124 shutdown_connections 2023-11-17T05:28:10.779 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:10.777+0000 7f27168e7700 1 --2- 172.21.15.160:0/3872936124 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f26f802ff60 0x7f26f8032420 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:28:10.779 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:10.777+0000 7f27168e7700 1 --2- 172.21.15.160:0/3872936124 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27100f5410 0x7f27100fd3a0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:28:10.779 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:10.777+0000 7f27168e7700 1 -- 172.21.15.160:0/3872936124 >> 172.21.15.160:0/3872936124 conn(0x7f27100f0b70 msgr2=0x7f27100f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:28:10.779 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:10.777+0000 7f27168e7700 1 -- 172.21.15.160:0/3872936124 shutdown_connections 2023-11-17T05:28:10.779 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:10.777+0000 7f27168e7700 1 -- 172.21.15.160:0/3872936124 wait complete. 2023-11-17T05:28:10.779 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:28:11.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:10 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:28:10.776703+0000 mon.smithi003 (mon.0) 143 : audit [DBG] from='client.? 172.21.15.160:0/3872936124' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:28:12.223 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:28:12.224 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:28:12.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:11 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:28:11.414989+0000 mgr.smithi003.eddcuc (mgr.14164) 83 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:28:13.357 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:13.352+0000 7fcfdbfff700 1 -- 172.21.15.160:0/417264425 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfdc0f62a0 msgr2=0x7fcfdc0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:28:13.357 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:13.352+0000 7fcfdbfff700 1 --2- 172.21.15.160:0/417264425 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfdc0f62a0 0x7fcfdc0f66c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fcfd0009fb0 tx=0x7fcfd0009b30).stop 2023-11-17T05:28:13.358 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:13.352+0000 7fcfdbfff700 1 -- 172.21.15.160:0/417264425 shutdown_connections 2023-11-17T05:28:13.358 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:13.352+0000 7fcfdbfff700 1 --2- 172.21.15.160:0/417264425 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfdc0f62a0 0x7fcfdc0f66c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:28:13.358 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:13.352+0000 7fcfdbfff700 1 -- 172.21.15.160:0/417264425 >> 172.21.15.160:0/417264425 conn(0x7fcfdc0f1aa0 msgr2=0x7fcfdc0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:28:13.358 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:13.352+0000 7fcfdbfff700 1 -- 172.21.15.160:0/417264425 shutdown_connections 2023-11-17T05:28:13.358 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:13.352+0000 7fcfdbfff700 1 -- 172.21.15.160:0/417264425 wait complete. 2023-11-17T05:28:13.359 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:13.353+0000 7fcfdbfff700 1 Processor -- start 2023-11-17T05:28:13.359 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:13.353+0000 7fcfdbfff700 1 -- start start 2023-11-17T05:28:13.359 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:13.353+0000 7fcfdbfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfdc0f62a0 0x7fcfdc1064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:28:13.359 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:13.353+0000 7fcfdbfff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfdc1069a0 con 0x7fcfdc0f62a0 2023-11-17T05:28:13.359 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:13.354+0000 7fcfdaffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfdc0f62a0 0x7fcfdc1064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:28:13.359 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:13.354+0000 7fcfdaffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfdc0f62a0 0x7fcfdc1064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:39734/0 (socket says 172.21.15.160:39734) 2023-11-17T05:28:13.360 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:13.354+0000 7fcfdaffd700 1 -- 172.21.15.160:0/3156330865 learned_addr learned my addr 172.21.15.160:0/3156330865 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:28:13.360 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:13.354+0000 7fcfdaffd700 1 -- 172.21.15.160:0/3156330865 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd0005040 con 0x7fcfdc0f62a0 2023-11-17T05:28:13.360 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:13.354+0000 7fcfdaffd700 1 --2- 172.21.15.160:0/3156330865 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfdc0f62a0 0x7fcfdc1064b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fcfd0009830 tx=0x7fcfd000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:28:13.360 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:13.354+0000 7fcfc37fe700 1 -- 172.21.15.160:0/3156330865 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfd0009320 con 0x7fcfdc0f62a0 2023-11-17T05:28:13.360 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:13.354+0000 7fcfdbfff700 1 -- 172.21.15.160:0/3156330865 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfdc106ba0 con 0x7fcfdc0f62a0 2023-11-17T05:28:13.360 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:13.354+0000 7fcfc37fe700 1 -- 172.21.15.160:0/3156330865 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcfd00136c0 con 0x7fcfdc0f62a0 2023-11-17T05:28:13.360 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:13.354+0000 7fcfc37fe700 1 -- 172.21.15.160:0/3156330865 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfd001ee90 con 0x7fcfdc0f62a0 2023-11-17T05:28:13.361 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:13.354+0000 7fcfdbfff700 1 -- 172.21.15.160:0/3156330865 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfdc107030 con 0x7fcfdc0f62a0 2023-11-17T05:28:13.361 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:13.355+0000 7fcfc37fe700 1 -- 172.21.15.160:0/3156330865 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcfd001e420 con 0x7fcfdc0f62a0 2023-11-17T05:28:13.361 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:13.355+0000 7fcfdbfff700 1 -- 172.21.15.160:0/3156330865 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfdc045ad0 con 0x7fcfdc0f62a0 2023-11-17T05:28:13.361 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:13.355+0000 7fcfc37fe700 1 --2- 172.21.15.160:0/3156330865 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fcfc402ff10 0x7fcfc40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:28:13.361 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:13.355+0000 7fcfc37fe700 1 -- 172.21.15.160:0/3156330865 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcfd0040f30 con 0x7fcfdc0f62a0 2023-11-17T05:28:13.361 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:13.356+0000 7fcfda7fc700 1 --2- 172.21.15.160:0/3156330865 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fcfc402ff10 0x7fcfc40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:28:13.361 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:13.356+0000 7fcfda7fc700 1 --2- 172.21.15.160:0/3156330865 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fcfc402ff10 0x7fcfc40323d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fcfcc0096a0 tx=0x7fcfcc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:28:13.362 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:13.360+0000 7fcfc37fe700 1 -- 172.21.15.160:0/3156330865 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcfd006d2b0 con 0x7fcfdc0f62a0 2023-11-17T05:28:13.568 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:13.568+0000 7fcfdbfff700 1 -- 172.21.15.160:0/3156330865 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcfdc107ee0 con 0x7fcfdc0f62a0 2023-11-17T05:28:13.569 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:13.568+0000 7fcfc37fe700 1 -- 172.21.15.160:0/3156330865 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fcfd001c790 con 0x7fcfdc0f62a0 2023-11-17T05:28:13.570 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:28:13.570 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:28:13.571 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:13.570+0000 7fcfdbfff700 1 -- 172.21.15.160:0/3156330865 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fcfc402ff10 msgr2=0x7fcfc40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:28:13.571 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:13.570+0000 7fcfdbfff700 1 --2- 172.21.15.160:0/3156330865 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fcfc402ff10 0x7fcfc40323d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fcfcc0096a0 tx=0x7fcfcc006b40).stop 2023-11-17T05:28:13.571 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:13.570+0000 7fcfdbfff700 1 -- 172.21.15.160:0/3156330865 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfdc0f62a0 msgr2=0x7fcfdc1064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:28:13.571 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:13.570+0000 7fcfdbfff700 1 --2- 172.21.15.160:0/3156330865 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfdc0f62a0 0x7fcfdc1064b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fcfd0009830 tx=0x7fcfd000b040).stop 2023-11-17T05:28:13.571 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:13.570+0000 7fcfdbfff700 1 -- 172.21.15.160:0/3156330865 shutdown_connections 2023-11-17T05:28:13.571 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:13.570+0000 7fcfdbfff700 1 --2- 172.21.15.160:0/3156330865 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fcfc402ff10 0x7fcfc40323d0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:28:13.571 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:13.570+0000 7fcfdbfff700 1 --2- 172.21.15.160:0/3156330865 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfdc0f62a0 0x7fcfdc1064b0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:28:13.571 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:13.570+0000 7fcfdbfff700 1 -- 172.21.15.160:0/3156330865 >> 172.21.15.160:0/3156330865 conn(0x7fcfdc0f1aa0 msgr2=0x7fcfdc0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:28:13.571 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:13.570+0000 7fcfdbfff700 1 -- 172.21.15.160:0/3156330865 shutdown_connections 2023-11-17T05:28:13.572 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:13.570+0000 7fcfdbfff700 1 -- 172.21.15.160:0/3156330865 wait complete. 2023-11-17T05:28:13.572 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:28:14.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:14 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:28:13.415390+0000 mgr.smithi003.eddcuc (mgr.14164) 84 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:28:14.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:14 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:28:13.569590+0000 mon.smithi003 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.160:0/3156330865' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:28:14.983 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:28:14.983 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:28:16.181 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:16.179+0000 7fd32d5de700 1 -- 172.21.15.160:0/1327827492 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3280f47c0 msgr2=0x7fd3280f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:28:16.181 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:16.179+0000 7fd32d5de700 1 --2- 172.21.15.160:0/1327827492 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3280f47c0 0x7fd3280f4be0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fd31c009fb0 tx=0x7fd31c009b30).stop 2023-11-17T05:28:16.181 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:16.179+0000 7fd32d5de700 1 -- 172.21.15.160:0/1327827492 shutdown_connections 2023-11-17T05:28:16.181 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:16.179+0000 7fd32d5de700 1 --2- 172.21.15.160:0/1327827492 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3280f47c0 0x7fd3280f4be0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:28:16.182 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:16.179+0000 7fd32d5de700 1 -- 172.21.15.160:0/1327827492 >> 172.21.15.160:0/1327827492 conn(0x7fd3280f02e0 msgr2=0x7fd3280f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:28:16.182 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:16.179+0000 7fd32d5de700 1 -- 172.21.15.160:0/1327827492 shutdown_connections 2023-11-17T05:28:16.182 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:16.179+0000 7fd32d5de700 1 -- 172.21.15.160:0/1327827492 wait complete. 2023-11-17T05:28:16.182 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:16.180+0000 7fd32d5de700 1 Processor -- start 2023-11-17T05:28:16.182 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:16.180+0000 7fd32d5de700 1 -- start start 2023-11-17T05:28:16.182 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:16.180+0000 7fd32d5de700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3280f47c0 0x7fd328106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:28:16.182 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:16.180+0000 7fd32d5de700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd328106bf0 con 0x7fd3280f47c0 2023-11-17T05:28:16.182 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:16.181+0000 7fd326ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3280f47c0 0x7fd328106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:28:16.182 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:16.181+0000 7fd326ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3280f47c0 0x7fd328106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:39758/0 (socket says 172.21.15.160:39758) 2023-11-17T05:28:16.182 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:16.181+0000 7fd326ffd700 1 -- 172.21.15.160:0/1385700200 learned_addr learned my addr 172.21.15.160:0/1385700200 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:28:16.185 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:16.181+0000 7fd326ffd700 1 -- 172.21.15.160:0/1385700200 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd31c005040 con 0x7fd3280f47c0 2023-11-17T05:28:16.185 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:16.181+0000 7fd326ffd700 1 --2- 172.21.15.160:0/1385700200 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3280f47c0 0x7fd328106700 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fd31c013510 tx=0x7fd31c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:28:16.185 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:16.181+0000 7fd30f7fe700 1 -- 172.21.15.160:0/1385700200 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd31c009320 con 0x7fd3280f47c0 2023-11-17T05:28:16.185 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:16.181+0000 7fd32d5de700 1 -- 172.21.15.160:0/1385700200 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd328106df0 con 0x7fd3280f47c0 2023-11-17T05:28:16.185 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:16.181+0000 7fd30f7fe700 1 -- 172.21.15.160:0/1385700200 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd31c013f00 con 0x7fd3280f47c0 2023-11-17T05:28:16.185 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:16.181+0000 7fd30f7fe700 1 -- 172.21.15.160:0/1385700200 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd31c014d90 con 0x7fd3280f47c0 2023-11-17T05:28:16.185 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:16.181+0000 7fd32d5de700 1 -- 172.21.15.160:0/1385700200 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd328107280 con 0x7fd3280f47c0 2023-11-17T05:28:16.185 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:16.182+0000 7fd30f7fe700 1 -- 172.21.15.160:0/1385700200 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd31c0095f0 con 0x7fd3280f47c0 2023-11-17T05:28:16.185 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:16.182+0000 7fd32d5de700 1 -- 172.21.15.160:0/1385700200 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd328045ad0 con 0x7fd3280f47c0 2023-11-17T05:28:16.185 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:16.182+0000 7fd30f7fe700 1 --2- 172.21.15.160:0/1385700200 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fd31002ffb0 0x7fd310032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:28:16.186 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:16.182+0000 7fd30f7fe700 1 -- 172.21.15.160:0/1385700200 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd31c0410a0 con 0x7fd3280f47c0 2023-11-17T05:28:16.186 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:16.182+0000 7fd3267fc700 1 --2- 172.21.15.160:0/1385700200 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fd31002ffb0 0x7fd310032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:28:16.187 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:16.183+0000 7fd3267fc700 1 --2- 172.21.15.160:0/1385700200 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fd31002ffb0 0x7fd310032470 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fd3180096a0 tx=0x7fd318006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:28:16.188 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:16.187+0000 7fd30f7fe700 1 -- 172.21.15.160:0/1385700200 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd31c06d2b0 con 0x7fd3280f47c0 2023-11-17T05:28:16.394 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:16.393+0000 7fd32d5de700 1 -- 172.21.15.160:0/1385700200 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd328108130 con 0x7fd3280f47c0 2023-11-17T05:28:16.395 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:16.394+0000 7fd30f7fe700 1 -- 172.21.15.160:0/1385700200 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fd31c01d7f0 con 0x7fd3280f47c0 2023-11-17T05:28:16.395 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:28:16.396 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:28:16.397 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:16.396+0000 7fd32d5de700 1 -- 172.21.15.160:0/1385700200 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fd31002ffb0 msgr2=0x7fd310032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:28:16.397 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:16.396+0000 7fd32d5de700 1 --2- 172.21.15.160:0/1385700200 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fd31002ffb0 0x7fd310032470 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fd3180096a0 tx=0x7fd318006b40).stop 2023-11-17T05:28:16.397 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:16.396+0000 7fd32d5de700 1 -- 172.21.15.160:0/1385700200 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3280f47c0 msgr2=0x7fd328106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:28:16.397 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:16.396+0000 7fd32d5de700 1 --2- 172.21.15.160:0/1385700200 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3280f47c0 0x7fd328106700 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fd31c013510 tx=0x7fd31c00b040).stop 2023-11-17T05:28:16.397 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:16.396+0000 7fd32d5de700 1 -- 172.21.15.160:0/1385700200 shutdown_connections 2023-11-17T05:28:16.397 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:16.396+0000 7fd32d5de700 1 --2- 172.21.15.160:0/1385700200 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fd31002ffb0 0x7fd310032470 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:28:16.398 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:16.396+0000 7fd32d5de700 1 --2- 172.21.15.160:0/1385700200 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd3280f47c0 0x7fd328106700 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:28:16.398 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:16.396+0000 7fd32d5de700 1 -- 172.21.15.160:0/1385700200 >> 172.21.15.160:0/1385700200 conn(0x7fd3280f02e0 msgr2=0x7fd3280f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:28:16.398 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:16.396+0000 7fd32d5de700 1 -- 172.21.15.160:0/1385700200 shutdown_connections 2023-11-17T05:28:16.398 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:16.396+0000 7fd32d5de700 1 -- 172.21.15.160:0/1385700200 wait complete. 2023-11-17T05:28:16.398 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:28:16.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:16 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:28:15.415693+0000 mgr.smithi003.eddcuc (mgr.14164) 85 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:28:16.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:16 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:28:16.395524+0000 mon.smithi003 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.160:0/1385700200' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:28:17.822 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:28:17.825 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:28:18.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:18 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:28:17.416013+0000 mgr.smithi003.eddcuc (mgr.14164) 86 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:28:18.991 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:18.988+0000 7f0fcecd9700 1 -- 172.21.15.160:0/2745416712 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fc80f62a0 msgr2=0x7f0fc80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:28:18.992 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:18.988+0000 7f0fcecd9700 1 --2- 172.21.15.160:0/2745416712 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fc80f62a0 0x7f0fc80f66c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f0fc4009fb0 tx=0x7f0fc4009b30).stop 2023-11-17T05:28:18.992 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:18.988+0000 7f0fcecd9700 1 -- 172.21.15.160:0/2745416712 shutdown_connections 2023-11-17T05:28:18.992 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:18.988+0000 7f0fcecd9700 1 --2- 172.21.15.160:0/2745416712 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fc80f62a0 0x7f0fc80f66c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:28:18.992 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:18.988+0000 7f0fcecd9700 1 -- 172.21.15.160:0/2745416712 >> 172.21.15.160:0/2745416712 conn(0x7f0fc80f1aa0 msgr2=0x7f0fc80f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:28:18.992 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:18.988+0000 7f0fcecd9700 1 -- 172.21.15.160:0/2745416712 shutdown_connections 2023-11-17T05:28:18.992 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:18.988+0000 7f0fcecd9700 1 -- 172.21.15.160:0/2745416712 wait complete. 2023-11-17T05:28:18.992 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:18.988+0000 7f0fcecd9700 1 Processor -- start 2023-11-17T05:28:18.992 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:18.988+0000 7f0fcecd9700 1 -- start start 2023-11-17T05:28:18.992 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:18.988+0000 7f0fcecd9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fc80f62a0 0x7f0fc81064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:28:18.992 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:18.988+0000 7f0fcecd9700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc81069a0 con 0x7f0fc80f62a0 2023-11-17T05:28:18.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:18.989+0000 7f0fcdcd7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fc80f62a0 0x7f0fc81064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:28:18.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:18.989+0000 7f0fcdcd7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fc80f62a0 0x7f0fc81064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:39764/0 (socket says 172.21.15.160:39764) 2023-11-17T05:28:18.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:18.989+0000 7f0fcdcd7700 1 -- 172.21.15.160:0/4216740436 learned_addr learned my addr 172.21.15.160:0/4216740436 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:28:18.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:18.989+0000 7f0fcdcd7700 1 -- 172.21.15.160:0/4216740436 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc4005040 con 0x7f0fc80f62a0 2023-11-17T05:28:18.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:18.989+0000 7f0fcdcd7700 1 --2- 172.21.15.160:0/4216740436 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fc80f62a0 0x7f0fc81064b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f0fc4009830 tx=0x7f0fc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:28:18.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:18.989+0000 7f0fbe7fc700 1 -- 172.21.15.160:0/4216740436 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fc4009320 con 0x7f0fc80f62a0 2023-11-17T05:28:18.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:18.989+0000 7f0fcecd9700 1 -- 172.21.15.160:0/4216740436 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fc8106ba0 con 0x7f0fc80f62a0 2023-11-17T05:28:19.000 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:18.989+0000 7f0fbe7fc700 1 -- 172.21.15.160:0/4216740436 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0fc40136c0 con 0x7f0fc80f62a0 2023-11-17T05:28:19.000 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:18.989+0000 7f0fbe7fc700 1 -- 172.21.15.160:0/4216740436 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fc401ee90 con 0x7f0fc80f62a0 2023-11-17T05:28:19.000 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:18.989+0000 7f0fcecd9700 1 -- 172.21.15.160:0/4216740436 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fc8107030 con 0x7f0fc80f62a0 2023-11-17T05:28:19.000 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:18.990+0000 7f0fbe7fc700 1 -- 172.21.15.160:0/4216740436 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0fc401e420 con 0x7f0fc80f62a0 2023-11-17T05:28:19.000 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:18.990+0000 7f0fcecd9700 1 -- 172.21.15.160:0/4216740436 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fc8045ad0 con 0x7f0fc80f62a0 2023-11-17T05:28:19.000 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:18.990+0000 7f0fbe7fc700 1 --2- 172.21.15.160:0/4216740436 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f0fb402ff10 0x7f0fb40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:28:19.000 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:18.990+0000 7f0fbe7fc700 1 -- 172.21.15.160:0/4216740436 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0fc4040fc0 con 0x7f0fc80f62a0 2023-11-17T05:28:19.000 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:18.990+0000 7f0fcd4d6700 1 --2- 172.21.15.160:0/4216740436 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f0fb402ff10 0x7f0fb40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:28:19.000 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:18.991+0000 7f0fcd4d6700 1 --2- 172.21.15.160:0/4216740436 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f0fb402ff10 0x7f0fb40323d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f0fb80096a0 tx=0x7f0fb8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:28:19.001 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:18.995+0000 7f0fbe7fc700 1 -- 172.21.15.160:0/4216740436 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0fc40095f0 con 0x7f0fc80f62a0 2023-11-17T05:28:19.203 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:19.202+0000 7f0fcecd9700 1 -- 172.21.15.160:0/4216740436 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0fc8107ee0 con 0x7f0fc80f62a0 2023-11-17T05:28:19.204 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:19.203+0000 7f0fbe7fc700 1 -- 172.21.15.160:0/4216740436 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f0fc402c2c0 con 0x7f0fc80f62a0 2023-11-17T05:28:19.204 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:28:19.204 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:28:19.207 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:19.204+0000 7f0fcecd9700 1 -- 172.21.15.160:0/4216740436 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f0fb402ff10 msgr2=0x7f0fb40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:28:19.207 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:19.204+0000 7f0fcecd9700 1 --2- 172.21.15.160:0/4216740436 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f0fb402ff10 0x7f0fb40323d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f0fb80096a0 tx=0x7f0fb8006b40).stop 2023-11-17T05:28:19.207 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:19.204+0000 7f0fcecd9700 1 -- 172.21.15.160:0/4216740436 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fc80f62a0 msgr2=0x7f0fc81064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:28:19.207 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:19.204+0000 7f0fcecd9700 1 --2- 172.21.15.160:0/4216740436 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fc80f62a0 0x7f0fc81064b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f0fc4009830 tx=0x7f0fc400b040).stop 2023-11-17T05:28:19.207 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:19.204+0000 7f0fcecd9700 1 -- 172.21.15.160:0/4216740436 shutdown_connections 2023-11-17T05:28:19.208 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:19.204+0000 7f0fcecd9700 1 --2- 172.21.15.160:0/4216740436 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f0fb402ff10 0x7f0fb40323d0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:28:19.208 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:19.204+0000 7f0fcecd9700 1 --2- 172.21.15.160:0/4216740436 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fc80f62a0 0x7f0fc81064b0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:28:19.208 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:19.204+0000 7f0fcecd9700 1 -- 172.21.15.160:0/4216740436 >> 172.21.15.160:0/4216740436 conn(0x7f0fc80f1aa0 msgr2=0x7f0fc80f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:28:19.208 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:19.204+0000 7f0fcecd9700 1 -- 172.21.15.160:0/4216740436 shutdown_connections 2023-11-17T05:28:19.208 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:19.204+0000 7f0fcecd9700 1 -- 172.21.15.160:0/4216740436 wait complete. 2023-11-17T05:28:19.208 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:28:20.275 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:19 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:28:19.203988+0000 mon.smithi003 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.160:0/4216740436' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:28:20.275 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:19 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:28:19.416317+0000 mgr.smithi003.eddcuc (mgr.14164) 87 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:28:20.620 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:28:20.621 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:28:21.792 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:21.789+0000 7f47afbf9700 1 -- 172.21.15.160:0/279825273 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47a80f6510 msgr2=0x7f47a80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:28:21.792 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:21.789+0000 7f47afbf9700 1 --2- 172.21.15.160:0/279825273 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47a80f6510 0x7f47a80f6930 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f4798009fb0 tx=0x7f4798009b30).stop 2023-11-17T05:28:21.793 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:21.789+0000 7f47afbf9700 1 -- 172.21.15.160:0/279825273 shutdown_connections 2023-11-17T05:28:21.793 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:21.789+0000 7f47afbf9700 1 --2- 172.21.15.160:0/279825273 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47a80f6510 0x7f47a80f6930 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:28:21.793 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:21.789+0000 7f47afbf9700 1 -- 172.21.15.160:0/279825273 >> 172.21.15.160:0/279825273 conn(0x7f47a80f1c70 msgr2=0x7f47a80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:28:21.793 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:21.789+0000 7f47afbf9700 1 -- 172.21.15.160:0/279825273 shutdown_connections 2023-11-17T05:28:21.793 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:21.789+0000 7f47afbf9700 1 -- 172.21.15.160:0/279825273 wait complete. 2023-11-17T05:28:21.793 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:21.790+0000 7f47afbf9700 1 Processor -- start 2023-11-17T05:28:21.793 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:21.790+0000 7f47afbf9700 1 -- start start 2023-11-17T05:28:21.793 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:21.790+0000 7f47afbf9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47a80f6510 0x7f47a8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:28:21.793 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:21.790+0000 7f47afbf9700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a8106c20 con 0x7f47a80f6510 2023-11-17T05:28:21.793 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:21.791+0000 7f47ad995700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47a80f6510 0x7f47a8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:28:21.793 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:21.791+0000 7f47ad995700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47a80f6510 0x7f47a8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:38272/0 (socket says 172.21.15.160:38272) 2023-11-17T05:28:21.794 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:21.791+0000 7f47ad995700 1 -- 172.21.15.160:0/2211577877 learned_addr learned my addr 172.21.15.160:0/2211577877 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:28:21.794 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:21.791+0000 7f47ad995700 1 -- 172.21.15.160:0/2211577877 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4798005040 con 0x7f47a80f6510 2023-11-17T05:28:21.794 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:21.791+0000 7f47ad995700 1 --2- 172.21.15.160:0/2211577877 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47a80f6510 0x7f47a8106730 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f4798014000 tx=0x7f479800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:28:21.794 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:21.791+0000 7f479e7fc700 1 -- 172.21.15.160:0/2211577877 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47980095f0 con 0x7f47a80f6510 2023-11-17T05:28:21.794 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:21.791+0000 7f479e7fc700 1 -- 172.21.15.160:0/2211577877 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4798007e70 con 0x7f47a80f6510 2023-11-17T05:28:21.794 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:21.791+0000 7f479e7fc700 1 -- 172.21.15.160:0/2211577877 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4798014ea0 con 0x7f47a80f6510 2023-11-17T05:28:21.794 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:21.791+0000 7f47afbf9700 1 -- 172.21.15.160:0/2211577877 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47a8106e20 con 0x7f47a80f6510 2023-11-17T05:28:21.794 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:21.791+0000 7f47afbf9700 1 -- 172.21.15.160:0/2211577877 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47a81072b0 con 0x7f47a80f6510 2023-11-17T05:28:21.795 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:21.792+0000 7f479e7fc700 1 -- 172.21.15.160:0/2211577877 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4798025070 con 0x7f47a80f6510 2023-11-17T05:28:21.795 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:21.792+0000 7f47afbf9700 1 -- 172.21.15.160:0/2211577877 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47a8045ad0 con 0x7f47a80f6510 2023-11-17T05:28:21.795 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:21.792+0000 7f479e7fc700 1 --2- 172.21.15.160:0/2211577877 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f479402ff60 0x7f4794032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:28:21.795 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:21.792+0000 7f479e7fc700 1 -- 172.21.15.160:0/2211577877 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4798040a00 con 0x7f47a80f6510 2023-11-17T05:28:21.795 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:21.793+0000 7f47ad194700 1 --2- 172.21.15.160:0/2211577877 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f479402ff60 0x7f4794032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:28:21.796 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:21.793+0000 7f47ad194700 1 --2- 172.21.15.160:0/2211577877 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f479402ff60 0x7f4794032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f47a40096a0 tx=0x7f47a4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:28:21.798 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:21.797+0000 7f479e7fc700 1 -- 172.21.15.160:0/2211577877 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f47980079c0 con 0x7f47a80f6510 2023-11-17T05:28:22.004 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:22.003+0000 7f47afbf9700 1 -- 172.21.15.160:0/2211577877 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f47a80fbf60 con 0x7f47a80f6510 2023-11-17T05:28:22.004 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:22.004+0000 7f479e7fc700 1 -- 172.21.15.160:0/2211577877 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f479800f300 con 0x7f47a80f6510 2023-11-17T05:28:22.005 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:28:22.005 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:28:22.006 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:22.005+0000 7f47afbf9700 1 -- 172.21.15.160:0/2211577877 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f479402ff60 msgr2=0x7f4794032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:28:22.007 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:22.005+0000 7f47afbf9700 1 --2- 172.21.15.160:0/2211577877 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f479402ff60 0x7f4794032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f47a40096a0 tx=0x7f47a4006b40).stop 2023-11-17T05:28:22.007 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:22.005+0000 7f47afbf9700 1 -- 172.21.15.160:0/2211577877 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47a80f6510 msgr2=0x7f47a8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:28:22.007 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:22.005+0000 7f47afbf9700 1 --2- 172.21.15.160:0/2211577877 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47a80f6510 0x7f47a8106730 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f4798014000 tx=0x7f479800b040).stop 2023-11-17T05:28:22.007 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:22.005+0000 7f47afbf9700 1 -- 172.21.15.160:0/2211577877 shutdown_connections 2023-11-17T05:28:22.007 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:22.005+0000 7f47afbf9700 1 --2- 172.21.15.160:0/2211577877 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f479402ff60 0x7f4794032420 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:28:22.007 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:22.005+0000 7f47afbf9700 1 --2- 172.21.15.160:0/2211577877 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47a80f6510 0x7f47a8106730 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:28:22.007 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:22.006+0000 7f47afbf9700 1 -- 172.21.15.160:0/2211577877 >> 172.21.15.160:0/2211577877 conn(0x7f47a80f1c70 msgr2=0x7f47a80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:28:22.007 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:22.006+0000 7f47afbf9700 1 -- 172.21.15.160:0/2211577877 shutdown_connections 2023-11-17T05:28:22.007 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:22.006+0000 7f47afbf9700 1 -- 172.21.15.160:0/2211577877 wait complete. 2023-11-17T05:28:22.008 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:28:22.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:22 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:28:21.416691+0000 mgr.smithi003.eddcuc (mgr.14164) 88 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:28:22.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:22 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:28:22.004924+0000 mon.smithi003 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.160:0/2211577877' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:28:23.428 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:28:23.428 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:28:24.637 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:24.636+0000 7f3599989700 1 -- 172.21.15.160:0/1163990016 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35940f5410 msgr2=0x7f35940f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:28:24.637 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:24.636+0000 7f3599989700 1 --2- 172.21.15.160:0/1163990016 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35940f5410 0x7f35940f5830 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f3588009fb0 tx=0x7f3588009b30).stop 2023-11-17T05:28:24.637 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:24.636+0000 7f3599989700 1 -- 172.21.15.160:0/1163990016 shutdown_connections 2023-11-17T05:28:24.638 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:24.636+0000 7f3599989700 1 --2- 172.21.15.160:0/1163990016 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35940f5410 0x7f35940f5830 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:28:24.638 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:24.636+0000 7f3599989700 1 -- 172.21.15.160:0/1163990016 >> 172.21.15.160:0/1163990016 conn(0x7f35940f0b70 msgr2=0x7f35940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:28:24.638 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:24.636+0000 7f3599989700 1 -- 172.21.15.160:0/1163990016 shutdown_connections 2023-11-17T05:28:24.638 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:24.636+0000 7f3599989700 1 -- 172.21.15.160:0/1163990016 wait complete. 2023-11-17T05:28:24.638 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:24.636+0000 7f3599989700 1 Processor -- start 2023-11-17T05:28:24.638 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:24.636+0000 7f3599989700 1 -- start start 2023-11-17T05:28:24.638 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:24.636+0000 7f3599989700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35940f5410 0x7f3594109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:28:24.638 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:24.636+0000 7f3599989700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3594060240 con 0x7f35940f5410 2023-11-17T05:28:24.639 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:24.637+0000 7f3592ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35940f5410 0x7f3594109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:28:24.639 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:24.637+0000 7f3592ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35940f5410 0x7f3594109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:38282/0 (socket says 172.21.15.160:38282) 2023-11-17T05:28:24.640 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:24.637+0000 7f3592ffd700 1 -- 172.21.15.160:0/2777674052 learned_addr learned my addr 172.21.15.160:0/2777674052 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:28:24.640 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:24.637+0000 7f3592ffd700 1 -- 172.21.15.160:0/2777674052 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3588005040 con 0x7f35940f5410 2023-11-17T05:28:24.640 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:24.637+0000 7f3592ffd700 1 --2- 172.21.15.160:0/2777674052 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35940f5410 0x7f3594109400 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f3588013510 tx=0x7f358800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:28:24.640 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:24.637+0000 7f357b7fe700 1 -- 172.21.15.160:0/2777674052 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3588009320 con 0x7f35940f5410 2023-11-17T05:28:24.640 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:24.637+0000 7f357b7fe700 1 -- 172.21.15.160:0/2777674052 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3588013f00 con 0x7f35940f5410 2023-11-17T05:28:24.640 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:24.637+0000 7f3599989700 1 -- 172.21.15.160:0/2777674052 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3594060440 con 0x7f35940f5410 2023-11-17T05:28:24.640 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:24.638+0000 7f357b7fe700 1 -- 172.21.15.160:0/2777674052 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3588014d90 con 0x7f35940f5410 2023-11-17T05:28:24.640 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:24.638+0000 7f3599989700 1 -- 172.21.15.160:0/2777674052 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3594109ab0 con 0x7f35940f5410 2023-11-17T05:28:24.640 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:24.638+0000 7f357b7fe700 1 -- 172.21.15.160:0/2777674052 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f35880095f0 con 0x7f35940f5410 2023-11-17T05:28:24.640 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:24.638+0000 7f3599989700 1 -- 172.21.15.160:0/2777674052 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3594045ad0 con 0x7f35940f5410 2023-11-17T05:28:24.640 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:24.639+0000 7f357b7fe700 1 --2- 172.21.15.160:0/2777674052 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f357c02ff60 0x7f357c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:28:24.640 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:24.639+0000 7f357b7fe700 1 -- 172.21.15.160:0/2777674052 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3588041000 con 0x7f35940f5410 2023-11-17T05:28:24.641 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:24.639+0000 7f35927fc700 1 --2- 172.21.15.160:0/2777674052 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f357c02ff60 0x7f357c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:28:24.641 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:24.639+0000 7f35927fc700 1 --2- 172.21.15.160:0/2777674052 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f357c02ff60 0x7f357c032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f35840096a0 tx=0x7f3584006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:28:24.644 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:24.644+0000 7f357b7fe700 1 -- 172.21.15.160:0/2777674052 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3588014420 con 0x7f35940f5410 2023-11-17T05:28:24.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:28:23.417061+0000 mgr.smithi003.eddcuc (mgr.14164) 89 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:28:24.848 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:24.847+0000 7f3599989700 1 -- 172.21.15.160:0/2777674052 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f35940fae60 con 0x7f35940f5410 2023-11-17T05:28:24.849 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:24.848+0000 7f357b7fe700 1 -- 172.21.15.160:0/2777674052 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f3588010230 con 0x7f35940f5410 2023-11-17T05:28:24.849 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:28:24.849 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:28:24.851 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:24.850+0000 7f3599989700 1 -- 172.21.15.160:0/2777674052 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f357c02ff60 msgr2=0x7f357c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:28:24.851 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:24.850+0000 7f3599989700 1 --2- 172.21.15.160:0/2777674052 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f357c02ff60 0x7f357c032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f35840096a0 tx=0x7f3584006b40).stop 2023-11-17T05:28:24.851 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:24.850+0000 7f3599989700 1 -- 172.21.15.160:0/2777674052 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35940f5410 msgr2=0x7f3594109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:28:24.851 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:24.850+0000 7f3599989700 1 --2- 172.21.15.160:0/2777674052 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35940f5410 0x7f3594109400 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f3588013510 tx=0x7f358800b040).stop 2023-11-17T05:28:24.851 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:24.850+0000 7f3599989700 1 -- 172.21.15.160:0/2777674052 shutdown_connections 2023-11-17T05:28:24.851 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:24.850+0000 7f3599989700 1 --2- 172.21.15.160:0/2777674052 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f357c02ff60 0x7f357c032420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:28:24.851 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:24.850+0000 7f3599989700 1 --2- 172.21.15.160:0/2777674052 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35940f5410 0x7f3594109400 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:28:24.851 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:24.850+0000 7f3599989700 1 -- 172.21.15.160:0/2777674052 >> 172.21.15.160:0/2777674052 conn(0x7f35940f0b70 msgr2=0x7f35940f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:28:24.885 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:24.850+0000 7f3599989700 1 -- 172.21.15.160:0/2777674052 shutdown_connections 2023-11-17T05:28:24.885 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:24.850+0000 7f3599989700 1 -- 172.21.15.160:0/2777674052 wait complete. 2023-11-17T05:28:24.886 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:28:25.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:25 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:28:24.849409+0000 mon.smithi003 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.160:0/2777674052' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:28:26.241 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:28:26.241 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:28:26.771 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:26 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:28:25.417403+0000 mgr.smithi003.eddcuc (mgr.14164) 90 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:28:27.518 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:27.515+0000 7f16c8cb8700 1 -- 172.21.15.160:0/3546492077 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16c40f5410 msgr2=0x7f16c40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:28:27.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:27.515+0000 7f16c8cb8700 1 --2- 172.21.15.160:0/3546492077 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16c40f5410 0x7f16c40f5830 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f16ac009fb0 tx=0x7f16ac009b30).stop 2023-11-17T05:28:27.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:27.515+0000 7f16c8cb8700 1 -- 172.21.15.160:0/3546492077 shutdown_connections 2023-11-17T05:28:27.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:27.515+0000 7f16c8cb8700 1 --2- 172.21.15.160:0/3546492077 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16c40f5410 0x7f16c40f5830 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:28:27.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:27.515+0000 7f16c8cb8700 1 -- 172.21.15.160:0/3546492077 >> 172.21.15.160:0/3546492077 conn(0x7f16c40f0b70 msgr2=0x7f16c40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:28:27.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:27.516+0000 7f16c8cb8700 1 -- 172.21.15.160:0/3546492077 shutdown_connections 2023-11-17T05:28:27.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:27.516+0000 7f16c8cb8700 1 -- 172.21.15.160:0/3546492077 wait complete. 2023-11-17T05:28:27.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:27.516+0000 7f16c8cb8700 1 Processor -- start 2023-11-17T05:28:27.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:27.516+0000 7f16c8cb8700 1 -- start start 2023-11-17T05:28:27.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:27.516+0000 7f16c8cb8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16c40f5410 0x7f16c41066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:28:27.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:27.516+0000 7f16c8cb8700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c4106bc0 con 0x7f16c40f5410 2023-11-17T05:28:27.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:27.517+0000 7f16c259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16c40f5410 0x7f16c41066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:28:27.520 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:27.517+0000 7f16c259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16c40f5410 0x7f16c41066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:38296/0 (socket says 172.21.15.160:38296) 2023-11-17T05:28:27.520 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:27.517+0000 7f16c259c700 1 -- 172.21.15.160:0/2131392435 learned_addr learned my addr 172.21.15.160:0/2131392435 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:28:27.520 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:27.517+0000 7f16c259c700 1 -- 172.21.15.160:0/2131392435 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16ac005040 con 0x7f16c40f5410 2023-11-17T05:28:27.520 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:27.517+0000 7f16c259c700 1 --2- 172.21.15.160:0/2131392435 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16c40f5410 0x7f16c41066d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f16ac013510 tx=0x7f16ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:28:27.520 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:27.517+0000 7f16baffd700 1 -- 172.21.15.160:0/2131392435 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16ac009320 con 0x7f16c40f5410 2023-11-17T05:28:27.520 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:27.517+0000 7f16c8cb8700 1 -- 172.21.15.160:0/2131392435 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16c4106dc0 con 0x7f16c40f5410 2023-11-17T05:28:27.520 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:27.517+0000 7f16baffd700 1 -- 172.21.15.160:0/2131392435 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f16ac013f00 con 0x7f16c40f5410 2023-11-17T05:28:27.520 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:27.517+0000 7f16baffd700 1 -- 172.21.15.160:0/2131392435 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16ac014d90 con 0x7f16c40f5410 2023-11-17T05:28:27.520 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:27.517+0000 7f16c8cb8700 1 -- 172.21.15.160:0/2131392435 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16c4107250 con 0x7f16c40f5410 2023-11-17T05:28:27.521 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:27.518+0000 7f16baffd700 1 -- 172.21.15.160:0/2131392435 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f16ac0095f0 con 0x7f16c40f5410 2023-11-17T05:28:27.521 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:27.518+0000 7f16c8cb8700 1 -- 172.21.15.160:0/2131392435 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16c4045ad0 con 0x7f16c40f5410 2023-11-17T05:28:27.521 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:27.518+0000 7f16baffd700 1 --2- 172.21.15.160:0/2131392435 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f16b002ff60 0x7f16b0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:28:27.521 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:27.518+0000 7f16baffd700 1 -- 172.21.15.160:0/2131392435 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f16ac040fe0 con 0x7f16c40f5410 2023-11-17T05:28:27.521 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:27.518+0000 7f16c1d9b700 1 --2- 172.21.15.160:0/2131392435 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f16b002ff60 0x7f16b0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:28:27.521 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:27.519+0000 7f16c1d9b700 1 --2- 172.21.15.160:0/2131392435 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f16b002ff60 0x7f16b0032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f16b40096a0 tx=0x7f16b4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:28:27.523 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:27.523+0000 7f16baffd700 1 -- 172.21.15.160:0/2131392435 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f16ac01d7f0 con 0x7f16c40f5410 2023-11-17T05:28:27.728 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:27.727+0000 7f16c8cb8700 1 -- 172.21.15.160:0/2131392435 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f16c4108100 con 0x7f16c40f5410 2023-11-17T05:28:27.728 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:27.728+0000 7f16baffd700 1 -- 172.21.15.160:0/2131392435 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f16ac010230 con 0x7f16c40f5410 2023-11-17T05:28:27.729 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:28:27.729 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:28:27.730 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:27.729+0000 7f16c8cb8700 1 -- 172.21.15.160:0/2131392435 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f16b002ff60 msgr2=0x7f16b0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:28:27.730 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:27.729+0000 7f16c8cb8700 1 --2- 172.21.15.160:0/2131392435 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f16b002ff60 0x7f16b0032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f16b40096a0 tx=0x7f16b4006b40).stop 2023-11-17T05:28:27.730 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:27.729+0000 7f16c8cb8700 1 -- 172.21.15.160:0/2131392435 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16c40f5410 msgr2=0x7f16c41066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:28:27.730 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:27.729+0000 7f16c8cb8700 1 --2- 172.21.15.160:0/2131392435 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16c40f5410 0x7f16c41066d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f16ac013510 tx=0x7f16ac00b040).stop 2023-11-17T05:28:27.730 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:27.729+0000 7f16c8cb8700 1 -- 172.21.15.160:0/2131392435 shutdown_connections 2023-11-17T05:28:27.731 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:27.729+0000 7f16c8cb8700 1 --2- 172.21.15.160:0/2131392435 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f16b002ff60 0x7f16b0032420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:28:27.731 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:27.729+0000 7f16c8cb8700 1 --2- 172.21.15.160:0/2131392435 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16c40f5410 0x7f16c41066d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:28:27.731 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:27.729+0000 7f16c8cb8700 1 -- 172.21.15.160:0/2131392435 >> 172.21.15.160:0/2131392435 conn(0x7f16c40f0b70 msgr2=0x7f16c40f1770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:28:27.731 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:27.729+0000 7f16c8cb8700 1 -- 172.21.15.160:0/2131392435 shutdown_connections 2023-11-17T05:28:27.731 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:27.729+0000 7f16c8cb8700 1 -- 172.21.15.160:0/2131392435 wait complete. 2023-11-17T05:28:27.731 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:28:28.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:28:27.417705+0000 mgr.smithi003.eddcuc (mgr.14164) 91 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:28:28.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:28:27.548467+0000 mon.smithi003 (mon.0) 149 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.eddcuc/mirror_snapshot_schedule"}]: dispatch 2023-11-17T05:28:28.771 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:28:27.552206+0000 mon.smithi003 (mon.0) 150 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.eddcuc/trash_purge_schedule"}]: dispatch 2023-11-17T05:28:28.771 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:28:27.728809+0000 mon.smithi003 (mon.0) 151 : audit [DBG] from='client.? 172.21.15.160:0/2131392435' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:28:29.106 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:28:29.106 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:28:30.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:29 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:28:29.418031+0000 mgr.smithi003.eddcuc (mgr.14164) 92 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:28:30.312 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:30.310+0000 7fbfd8b8c700 1 -- 172.21.15.160:0/2731019825 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbfd40f4b00 msgr2=0x7fbfd40f4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:28:30.312 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:30.310+0000 7fbfd8b8c700 1 --2- 172.21.15.160:0/2731019825 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbfd40f4b00 0x7fbfd40f4f20 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fbfc4009fb0 tx=0x7fbfc4009b30).stop 2023-11-17T05:28:30.312 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:30.310+0000 7fbfd8b8c700 1 -- 172.21.15.160:0/2731019825 shutdown_connections 2023-11-17T05:28:30.312 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:30.310+0000 7fbfd8b8c700 1 --2- 172.21.15.160:0/2731019825 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbfd40f4b00 0x7fbfd40f4f20 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:28:30.312 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:30.310+0000 7fbfd8b8c700 1 -- 172.21.15.160:0/2731019825 >> 172.21.15.160:0/2731019825 conn(0x7fbfd40f0260 msgr2=0x7fbfd40f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:28:30.313 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:30.310+0000 7fbfd8b8c700 1 -- 172.21.15.160:0/2731019825 shutdown_connections 2023-11-17T05:28:30.313 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:30.310+0000 7fbfd8b8c700 1 -- 172.21.15.160:0/2731019825 wait complete. 2023-11-17T05:28:30.313 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:30.311+0000 7fbfd8b8c700 1 Processor -- start 2023-11-17T05:28:30.313 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:30.311+0000 7fbfd8b8c700 1 -- start start 2023-11-17T05:28:30.313 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:30.311+0000 7fbfd8b8c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbfd40f4b00 0x7fbfd410b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:28:30.313 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:30.311+0000 7fbfd8b8c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd4107210 con 0x7fbfd40f4b00 2023-11-17T05:28:30.313 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:30.311+0000 7fbfd259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbfd40f4b00 0x7fbfd410b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:28:30.313 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:30.311+0000 7fbfd259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbfd40f4b00 0x7fbfd410b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:44426/0 (socket says 172.21.15.160:44426) 2023-11-17T05:28:30.313 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:30.311+0000 7fbfd259c700 1 -- 172.21.15.160:0/2602049573 learned_addr learned my addr 172.21.15.160:0/2602049573 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:28:30.313 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:30.311+0000 7fbfd259c700 1 -- 172.21.15.160:0/2602049573 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfc4005040 con 0x7fbfd40f4b00 2023-11-17T05:28:30.313 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:30.312+0000 7fbfd259c700 1 --2- 172.21.15.160:0/2602049573 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbfd40f4b00 0x7fbfd410b820 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fbfc4013510 tx=0x7fbfc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:28:30.315 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:30.312+0000 7fbfc2ffd700 1 -- 172.21.15.160:0/2602049573 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfc4009320 con 0x7fbfd40f4b00 2023-11-17T05:28:30.315 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:30.312+0000 7fbfc2ffd700 1 -- 172.21.15.160:0/2602049573 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbfc4013f00 con 0x7fbfd40f4b00 2023-11-17T05:28:30.315 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:30.312+0000 7fbfd8b8c700 1 -- 172.21.15.160:0/2602049573 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfd40f1870 con 0x7fbfd40f4b00 2023-11-17T05:28:30.315 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:30.312+0000 7fbfc2ffd700 1 -- 172.21.15.160:0/2602049573 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfc4014d90 con 0x7fbfd40f4b00 2023-11-17T05:28:30.315 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:30.312+0000 7fbfd8b8c700 1 -- 172.21.15.160:0/2602049573 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfd410bd10 con 0x7fbfd40f4b00 2023-11-17T05:28:30.315 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:30.313+0000 7fbfd8b8c700 1 -- 172.21.15.160:0/2602049573 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfd4045ad0 con 0x7fbfd40f4b00 2023-11-17T05:28:30.315 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:30.313+0000 7fbfc2ffd700 1 -- 172.21.15.160:0/2602049573 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbfc40095f0 con 0x7fbfd40f4b00 2023-11-17T05:28:30.315 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:30.313+0000 7fbfc2ffd700 1 --2- 172.21.15.160:0/2602049573 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fbfbc02ff60 0x7fbfbc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:28:30.315 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:30.313+0000 7fbfc2ffd700 1 -- 172.21.15.160:0/2602049573 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbfc4041000 con 0x7fbfd40f4b00 2023-11-17T05:28:30.316 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:30.313+0000 7fbfd1d9b700 1 --2- 172.21.15.160:0/2602049573 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fbfbc02ff60 0x7fbfbc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:28:30.316 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:30.314+0000 7fbfd1d9b700 1 --2- 172.21.15.160:0/2602049573 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fbfbc02ff60 0x7fbfbc032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fbfc80096a0 tx=0x7fbfc8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:28:30.323 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:30.318+0000 7fbfc2ffd700 1 -- 172.21.15.160:0/2602049573 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbfc4014420 con 0x7fbfd40f4b00 2023-11-17T05:28:30.521 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:30.520+0000 7fbfd8b8c700 1 -- 172.21.15.160:0/2602049573 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbfd4027a00 con 0x7fbfd40f4b00 2023-11-17T05:28:30.522 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:30.521+0000 7fbfc2ffd700 1 -- 172.21.15.160:0/2602049573 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fbfc401d7f0 con 0x7fbfd40f4b00 2023-11-17T05:28:30.523 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:28:30.523 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:28:30.523 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:30.522+0000 7fbfd8b8c700 1 -- 172.21.15.160:0/2602049573 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fbfbc02ff60 msgr2=0x7fbfbc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:28:30.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:30.522+0000 7fbfd8b8c700 1 --2- 172.21.15.160:0/2602049573 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fbfbc02ff60 0x7fbfbc032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fbfc80096a0 tx=0x7fbfc8006b40).stop 2023-11-17T05:28:30.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:30.522+0000 7fbfd8b8c700 1 -- 172.21.15.160:0/2602049573 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbfd40f4b00 msgr2=0x7fbfd410b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:28:30.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:30.522+0000 7fbfd8b8c700 1 --2- 172.21.15.160:0/2602049573 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbfd40f4b00 0x7fbfd410b820 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fbfc4013510 tx=0x7fbfc400b040).stop 2023-11-17T05:28:30.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:30.522+0000 7fbfd8b8c700 1 -- 172.21.15.160:0/2602049573 shutdown_connections 2023-11-17T05:28:30.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:30.522+0000 7fbfd8b8c700 1 --2- 172.21.15.160:0/2602049573 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fbfbc02ff60 0x7fbfbc032420 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:28:30.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:30.522+0000 7fbfd8b8c700 1 --2- 172.21.15.160:0/2602049573 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbfd40f4b00 0x7fbfd410b820 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:28:30.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:30.522+0000 7fbfd8b8c700 1 -- 172.21.15.160:0/2602049573 >> 172.21.15.160:0/2602049573 conn(0x7fbfd40f0260 msgr2=0x7fbfd40f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:28:30.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:30.523+0000 7fbfd8b8c700 1 -- 172.21.15.160:0/2602049573 shutdown_connections 2023-11-17T05:28:30.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:30.523+0000 7fbfd8b8c700 1 -- 172.21.15.160:0/2602049573 wait complete. 2023-11-17T05:28:30.525 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:28:31.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:30 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:28:30.522199+0000 mon.smithi003 (mon.0) 152 : audit [DBG] from='client.? 172.21.15.160:0/2602049573' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:28:31.957 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:28:31.957 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:28:32.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:31 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:28:31.418401+0000 mgr.smithi003.eddcuc (mgr.14164) 93 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:28:33.129 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:33.128+0000 7f7a17fff700 1 -- 172.21.15.160:0/3491026878 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a180f62a0 msgr2=0x7f7a180f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:28:46.643 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:33.128+0000 7f7a17fff700 1 --2- 172.21.15.160:0/3491026878 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a180f62a0 0x7f7a180f66c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f7a0c009fb0 tx=0x7f7a0c009b30).stop 2023-11-17T05:28:46.643 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:33.128+0000 7f7a17fff700 1 -- 172.21.15.160:0/3491026878 shutdown_connections 2023-11-17T05:28:46.643 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:33.128+0000 7f7a17fff700 1 --2- 172.21.15.160:0/3491026878 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a180f62a0 0x7f7a180f66c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:28:46.644 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:33.128+0000 7f7a17fff700 1 -- 172.21.15.160:0/3491026878 >> 172.21.15.160:0/3491026878 conn(0x7f7a180f1aa0 msgr2=0x7f7a180f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:28:46.644 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:33.128+0000 7f7a17fff700 1 -- 172.21.15.160:0/3491026878 shutdown_connections 2023-11-17T05:28:46.644 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:33.128+0000 7f7a17fff700 1 -- 172.21.15.160:0/3491026878 wait complete. 2023-11-17T05:28:46.644 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:33.128+0000 7f7a17fff700 1 Processor -- start 2023-11-17T05:28:46.644 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:33.128+0000 7f7a17fff700 1 -- start start 2023-11-17T05:28:46.644 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:33.129+0000 7f7a17fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a180f62a0 0x7f7a1806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:28:46.644 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:33.129+0000 7f7a17fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a18069e20 con 0x7f7a180f62a0 2023-11-17T05:28:46.646 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:28:33.342208+0000 mon.smithi003 (mon.0) 153 : audit [DBG] from='client.? 172.21.15.160:0/3025974165' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:28:46.647 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:34 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:28:33.418816+0000 mgr.smithi003.eddcuc (mgr.14164) 94 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:28:46.647 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:36 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:28:35.419178+0000 mgr.smithi003.eddcuc (mgr.14164) 95 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:28:46.647 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:28:37.419536+0000 mgr.smithi003.eddcuc (mgr.14164) 96 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:28:46.647 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:39 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:28:39.419757+0000 mgr.smithi003.eddcuc (mgr.14164) 97 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:28:46.647 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:42 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:28:41.420050+0000 mgr.smithi003.eddcuc (mgr.14164) 98 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:28:46.647 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:44 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:28:43.420366+0000 mgr.smithi003.eddcuc (mgr.14164) 99 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:28:46.648 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:33.129+0000 7f7a16ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a180f62a0 0x7f7a1806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:28:46.648 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:33.129+0000 7f7a16ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a180f62a0 0x7f7a1806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:44444/0 (socket says 172.21.15.160:44444) 2023-11-17T05:28:46.648 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:33.129+0000 7f7a16ffd700 1 -- 172.21.15.160:0/3025974165 learned_addr learned my addr 172.21.15.160:0/3025974165 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:28:46.648 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:33.129+0000 7f7a16ffd700 1 -- 172.21.15.160:0/3025974165 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a0c005040 con 0x7f7a180f62a0 2023-11-17T05:28:46.649 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:33.129+0000 7f7a16ffd700 1 --2- 172.21.15.160:0/3025974165 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a180f62a0 0x7f7a1806b9a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f7a0c013510 tx=0x7f7a0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:28:46.649 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:33.129+0000 7f79ff7fe700 1 -- 172.21.15.160:0/3025974165 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a0c009320 con 0x7f7a180f62a0 2023-11-17T05:28:46.649 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:33.130+0000 7f79ff7fe700 1 -- 172.21.15.160:0/3025974165 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7a0c013f00 con 0x7f7a180f62a0 2023-11-17T05:28:46.649 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:33.130+0000 7f7a17fff700 1 -- 172.21.15.160:0/3025974165 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a18069fc0 con 0x7f7a180f62a0 2023-11-17T05:28:46.649 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:33.130+0000 7f79ff7fe700 1 -- 172.21.15.160:0/3025974165 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a0c014dc0 con 0x7f7a180f62a0 2023-11-17T05:28:46.649 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:33.130+0000 7f7a17fff700 1 -- 172.21.15.160:0/3025974165 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a1806a450 con 0x7f7a180f62a0 2023-11-17T05:28:46.649 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:33.130+0000 7f79ff7fe700 1 -- 172.21.15.160:0/3025974165 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7a0c01d3d0 con 0x7f7a180f62a0 2023-11-17T05:28:46.649 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:33.130+0000 7f79ff7fe700 1 --2- 172.21.15.160:0/3025974165 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f7a0002ff60 0x7f7a00032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:28:46.649 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:33.131+0000 7f79ff7fe700 1 -- 172.21.15.160:0/3025974165 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7a0c0410e0 con 0x7f7a180f62a0 2023-11-17T05:28:46.649 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:33.131+0000 7f7a17fff700 1 -- 172.21.15.160:0/3025974165 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a18059100 con 0x7f7a180f62a0 2023-11-17T05:28:46.649 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:33.131+0000 7f7a167fc700 1 --2- 172.21.15.160:0/3025974165 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f7a0002ff60 0x7f7a00032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:28:46.649 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:33.131+0000 7f7a167fc700 1 --2- 172.21.15.160:0/3025974165 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f7a0002ff60 0x7f7a00032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f7a080096a0 tx=0x7f7a08006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:28:46.649 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:33.135+0000 7f79ff7fe700 1 -- 172.21.15.160:0/3025974165 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7a0c012900 con 0x7f7a180f62a0 2023-11-17T05:28:46.650 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:33.340+0000 7f7a17fff700 1 -- 172.21.15.160:0/3025974165 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7a1806ab50 con 0x7f7a180f62a0 2023-11-17T05:28:46.650 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:33.341+0000 7f79ff7fe700 1 -- 172.21.15.160:0/3025974165 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f7a0c02b020 con 0x7f7a180f62a0 2023-11-17T05:28:46.650 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:28:46.650 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:28:46.651 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:33.342+0000 7f7a17fff700 1 -- 172.21.15.160:0/3025974165 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f7a0002ff60 msgr2=0x7f7a00032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:28:46.652 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:33.342+0000 7f7a17fff700 1 --2- 172.21.15.160:0/3025974165 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f7a0002ff60 0x7f7a00032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f7a080096a0 tx=0x7f7a08006b40).stop 2023-11-17T05:28:46.652 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:33.342+0000 7f7a17fff700 1 -- 172.21.15.160:0/3025974165 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a180f62a0 msgr2=0x7f7a1806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:28:46.652 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:33.342+0000 7f7a17fff700 1 --2- 172.21.15.160:0/3025974165 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a180f62a0 0x7f7a1806b9a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f7a0c013510 tx=0x7f7a0c00b040).stop 2023-11-17T05:28:46.652 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:33.342+0000 7f7a17fff700 1 -- 172.21.15.160:0/3025974165 shutdown_connections 2023-11-17T05:28:46.652 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:33.342+0000 7f7a17fff700 1 --2- 172.21.15.160:0/3025974165 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f7a0002ff60 0x7f7a00032420 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:28:46.652 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:33.342+0000 7f7a17fff700 1 --2- 172.21.15.160:0/3025974165 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a180f62a0 0x7f7a1806b9a0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:28:46.652 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:33.342+0000 7f7a17fff700 1 -- 172.21.15.160:0/3025974165 >> 172.21.15.160:0/3025974165 conn(0x7f7a180f1aa0 msgr2=0x7f7a180f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:28:46.652 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:33.342+0000 7f7a17fff700 1 -- 172.21.15.160:0/3025974165 shutdown_connections 2023-11-17T05:28:46.652 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:33.343+0000 7f7a17fff700 1 -- 172.21.15.160:0/3025974165 wait complete. 2023-11-17T05:28:46.652 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:28:46.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:46 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:28:45.420738+0000 mgr.smithi003.eddcuc (mgr.14164) 100 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:28:47.654 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:28:47.654 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:28:48.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:48 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:28:47.421030+0000 mgr.smithi003.eddcuc (mgr.14164) 101 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:28:53.748 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:49 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:28:49.026740+0000 mon.smithi003 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.160:0/1518990448' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:28:53.748 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:49 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:28:49.421382+0000 mgr.smithi003.eddcuc (mgr.14164) 102 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:28:53.749 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:52 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:28:51.421678+0000 mgr.smithi003.eddcuc (mgr.14164) 103 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:28:53.822 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:48.811+0000 7fc7f7101700 1 -- 172.21.15.160:0/2203433147 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7f00f62a0 msgr2=0x7fc7f00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:28:53.822 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:48.811+0000 7fc7f7101700 1 --2- 172.21.15.160:0/2203433147 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7f00f62a0 0x7fc7f00f66c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fc7ec009fb0 tx=0x7fc7ec009b30).stop 2023-11-17T05:28:53.822 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:48.811+0000 7fc7f7101700 1 -- 172.21.15.160:0/2203433147 shutdown_connections 2023-11-17T05:28:53.822 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:48.811+0000 7fc7f7101700 1 --2- 172.21.15.160:0/2203433147 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7f00f62a0 0x7fc7f00f66c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:28:53.822 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:48.811+0000 7fc7f7101700 1 -- 172.21.15.160:0/2203433147 >> 172.21.15.160:0/2203433147 conn(0x7fc7f00f1aa0 msgr2=0x7fc7f00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:28:53.822 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:48.811+0000 7fc7f7101700 1 -- 172.21.15.160:0/2203433147 shutdown_connections 2023-11-17T05:28:53.822 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:48.811+0000 7fc7f7101700 1 -- 172.21.15.160:0/2203433147 wait complete. 2023-11-17T05:28:53.822 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:48.811+0000 7fc7f7101700 1 Processor -- start 2023-11-17T05:28:53.822 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:48.811+0000 7fc7f7101700 1 -- start start 2023-11-17T05:28:53.822 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:48.812+0000 7fc7f7101700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7f00f62a0 0x7fc7f01064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:28:53.823 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:48.812+0000 7fc7f7101700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f01069a0 con 0x7fc7f00f62a0 2023-11-17T05:28:53.823 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:48.812+0000 7fc7f60ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7f00f62a0 0x7fc7f01064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:28:53.823 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:48.812+0000 7fc7f60ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7f00f62a0 0x7fc7f01064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:59164/0 (socket says 172.21.15.160:59164) 2023-11-17T05:28:53.823 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:48.812+0000 7fc7f60ff700 1 -- 172.21.15.160:0/1518990448 learned_addr learned my addr 172.21.15.160:0/1518990448 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:28:53.823 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:48.812+0000 7fc7f60ff700 1 -- 172.21.15.160:0/1518990448 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7ec005040 con 0x7fc7f00f62a0 2023-11-17T05:28:53.823 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:48.812+0000 7fc7f60ff700 1 --2- 172.21.15.160:0/1518990448 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7f00f62a0 0x7fc7f01064b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fc7ec009830 tx=0x7fc7ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:28:53.823 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:48.812+0000 7fc7e6ffd700 1 -- 172.21.15.160:0/1518990448 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7ec009320 con 0x7fc7f00f62a0 2023-11-17T05:28:53.823 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:48.812+0000 7fc7e6ffd700 1 -- 172.21.15.160:0/1518990448 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc7ec0136c0 con 0x7fc7f00f62a0 2023-11-17T05:28:53.823 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:48.812+0000 7fc7f7101700 1 -- 172.21.15.160:0/1518990448 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7f0106ba0 con 0x7fc7f00f62a0 2023-11-17T05:28:53.823 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:48.812+0000 7fc7e6ffd700 1 -- 172.21.15.160:0/1518990448 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7ec01ee90 con 0x7fc7f00f62a0 2023-11-17T05:28:53.823 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:48.812+0000 7fc7f7101700 1 -- 172.21.15.160:0/1518990448 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7f0107030 con 0x7fc7f00f62a0 2023-11-17T05:28:53.823 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:48.813+0000 7fc7e6ffd700 1 -- 172.21.15.160:0/1518990448 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc7ec01e420 con 0x7fc7f00f62a0 2023-11-17T05:28:53.823 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:48.813+0000 7fc7f7101700 1 -- 172.21.15.160:0/1518990448 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7f0045ad0 con 0x7fc7f00f62a0 2023-11-17T05:28:53.823 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:48.813+0000 7fc7e6ffd700 1 --2- 172.21.15.160:0/1518990448 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc7dc02ff10 0x7fc7dc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:28:53.824 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:48.813+0000 7fc7e6ffd700 1 -- 172.21.15.160:0/1518990448 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc7ec040fe0 con 0x7fc7f00f62a0 2023-11-17T05:28:53.824 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:48.814+0000 7fc7f58fe700 1 --2- 172.21.15.160:0/1518990448 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc7dc02ff10 0x7fc7dc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:28:53.824 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:48.814+0000 7fc7f58fe700 1 --2- 172.21.15.160:0/1518990448 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc7dc02ff10 0x7fc7dc0323d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fc7e00096a0 tx=0x7fc7e0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:28:53.824 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:48.818+0000 7fc7e6ffd700 1 -- 172.21.15.160:0/1518990448 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc7ec0095f0 con 0x7fc7f00f62a0 2023-11-17T05:28:53.824 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:49.025+0000 7fc7f7101700 1 -- 172.21.15.160:0/1518990448 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc7f0107ee0 con 0x7fc7f00f62a0 2023-11-17T05:28:53.824 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:49.026+0000 7fc7e6ffd700 1 -- 172.21.15.160:0/1518990448 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fc7ec02c2c0 con 0x7fc7f00f62a0 2023-11-17T05:28:53.824 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:28:53.824 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:28:53.825 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:49.027+0000 7fc7f7101700 1 -- 172.21.15.160:0/1518990448 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc7dc02ff10 msgr2=0x7fc7dc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:28:53.825 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:49.027+0000 7fc7f7101700 1 --2- 172.21.15.160:0/1518990448 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc7dc02ff10 0x7fc7dc0323d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fc7e00096a0 tx=0x7fc7e0006b40).stop 2023-11-17T05:28:53.826 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:49.027+0000 7fc7f7101700 1 -- 172.21.15.160:0/1518990448 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7f00f62a0 msgr2=0x7fc7f01064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:28:53.826 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:49.027+0000 7fc7f7101700 1 --2- 172.21.15.160:0/1518990448 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7f00f62a0 0x7fc7f01064b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fc7ec009830 tx=0x7fc7ec00b040).stop 2023-11-17T05:28:53.826 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:49.027+0000 7fc7f7101700 1 -- 172.21.15.160:0/1518990448 shutdown_connections 2023-11-17T05:28:53.826 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:49.027+0000 7fc7f7101700 1 --2- 172.21.15.160:0/1518990448 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc7dc02ff10 0x7fc7dc0323d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:28:53.826 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:49.027+0000 7fc7f7101700 1 --2- 172.21.15.160:0/1518990448 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7f00f62a0 0x7fc7f01064b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:28:53.826 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:49.027+0000 7fc7f7101700 1 -- 172.21.15.160:0/1518990448 >> 172.21.15.160:0/1518990448 conn(0x7fc7f00f1aa0 msgr2=0x7fc7f00f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:28:53.826 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:49.027+0000 7fc7f7101700 1 -- 172.21.15.160:0/1518990448 shutdown_connections 2023-11-17T05:28:53.826 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:49.027+0000 7fc7f7101700 1 -- 172.21.15.160:0/1518990448 wait complete. 2023-11-17T05:28:53.826 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:28:54.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:54 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:28:53.422004+0000 mgr.smithi003.eddcuc (mgr.14164) 104 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:28:54.828 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:28:54.829 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:28:56.020 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:56.019+0000 7f0d56cdc700 1 -- 172.21.15.160:0/1179395672 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d500f5410 msgr2=0x7f0d500f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:28:56.020 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:56.019+0000 7f0d56cdc700 1 --2- 172.21.15.160:0/1179395672 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d500f5410 0x7f0d500f5830 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f0d44009fb0 tx=0x7f0d44009b30).stop 2023-11-17T05:28:56.020 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:56.019+0000 7f0d56cdc700 1 -- 172.21.15.160:0/1179395672 shutdown_connections 2023-11-17T05:28:56.021 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:56.019+0000 7f0d56cdc700 1 --2- 172.21.15.160:0/1179395672 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d500f5410 0x7f0d500f5830 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:28:56.021 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:56.019+0000 7f0d56cdc700 1 -- 172.21.15.160:0/1179395672 >> 172.21.15.160:0/1179395672 conn(0x7f0d500f0b70 msgr2=0x7f0d500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:28:56.021 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:56.019+0000 7f0d56cdc700 1 -- 172.21.15.160:0/1179395672 shutdown_connections 2023-11-17T05:28:56.021 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:56.019+0000 7f0d56cdc700 1 -- 172.21.15.160:0/1179395672 wait complete. 2023-11-17T05:28:56.022 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:56.020+0000 7f0d56cdc700 1 Processor -- start 2023-11-17T05:28:56.022 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:56.020+0000 7f0d56cdc700 1 -- start start 2023-11-17T05:28:56.022 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:56.020+0000 7f0d56cdc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d500f5410 0x7f0d501055d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:28:56.022 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:56.020+0000 7f0d56cdc700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d50105ac0 con 0x7f0d500f5410 2023-11-17T05:28:56.022 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:56.021+0000 7f0d54a78700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d500f5410 0x7f0d501055d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:28:56.022 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:56.021+0000 7f0d54a78700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d500f5410 0x7f0d501055d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:55160/0 (socket says 172.21.15.160:55160) 2023-11-17T05:28:56.023 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:56.021+0000 7f0d54a78700 1 -- 172.21.15.160:0/4110538278 learned_addr learned my addr 172.21.15.160:0/4110538278 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:28:56.023 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:56.021+0000 7f0d54a78700 1 -- 172.21.15.160:0/4110538278 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d44005040 con 0x7f0d500f5410 2023-11-17T05:28:56.023 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:56.021+0000 7f0d54a78700 1 --2- 172.21.15.160:0/4110538278 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d500f5410 0x7f0d501055d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f0d44013510 tx=0x7f0d4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:28:56.023 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:56.021+0000 7f0d4d7fa700 1 -- 172.21.15.160:0/4110538278 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d44009320 con 0x7f0d500f5410 2023-11-17T05:28:56.023 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:56.021+0000 7f0d4d7fa700 1 -- 172.21.15.160:0/4110538278 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d44013f00 con 0x7f0d500f5410 2023-11-17T05:28:56.023 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:56.021+0000 7f0d56cdc700 1 -- 172.21.15.160:0/4110538278 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d50105cc0 con 0x7f0d500f5410 2023-11-17T05:28:56.023 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:56.021+0000 7f0d4d7fa700 1 -- 172.21.15.160:0/4110538278 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d44014d90 con 0x7f0d500f5410 2023-11-17T05:28:56.023 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:56.021+0000 7f0d56cdc700 1 -- 172.21.15.160:0/4110538278 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d50106150 con 0x7f0d500f5410 2023-11-17T05:28:56.024 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:56.022+0000 7f0d4d7fa700 1 -- 172.21.15.160:0/4110538278 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0d44014420 con 0x7f0d500f5410 2023-11-17T05:28:56.024 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:56.022+0000 7f0d56cdc700 1 -- 172.21.15.160:0/4110538278 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d50045ad0 con 0x7f0d500f5410 2023-11-17T05:28:56.024 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:56.022+0000 7f0d4d7fa700 1 --2- 172.21.15.160:0/4110538278 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f0d4002ff60 0x7f0d40032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:28:56.024 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:56.022+0000 7f0d4d7fa700 1 -- 172.21.15.160:0/4110538278 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0d440410c0 con 0x7f0d500f5410 2023-11-17T05:28:56.024 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:56.022+0000 7f0d4ffff700 1 --2- 172.21.15.160:0/4110538278 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f0d4002ff60 0x7f0d40032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:28:56.024 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:56.023+0000 7f0d4ffff700 1 --2- 172.21.15.160:0/4110538278 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f0d4002ff60 0x7f0d40032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f0d3c0096a0 tx=0x7f0d3c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:28:56.028 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:56.027+0000 7f0d4d7fa700 1 -- 172.21.15.160:0/4110538278 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0d440095f0 con 0x7f0d500f5410 2023-11-17T05:28:56.235 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:56.234+0000 7f0d56cdc700 1 -- 172.21.15.160:0/4110538278 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0d50107000 con 0x7f0d500f5410 2023-11-17T05:28:56.236 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:56.236+0000 7f0d4d7fa700 1 -- 172.21.15.160:0/4110538278 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f0d4401db70 con 0x7f0d500f5410 2023-11-17T05:28:56.237 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:28:56.291 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:28:56.327 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:56.236+0000 7f0d56cdc700 1 -- 172.21.15.160:0/4110538278 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f0d4002ff60 msgr2=0x7f0d40032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:28:56.327 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:56.236+0000 7f0d56cdc700 1 --2- 172.21.15.160:0/4110538278 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f0d4002ff60 0x7f0d40032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f0d3c0096a0 tx=0x7f0d3c006b40).stop 2023-11-17T05:28:56.327 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:56.236+0000 7f0d56cdc700 1 -- 172.21.15.160:0/4110538278 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d500f5410 msgr2=0x7f0d501055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:28:56.327 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:56.236+0000 7f0d56cdc700 1 --2- 172.21.15.160:0/4110538278 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d500f5410 0x7f0d501055d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f0d44013510 tx=0x7f0d4400b040).stop 2023-11-17T05:28:56.327 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:56.237+0000 7f0d56cdc700 1 -- 172.21.15.160:0/4110538278 shutdown_connections 2023-11-17T05:28:56.327 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:56.237+0000 7f0d56cdc700 1 --2- 172.21.15.160:0/4110538278 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f0d4002ff60 0x7f0d40032420 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:28:56.327 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:56.237+0000 7f0d56cdc700 1 --2- 172.21.15.160:0/4110538278 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d500f5410 0x7f0d501055d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:28:56.327 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:56.237+0000 7f0d56cdc700 1 -- 172.21.15.160:0/4110538278 >> 172.21.15.160:0/4110538278 conn(0x7f0d500f0b70 msgr2=0x7f0d500f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:28:56.327 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:56.237+0000 7f0d56cdc700 1 -- 172.21.15.160:0/4110538278 shutdown_connections 2023-11-17T05:28:56.327 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:56.237+0000 7f0d56cdc700 1 -- 172.21.15.160:0/4110538278 wait complete. 2023-11-17T05:28:56.327 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:28:56.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:56 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:28:55.422337+0000 mgr.smithi003.eddcuc (mgr.14164) 105 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:28:56.771 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:56 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:28:56.236195+0000 mon.smithi003 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.160:0/4110538278' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:28:57.636 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:28:57.637 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:28:58.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:58 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:28:57.422674+0000 mgr.smithi003.eddcuc (mgr.14164) 106 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:28:58.857 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:58.855+0000 7fc4b3935700 1 -- 172.21.15.160:0/717316176 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4ac0f6430 msgr2=0x7fc4ac0f6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:28:58.857 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:58.855+0000 7fc4b3935700 1 --2- 172.21.15.160:0/717316176 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4ac0f6430 0x7fc4ac0f6850 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fc4a4009fb0 tx=0x7fc4a4009b30).stop 2023-11-17T05:28:58.857 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:58.855+0000 7fc4b3935700 1 -- 172.21.15.160:0/717316176 shutdown_connections 2023-11-17T05:28:58.857 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:58.855+0000 7fc4b3935700 1 --2- 172.21.15.160:0/717316176 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4ac0f6430 0x7fc4ac0f6850 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:28:58.857 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:58.855+0000 7fc4b3935700 1 -- 172.21.15.160:0/717316176 >> 172.21.15.160:0/717316176 conn(0x7fc4ac0f1b90 msgr2=0x7fc4ac0f3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:28:58.857 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:58.855+0000 7fc4b3935700 1 -- 172.21.15.160:0/717316176 shutdown_connections 2023-11-17T05:28:58.857 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:58.855+0000 7fc4b3935700 1 -- 172.21.15.160:0/717316176 wait complete. 2023-11-17T05:28:58.857 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:58.856+0000 7fc4b3935700 1 Processor -- start 2023-11-17T05:28:58.857 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:58.856+0000 7fc4b3935700 1 -- start start 2023-11-17T05:28:58.858 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:58.856+0000 7fc4b3935700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4ac0f6430 0x7fc4ac1069d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:28:58.858 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:58.856+0000 7fc4b3935700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ac106ec0 con 0x7fc4ac0f6430 2023-11-17T05:28:58.859 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:58.856+0000 7fc4b16d1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4ac0f6430 0x7fc4ac1069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:28:58.859 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:58.856+0000 7fc4b16d1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4ac0f6430 0x7fc4ac1069d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:55172/0 (socket says 172.21.15.160:55172) 2023-11-17T05:28:58.859 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:58.856+0000 7fc4b16d1700 1 -- 172.21.15.160:0/3133865787 learned_addr learned my addr 172.21.15.160:0/3133865787 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:28:58.859 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:58.857+0000 7fc4b16d1700 1 -- 172.21.15.160:0/3133865787 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a4005040 con 0x7fc4ac0f6430 2023-11-17T05:28:58.859 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:58.857+0000 7fc4b16d1700 1 --2- 172.21.15.160:0/3133865787 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4ac0f6430 0x7fc4ac1069d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fc4a4013510 tx=0x7fc4a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:28:58.860 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:58.857+0000 7fc4a1ffb700 1 -- 172.21.15.160:0/3133865787 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4a4009320 con 0x7fc4ac0f6430 2023-11-17T05:28:58.860 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:58.857+0000 7fc4b3935700 1 -- 172.21.15.160:0/3133865787 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4ac1070c0 con 0x7fc4ac0f6430 2023-11-17T05:28:58.860 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:58.857+0000 7fc4a1ffb700 1 -- 172.21.15.160:0/3133865787 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc4a4013f00 con 0x7fc4ac0f6430 2023-11-17T05:28:58.860 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:58.857+0000 7fc4a1ffb700 1 -- 172.21.15.160:0/3133865787 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4a4014d90 con 0x7fc4ac0f6430 2023-11-17T05:28:58.860 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:58.857+0000 7fc4b3935700 1 -- 172.21.15.160:0/3133865787 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4ac107550 con 0x7fc4ac0f6430 2023-11-17T05:28:58.860 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:58.858+0000 7fc4a1ffb700 1 -- 172.21.15.160:0/3133865787 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc4a40095f0 con 0x7fc4ac0f6430 2023-11-17T05:28:58.860 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:58.858+0000 7fc4a1ffb700 1 --2- 172.21.15.160:0/3133865787 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc49802ff60 0x7fc498032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:28:58.860 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:58.858+0000 7fc4b3935700 1 -- 172.21.15.160:0/3133865787 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4ac045ad0 con 0x7fc4ac0f6430 2023-11-17T05:28:58.860 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:58.858+0000 7fc4a1ffb700 1 -- 172.21.15.160:0/3133865787 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc4a4041000 con 0x7fc4ac0f6430 2023-11-17T05:28:58.860 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:58.858+0000 7fc4b0ed0700 1 --2- 172.21.15.160:0/3133865787 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc49802ff60 0x7fc498032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:28:58.861 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:58.859+0000 7fc4b0ed0700 1 --2- 172.21.15.160:0/3133865787 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc49802ff60 0x7fc498032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fc49c0096a0 tx=0x7fc49c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:28:58.864 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:58.863+0000 7fc4a1ffb700 1 -- 172.21.15.160:0/3133865787 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc4a406d2b0 con 0x7fc4ac0f6430 2023-11-17T05:28:59.068 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:59.067+0000 7fc4b3935700 1 -- 172.21.15.160:0/3133865787 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc4ac05ea50 con 0x7fc4ac0f6430 2023-11-17T05:28:59.068 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:59.068+0000 7fc4a1ffb700 1 -- 172.21.15.160:0/3133865787 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fc4a401d7f0 con 0x7fc4ac0f6430 2023-11-17T05:28:59.069 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:28:59.069 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:28:59.071 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:59.069+0000 7fc4b3935700 1 -- 172.21.15.160:0/3133865787 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc49802ff60 msgr2=0x7fc498032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:28:59.071 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:59.069+0000 7fc4b3935700 1 --2- 172.21.15.160:0/3133865787 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc49802ff60 0x7fc498032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fc49c0096a0 tx=0x7fc49c006b40).stop 2023-11-17T05:28:59.072 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:59.069+0000 7fc4b3935700 1 -- 172.21.15.160:0/3133865787 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4ac0f6430 msgr2=0x7fc4ac1069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:28:59.072 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:59.069+0000 7fc4b3935700 1 --2- 172.21.15.160:0/3133865787 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4ac0f6430 0x7fc4ac1069d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fc4a4013510 tx=0x7fc4a400b040).stop 2023-11-17T05:28:59.072 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:59.069+0000 7fc4b3935700 1 -- 172.21.15.160:0/3133865787 shutdown_connections 2023-11-17T05:28:59.072 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:59.069+0000 7fc4b3935700 1 --2- 172.21.15.160:0/3133865787 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc49802ff60 0x7fc498032420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:28:59.072 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:59.069+0000 7fc4b3935700 1 --2- 172.21.15.160:0/3133865787 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4ac0f6430 0x7fc4ac1069d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:28:59.072 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:59.069+0000 7fc4b3935700 1 -- 172.21.15.160:0/3133865787 >> 172.21.15.160:0/3133865787 conn(0x7fc4ac0f1b90 msgr2=0x7fc4ac0f2630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:28:59.072 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:59.069+0000 7fc4b3935700 1 -- 172.21.15.160:0/3133865787 shutdown_connections 2023-11-17T05:28:59.072 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:28:59.070+0000 7fc4b3935700 1 -- 172.21.15.160:0/3133865787 wait complete. 2023-11-17T05:28:59.072 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:29:00.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:59 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:28:59.068905+0000 mon.smithi003 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.160:0/3133865787' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:29:00.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:28:59 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:28:59.422959+0000 mgr.smithi003.eddcuc (mgr.14164) 107 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:29:00.488 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:29:00.488 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:29:01.643 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:01.640+0000 7f9ae7c0a700 1 -- 172.21.15.160:0/2665370771 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ae00f5410 msgr2=0x7f9ae00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:29:01.643 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:01.640+0000 7f9ae7c0a700 1 --2- 172.21.15.160:0/2665370771 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ae00f5410 0x7f9ae00f5830 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f9ad4009fb0 tx=0x7f9ad4009b30).stop 2023-11-17T05:29:01.643 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:01.640+0000 7f9ae7c0a700 1 -- 172.21.15.160:0/2665370771 shutdown_connections 2023-11-17T05:29:01.643 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:01.640+0000 7f9ae7c0a700 1 --2- 172.21.15.160:0/2665370771 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ae00f5410 0x7f9ae00f5830 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:29:01.643 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:01.640+0000 7f9ae7c0a700 1 -- 172.21.15.160:0/2665370771 >> 172.21.15.160:0/2665370771 conn(0x7f9ae00f0b70 msgr2=0x7f9ae00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:29:01.643 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:01.640+0000 7f9ae7c0a700 1 -- 172.21.15.160:0/2665370771 shutdown_connections 2023-11-17T05:29:01.643 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:01.640+0000 7f9ae7c0a700 1 -- 172.21.15.160:0/2665370771 wait complete. 2023-11-17T05:29:01.643 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:01.641+0000 7f9ae7c0a700 1 Processor -- start 2023-11-17T05:29:01.643 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:01.641+0000 7f9ae7c0a700 1 -- start start 2023-11-17T05:29:01.643 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:01.641+0000 7f9ae7c0a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ae00f5410 0x7f9ae00fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:29:01.644 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:01.641+0000 7f9ae7c0a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae00fb820 con 0x7f9ae00f5410 2023-11-17T05:29:01.644 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:01.641+0000 7f9ae59a6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ae00f5410 0x7f9ae00fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:29:01.644 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:01.641+0000 7f9ae59a6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ae00f5410 0x7f9ae00fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:50234/0 (socket says 172.21.15.160:50234) 2023-11-17T05:29:01.644 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:01.641+0000 7f9ae59a6700 1 -- 172.21.15.160:0/1839532975 learned_addr learned my addr 172.21.15.160:0/1839532975 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:29:01.644 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:01.641+0000 7f9ae59a6700 1 -- 172.21.15.160:0/1839532975 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad4005040 con 0x7f9ae00f5410 2023-11-17T05:29:01.644 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:01.642+0000 7f9ae59a6700 1 --2- 172.21.15.160:0/1839532975 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ae00f5410 0x7f9ae00fd3a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f9ad4013510 tx=0x7f9ad400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:29:01.644 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:01.642+0000 7f9ad27fc700 1 -- 172.21.15.160:0/1839532975 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ad4009320 con 0x7f9ae00f5410 2023-11-17T05:29:01.644 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:01.642+0000 7f9ae7c0a700 1 -- 172.21.15.160:0/1839532975 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ae00fb9c0 con 0x7f9ae00f5410 2023-11-17T05:29:01.644 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:01.642+0000 7f9ad27fc700 1 -- 172.21.15.160:0/1839532975 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ad4013f00 con 0x7f9ae00f5410 2023-11-17T05:29:01.644 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:01.642+0000 7f9ad27fc700 1 -- 172.21.15.160:0/1839532975 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ad4014d90 con 0x7f9ae00f5410 2023-11-17T05:29:01.644 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:01.642+0000 7f9ae7c0a700 1 -- 172.21.15.160:0/1839532975 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ae00fbe50 con 0x7f9ae00f5410 2023-11-17T05:29:01.645 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:01.643+0000 7f9ad27fc700 1 -- 172.21.15.160:0/1839532975 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9ad40263a0 con 0x7f9ae00f5410 2023-11-17T05:29:01.645 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:01.643+0000 7f9ad27fc700 1 --2- 172.21.15.160:0/1839532975 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f9acc02ff60 0x7f9acc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:29:01.645 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:01.643+0000 7f9ae7c0a700 1 -- 172.21.15.160:0/1839532975 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ae0059100 con 0x7f9ae00f5410 2023-11-17T05:29:01.645 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:01.643+0000 7f9ad27fc700 1 -- 172.21.15.160:0/1839532975 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9ad4040ff0 con 0x7f9ae00f5410 2023-11-17T05:29:01.645 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:01.643+0000 7f9ae51a5700 1 --2- 172.21.15.160:0/1839532975 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f9acc02ff60 0x7f9acc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:29:01.646 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:01.644+0000 7f9ae51a5700 1 --2- 172.21.15.160:0/1839532975 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f9acc02ff60 0x7f9acc032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f9adc0096a0 tx=0x7f9adc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:29:01.649 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:01.649+0000 7f9ad27fc700 1 -- 172.21.15.160:0/1839532975 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9ad4012920 con 0x7f9ae00f5410 2023-11-17T05:29:01.852 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:01.851+0000 7f9ae7c0a700 1 -- 172.21.15.160:0/1839532975 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9ae00fae60 con 0x7f9ae00f5410 2023-11-17T05:29:01.852 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:01.853+0000 7f9ad27fc700 1 -- 172.21.15.160:0/1839532975 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f9ad402b020 con 0x7f9ae00f5410 2023-11-17T05:29:01.853 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:29:01.853 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:29:01.854 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:01.853+0000 7f9ae7c0a700 1 -- 172.21.15.160:0/1839532975 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f9acc02ff60 msgr2=0x7f9acc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:29:01.854 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:01.853+0000 7f9ae7c0a700 1 --2- 172.21.15.160:0/1839532975 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f9acc02ff60 0x7f9acc032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f9adc0096a0 tx=0x7f9adc006b40).stop 2023-11-17T05:29:01.854 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:01.853+0000 7f9ae7c0a700 1 -- 172.21.15.160:0/1839532975 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ae00f5410 msgr2=0x7f9ae00fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:29:01.854 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:01.853+0000 7f9ae7c0a700 1 --2- 172.21.15.160:0/1839532975 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ae00f5410 0x7f9ae00fd3a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f9ad4013510 tx=0x7f9ad400b040).stop 2023-11-17T05:29:01.855 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:01.853+0000 7f9ae7c0a700 1 -- 172.21.15.160:0/1839532975 shutdown_connections 2023-11-17T05:29:01.855 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:01.853+0000 7f9ae7c0a700 1 --2- 172.21.15.160:0/1839532975 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f9acc02ff60 0x7f9acc032420 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:29:01.855 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:01.853+0000 7f9ae7c0a700 1 --2- 172.21.15.160:0/1839532975 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ae00f5410 0x7f9ae00fd3a0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:29:01.855 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:01.853+0000 7f9ae7c0a700 1 -- 172.21.15.160:0/1839532975 >> 172.21.15.160:0/1839532975 conn(0x7f9ae00f0b70 msgr2=0x7f9ae005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:29:01.855 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:01.853+0000 7f9ae7c0a700 1 -- 172.21.15.160:0/1839532975 shutdown_connections 2023-11-17T05:29:01.855 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:01.853+0000 7f9ae7c0a700 1 -- 172.21.15.160:0/1839532975 wait complete. 2023-11-17T05:29:01.855 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:29:02.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:29:02 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:29:01.423242+0000 mgr.smithi003.eddcuc (mgr.14164) 108 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:29:02.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:29:02 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:29:01.852943+0000 mon.smithi003 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.160:0/1839532975' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:29:03.239 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:29:03.239 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:29:04.341 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:04.340+0000 7fa0a3df7700 1 -- 172.21.15.160:0/2856966637 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa09c0f5410 msgr2=0x7fa09c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:29:04.342 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:04.340+0000 7fa0a3df7700 1 --2- 172.21.15.160:0/2856966637 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa09c0f5410 0x7fa09c0f5830 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fa08c009fb0 tx=0x7fa08c009b30).stop 2023-11-17T05:29:04.342 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:04.340+0000 7fa0a3df7700 1 -- 172.21.15.160:0/2856966637 shutdown_connections 2023-11-17T05:29:04.342 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:04.340+0000 7fa0a3df7700 1 --2- 172.21.15.160:0/2856966637 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa09c0f5410 0x7fa09c0f5830 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:29:04.342 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:04.340+0000 7fa0a3df7700 1 -- 172.21.15.160:0/2856966637 >> 172.21.15.160:0/2856966637 conn(0x7fa09c0f0b70 msgr2=0x7fa09c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:29:04.342 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:04.340+0000 7fa0a3df7700 1 -- 172.21.15.160:0/2856966637 shutdown_connections 2023-11-17T05:29:04.342 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:04.340+0000 7fa0a3df7700 1 -- 172.21.15.160:0/2856966637 wait complete. 2023-11-17T05:29:04.342 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:04.341+0000 7fa0a3df7700 1 Processor -- start 2023-11-17T05:29:04.342 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:04.341+0000 7fa0a3df7700 1 -- start start 2023-11-17T05:29:04.342 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:04.341+0000 7fa0a3df7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa09c0f5410 0x7fa09c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:29:04.342 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:04.341+0000 7fa0a3df7700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa09c106bd0 con 0x7fa09c0f5410 2023-11-17T05:29:04.342 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:04.341+0000 7fa0a1b93700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa09c0f5410 0x7fa09c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:29:04.342 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:04.341+0000 7fa0a1b93700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa09c0f5410 0x7fa09c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:50250/0 (socket says 172.21.15.160:50250) 2023-11-17T05:29:04.343 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:04.341+0000 7fa0a1b93700 1 -- 172.21.15.160:0/2113187417 learned_addr learned my addr 172.21.15.160:0/2113187417 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:29:04.343 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:04.341+0000 7fa0a1b93700 1 -- 172.21.15.160:0/2113187417 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa08c005040 con 0x7fa09c0f5410 2023-11-17T05:29:04.343 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:04.341+0000 7fa0a1b93700 1 --2- 172.21.15.160:0/2113187417 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa09c0f5410 0x7fa09c1066e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fa08c014000 tx=0x7fa08c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:29:04.343 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:04.342+0000 7fa0927fc700 1 -- 172.21.15.160:0/2113187417 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa08c0095f0 con 0x7fa09c0f5410 2023-11-17T05:29:04.343 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:04.342+0000 7fa0a3df7700 1 -- 172.21.15.160:0/2113187417 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa09c106dd0 con 0x7fa09c0f5410 2023-11-17T05:29:04.343 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:04.342+0000 7fa0927fc700 1 -- 172.21.15.160:0/2113187417 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa08c007e70 con 0x7fa09c0f5410 2023-11-17T05:29:04.343 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:04.342+0000 7fa0927fc700 1 -- 172.21.15.160:0/2113187417 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa08c014ea0 con 0x7fa09c0f5410 2023-11-17T05:29:04.343 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:04.342+0000 7fa0a3df7700 1 -- 172.21.15.160:0/2113187417 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa09c107260 con 0x7fa09c0f5410 2023-11-17T05:29:04.344 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:04.342+0000 7fa0927fc700 1 -- 172.21.15.160:0/2113187417 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa08c025070 con 0x7fa09c0f5410 2023-11-17T05:29:04.344 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:04.343+0000 7fa0a3df7700 1 -- 172.21.15.160:0/2113187417 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa09c045ad0 con 0x7fa09c0f5410 2023-11-17T05:29:04.344 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:04.343+0000 7fa0927fc700 1 --2- 172.21.15.160:0/2113187417 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa08802ff60 0x7fa088032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:29:04.344 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:04.343+0000 7fa0927fc700 1 -- 172.21.15.160:0/2113187417 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa08c040a30 con 0x7fa09c0f5410 2023-11-17T05:29:04.344 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:04.343+0000 7fa0a1392700 1 --2- 172.21.15.160:0/2113187417 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa08802ff60 0x7fa088032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:29:04.344 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:04.343+0000 7fa0a1392700 1 --2- 172.21.15.160:0/2113187417 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa08802ff60 0x7fa088032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fa0980096a0 tx=0x7fa098006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:29:04.347 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:04.348+0000 7fa0927fc700 1 -- 172.21.15.160:0/2113187417 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa08c0079c0 con 0x7fa09c0f5410 2023-11-17T05:29:04.553 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:04.553+0000 7fa0a3df7700 1 -- 172.21.15.160:0/2113187417 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa09c0fae60 con 0x7fa09c0f5410 2023-11-17T05:29:04.554 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:04.554+0000 7fa0927fc700 1 -- 172.21.15.160:0/2113187417 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fa08c00f300 con 0x7fa09c0f5410 2023-11-17T05:29:04.555 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:29:04.555 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:29:04.555 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:04.556+0000 7fa0a3df7700 1 -- 172.21.15.160:0/2113187417 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa08802ff60 msgr2=0x7fa088032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:29:04.555 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:04.556+0000 7fa0a3df7700 1 --2- 172.21.15.160:0/2113187417 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa08802ff60 0x7fa088032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fa0980096a0 tx=0x7fa098006b40).stop 2023-11-17T05:29:04.556 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:04.556+0000 7fa0a3df7700 1 -- 172.21.15.160:0/2113187417 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa09c0f5410 msgr2=0x7fa09c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:29:04.556 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:04.556+0000 7fa0a3df7700 1 --2- 172.21.15.160:0/2113187417 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa09c0f5410 0x7fa09c1066e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fa08c014000 tx=0x7fa08c00b040).stop 2023-11-17T05:29:04.556 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:04.556+0000 7fa0a3df7700 1 -- 172.21.15.160:0/2113187417 shutdown_connections 2023-11-17T05:29:04.556 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:04.556+0000 7fa0a3df7700 1 --2- 172.21.15.160:0/2113187417 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa08802ff60 0x7fa088032420 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:29:04.556 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:04.556+0000 7fa0a3df7700 1 --2- 172.21.15.160:0/2113187417 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa09c0f5410 0x7fa09c1066e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:29:04.556 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:04.556+0000 7fa0a3df7700 1 -- 172.21.15.160:0/2113187417 >> 172.21.15.160:0/2113187417 conn(0x7fa09c0f0b70 msgr2=0x7fa09c05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:29:04.556 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:04.556+0000 7fa0a3df7700 1 -- 172.21.15.160:0/2113187417 shutdown_connections 2023-11-17T05:29:04.556 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:04.556+0000 7fa0a3df7700 1 -- 172.21.15.160:0/2113187417 wait complete. 2023-11-17T05:29:04.556 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:29:04.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:29:04 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:29:03.423536+0000 mgr.smithi003.eddcuc (mgr.14164) 109 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:29:05.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:29:05 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:29:04.554262+0000 mon.smithi003 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.160:0/2113187417' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:29:05.960 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:29:05.961 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:29:06.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:29:06 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:29:05.423803+0000 mgr.smithi003.eddcuc (mgr.14164) 110 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:29:07.148 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:07.147+0000 7fab1f863700 1 -- 172.21.15.160:0/830497806 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab180f5410 msgr2=0x7fab180f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:29:07.148 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:07.147+0000 7fab1f863700 1 --2- 172.21.15.160:0/830497806 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab180f5410 0x7fab180f5830 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fab14009fb0 tx=0x7fab14009b30).stop 2023-11-17T05:29:07.148 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:07.147+0000 7fab1f863700 1 -- 172.21.15.160:0/830497806 shutdown_connections 2023-11-17T05:29:07.148 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:07.147+0000 7fab1f863700 1 --2- 172.21.15.160:0/830497806 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab180f5410 0x7fab180f5830 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:29:07.149 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:07.147+0000 7fab1f863700 1 -- 172.21.15.160:0/830497806 >> 172.21.15.160:0/830497806 conn(0x7fab180f0b70 msgr2=0x7fab180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:29:07.149 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:07.147+0000 7fab1f863700 1 -- 172.21.15.160:0/830497806 shutdown_connections 2023-11-17T05:29:07.149 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:07.147+0000 7fab1f863700 1 -- 172.21.15.160:0/830497806 wait complete. 2023-11-17T05:29:07.149 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:07.147+0000 7fab1f863700 1 Processor -- start 2023-11-17T05:29:07.149 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:07.147+0000 7fab1f863700 1 -- start start 2023-11-17T05:29:07.149 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:07.148+0000 7fab1f863700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab180f5410 0x7fab180fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:29:07.149 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:07.148+0000 7fab1f863700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab180fb820 con 0x7fab180f5410 2023-11-17T05:29:07.151 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:07.148+0000 7fab1d5ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab180f5410 0x7fab180fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:29:07.151 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:07.148+0000 7fab1d5ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab180f5410 0x7fab180fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:50264/0 (socket says 172.21.15.160:50264) 2023-11-17T05:29:07.151 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:07.148+0000 7fab1d5ff700 1 -- 172.21.15.160:0/1040349101 learned_addr learned my addr 172.21.15.160:0/1040349101 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:29:07.151 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:07.148+0000 7fab1d5ff700 1 -- 172.21.15.160:0/1040349101 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab14005040 con 0x7fab180f5410 2023-11-17T05:29:07.151 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:07.148+0000 7fab1d5ff700 1 --2- 172.21.15.160:0/1040349101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab180f5410 0x7fab180fd3a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fab14013510 tx=0x7fab1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:29:07.151 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:07.148+0000 7fab0dffb700 1 -- 172.21.15.160:0/1040349101 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab14009320 con 0x7fab180f5410 2023-11-17T05:29:07.151 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:07.148+0000 7fab1f863700 1 -- 172.21.15.160:0/1040349101 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab180fb9c0 con 0x7fab180f5410 2023-11-17T05:29:07.151 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:07.148+0000 7fab0dffb700 1 -- 172.21.15.160:0/1040349101 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fab14013f00 con 0x7fab180f5410 2023-11-17T05:29:07.151 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:07.148+0000 7fab0dffb700 1 -- 172.21.15.160:0/1040349101 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab14014d90 con 0x7fab180f5410 2023-11-17T05:29:07.151 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:07.149+0000 7fab1f863700 1 -- 172.21.15.160:0/1040349101 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab180fbe50 con 0x7fab180f5410 2023-11-17T05:29:07.151 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:07.149+0000 7fab1f863700 1 -- 172.21.15.160:0/1040349101 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab18059100 con 0x7fab180f5410 2023-11-17T05:29:07.152 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:07.150+0000 7fab0dffb700 1 -- 172.21.15.160:0/1040349101 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fab140095f0 con 0x7fab180f5410 2023-11-17T05:29:07.152 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:07.150+0000 7fab0dffb700 1 --2- 172.21.15.160:0/1040349101 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fab0402ff60 0x7fab04032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:29:07.152 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:07.150+0000 7fab0dffb700 1 -- 172.21.15.160:0/1040349101 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fab14041080 con 0x7fab180f5410 2023-11-17T05:29:07.152 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:07.150+0000 7fab1cdfe700 1 --2- 172.21.15.160:0/1040349101 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fab0402ff60 0x7fab04032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:29:07.152 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:07.151+0000 7fab1cdfe700 1 --2- 172.21.15.160:0/1040349101 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fab0402ff60 0x7fab04032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fab080096a0 tx=0x7fab08006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:29:07.155 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:07.155+0000 7fab0dffb700 1 -- 172.21.15.160:0/1040349101 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fab1401d7f0 con 0x7fab180f5410 2023-11-17T05:29:07.363 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:07.362+0000 7fab1f863700 1 -- 172.21.15.160:0/1040349101 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fab180fae60 con 0x7fab180f5410 2023-11-17T05:29:07.363 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:07.363+0000 7fab0dffb700 1 -- 172.21.15.160:0/1040349101 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fab14010230 con 0x7fab180f5410 2023-11-17T05:29:07.365 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:29:07.365 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:29:07.366 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:07.364+0000 7fab1f863700 1 -- 172.21.15.160:0/1040349101 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fab0402ff60 msgr2=0x7fab04032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:29:07.366 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:07.364+0000 7fab1f863700 1 --2- 172.21.15.160:0/1040349101 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fab0402ff60 0x7fab04032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fab080096a0 tx=0x7fab08006b40).stop 2023-11-17T05:29:07.366 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:07.364+0000 7fab1f863700 1 -- 172.21.15.160:0/1040349101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab180f5410 msgr2=0x7fab180fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:29:07.366 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:07.364+0000 7fab1f863700 1 --2- 172.21.15.160:0/1040349101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab180f5410 0x7fab180fd3a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fab14013510 tx=0x7fab1400b040).stop 2023-11-17T05:29:07.366 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:07.364+0000 7fab1f863700 1 -- 172.21.15.160:0/1040349101 shutdown_connections 2023-11-17T05:29:07.366 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:07.364+0000 7fab1f863700 1 --2- 172.21.15.160:0/1040349101 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fab0402ff60 0x7fab04032420 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:29:07.366 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:07.364+0000 7fab1f863700 1 --2- 172.21.15.160:0/1040349101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab180f5410 0x7fab180fd3a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:29:07.366 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:07.364+0000 7fab1f863700 1 -- 172.21.15.160:0/1040349101 >> 172.21.15.160:0/1040349101 conn(0x7fab180f0b70 msgr2=0x7fab1805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:29:07.367 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:07.364+0000 7fab1f863700 1 -- 172.21.15.160:0/1040349101 shutdown_connections 2023-11-17T05:29:07.367 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:07.364+0000 7fab1f863700 1 -- 172.21.15.160:0/1040349101 wait complete. 2023-11-17T05:29:07.367 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:29:07.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:29:07 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:29:07.363782+0000 mon.smithi003 (mon.0) 159 : audit [DBG] from='client.? 172.21.15.160:0/1040349101' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:29:08.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:29:08 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:29:07.424086+0000 mgr.smithi003.eddcuc (mgr.14164) 111 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:29:08.790 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:29:08.790 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:29:09.949 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:09.946+0000 7fab3719b700 1 -- 172.21.15.160:0/2876398244 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab300f6510 msgr2=0x7fab300f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:29:09.949 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:09.946+0000 7fab3719b700 1 --2- 172.21.15.160:0/2876398244 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab300f6510 0x7fab300f6930 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fab24009fb0 tx=0x7fab24009b30).stop 2023-11-17T05:29:09.949 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:09.946+0000 7fab3719b700 1 -- 172.21.15.160:0/2876398244 shutdown_connections 2023-11-17T05:29:09.949 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:09.946+0000 7fab3719b700 1 --2- 172.21.15.160:0/2876398244 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab300f6510 0x7fab300f6930 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:29:09.950 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:09.946+0000 7fab3719b700 1 -- 172.21.15.160:0/2876398244 >> 172.21.15.160:0/2876398244 conn(0x7fab300f1c70 msgr2=0x7fab300f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:29:09.950 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:09.946+0000 7fab3719b700 1 -- 172.21.15.160:0/2876398244 shutdown_connections 2023-11-17T05:29:09.950 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:09.946+0000 7fab3719b700 1 -- 172.21.15.160:0/2876398244 wait complete. 2023-11-17T05:29:09.950 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:09.947+0000 7fab3719b700 1 Processor -- start 2023-11-17T05:29:09.950 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:09.947+0000 7fab3719b700 1 -- start start 2023-11-17T05:29:09.950 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:09.947+0000 7fab3719b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab300f6510 0x7fab30106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:29:09.950 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:09.947+0000 7fab3719b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab30106c20 con 0x7fab300f6510 2023-11-17T05:29:09.950 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:09.947+0000 7fab34f37700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab300f6510 0x7fab30106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:29:09.950 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:09.947+0000 7fab34f37700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab300f6510 0x7fab30106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:50280/0 (socket says 172.21.15.160:50280) 2023-11-17T05:29:09.950 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:09.947+0000 7fab34f37700 1 -- 172.21.15.160:0/1400227703 learned_addr learned my addr 172.21.15.160:0/1400227703 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:29:09.950 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:09.947+0000 7fab34f37700 1 -- 172.21.15.160:0/1400227703 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab24005040 con 0x7fab300f6510 2023-11-17T05:29:09.950 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:09.948+0000 7fab34f37700 1 --2- 172.21.15.160:0/1400227703 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab300f6510 0x7fab30106730 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fab24013510 tx=0x7fab2400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:29:09.950 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:09.948+0000 7fab2d7fa700 1 -- 172.21.15.160:0/1400227703 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab24009320 con 0x7fab300f6510 2023-11-17T05:29:09.951 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:09.948+0000 7fab3719b700 1 -- 172.21.15.160:0/1400227703 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab30106e20 con 0x7fab300f6510 2023-11-17T05:29:09.951 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:09.948+0000 7fab2d7fa700 1 -- 172.21.15.160:0/1400227703 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fab24013f00 con 0x7fab300f6510 2023-11-17T05:29:09.951 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:09.948+0000 7fab2d7fa700 1 -- 172.21.15.160:0/1400227703 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab24014d90 con 0x7fab300f6510 2023-11-17T05:29:09.951 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:09.948+0000 7fab3719b700 1 -- 172.21.15.160:0/1400227703 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab301072b0 con 0x7fab300f6510 2023-11-17T05:29:09.951 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:09.949+0000 7fab2d7fa700 1 -- 172.21.15.160:0/1400227703 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fab240095f0 con 0x7fab300f6510 2023-11-17T05:29:09.952 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:09.949+0000 7fab3719b700 1 -- 172.21.15.160:0/1400227703 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab30045ad0 con 0x7fab300f6510 2023-11-17T05:29:09.952 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:09.949+0000 7fab2d7fa700 1 --2- 172.21.15.160:0/1400227703 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fab2002ff60 0x7fab20032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:29:09.952 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:09.949+0000 7fab2d7fa700 1 -- 172.21.15.160:0/1400227703 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fab24041080 con 0x7fab300f6510 2023-11-17T05:29:09.952 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:09.949+0000 7fab2ffff700 1 --2- 172.21.15.160:0/1400227703 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fab2002ff60 0x7fab20032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:29:09.952 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:09.949+0000 7fab2ffff700 1 --2- 172.21.15.160:0/1400227703 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fab2002ff60 0x7fab20032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fab1c0096a0 tx=0x7fab1c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:29:09.955 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:09.954+0000 7fab2d7fa700 1 -- 172.21.15.160:0/1400227703 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fab2401d7f0 con 0x7fab300f6510 2023-11-17T05:29:10.162 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:10.161+0000 7fab3719b700 1 -- 172.21.15.160:0/1400227703 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fab300fbf60 con 0x7fab300f6510 2023-11-17T05:29:10.162 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:10.162+0000 7fab2d7fa700 1 -- 172.21.15.160:0/1400227703 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fab24010230 con 0x7fab300f6510 2023-11-17T05:29:10.163 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:29:10.163 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:29:10.165 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:10.163+0000 7fab3719b700 1 -- 172.21.15.160:0/1400227703 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fab2002ff60 msgr2=0x7fab20032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:29:10.165 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:10.163+0000 7fab3719b700 1 --2- 172.21.15.160:0/1400227703 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fab2002ff60 0x7fab20032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fab1c0096a0 tx=0x7fab1c006b40).stop 2023-11-17T05:29:10.165 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:10.163+0000 7fab3719b700 1 -- 172.21.15.160:0/1400227703 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab300f6510 msgr2=0x7fab30106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:29:10.165 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:10.163+0000 7fab3719b700 1 --2- 172.21.15.160:0/1400227703 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab300f6510 0x7fab30106730 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fab24013510 tx=0x7fab2400b040).stop 2023-11-17T05:29:10.165 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:10.163+0000 7fab3719b700 1 -- 172.21.15.160:0/1400227703 shutdown_connections 2023-11-17T05:29:10.165 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:10.163+0000 7fab3719b700 1 --2- 172.21.15.160:0/1400227703 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fab2002ff60 0x7fab20032420 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:29:10.165 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:10.163+0000 7fab3719b700 1 --2- 172.21.15.160:0/1400227703 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab300f6510 0x7fab30106730 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:29:10.165 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:10.163+0000 7fab3719b700 1 -- 172.21.15.160:0/1400227703 >> 172.21.15.160:0/1400227703 conn(0x7fab300f1c70 msgr2=0x7fab300f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:29:10.166 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:10.163+0000 7fab3719b700 1 -- 172.21.15.160:0/1400227703 shutdown_connections 2023-11-17T05:29:10.166 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:10.163+0000 7fab3719b700 1 -- 172.21.15.160:0/1400227703 wait complete. 2023-11-17T05:29:10.166 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:29:10.271 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:29:09 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:29:09.424427+0000 mgr.smithi003.eddcuc (mgr.14164) 112 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:29:11.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:29:10 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:29:10.163029+0000 mon.smithi003 (mon.0) 160 : audit [DBG] from='client.? 172.21.15.160:0/1400227703' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:29:11.542 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:29:11.542 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:29:12.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:29:11 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:29:11.424836+0000 mgr.smithi003.eddcuc (mgr.14164) 113 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:29:12.716 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:12.712+0000 7f0f181a6700 1 -- 172.21.15.160:0/2178908715 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f100f6510 msgr2=0x7f0f100f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:29:12.716 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:12.712+0000 7f0f181a6700 1 --2- 172.21.15.160:0/2178908715 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f100f6510 0x7f0f100f6930 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f0f00009fb0 tx=0x7f0f00009b30).stop 2023-11-17T05:29:12.716 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:12.712+0000 7f0f181a6700 1 -- 172.21.15.160:0/2178908715 shutdown_connections 2023-11-17T05:29:12.716 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:12.712+0000 7f0f181a6700 1 --2- 172.21.15.160:0/2178908715 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f100f6510 0x7f0f100f6930 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:29:12.716 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:12.712+0000 7f0f181a6700 1 -- 172.21.15.160:0/2178908715 >> 172.21.15.160:0/2178908715 conn(0x7f0f100f1c70 msgr2=0x7f0f100f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:29:12.716 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:12.712+0000 7f0f181a6700 1 -- 172.21.15.160:0/2178908715 shutdown_connections 2023-11-17T05:29:12.725 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:12.712+0000 7f0f181a6700 1 -- 172.21.15.160:0/2178908715 wait complete. 2023-11-17T05:29:12.725 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:12.712+0000 7f0f181a6700 1 Processor -- start 2023-11-17T05:29:12.725 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:12.712+0000 7f0f181a6700 1 -- start start 2023-11-17T05:29:12.725 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:12.713+0000 7f0f181a6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f100f6510 0x7f0f10106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:29:12.725 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:12.713+0000 7f0f181a6700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f10106c20 con 0x7f0f100f6510 2023-11-17T05:29:12.725 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:12.713+0000 7f0f15f42700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f100f6510 0x7f0f10106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:29:12.725 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:12.713+0000 7f0f15f42700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f100f6510 0x7f0f10106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:58292/0 (socket says 172.21.15.160:58292) 2023-11-17T05:29:12.725 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:12.713+0000 7f0f15f42700 1 -- 172.21.15.160:0/2308733068 learned_addr learned my addr 172.21.15.160:0/2308733068 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:29:12.726 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:12.713+0000 7f0f15f42700 1 -- 172.21.15.160:0/2308733068 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f00005040 con 0x7f0f100f6510 2023-11-17T05:29:12.726 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:12.713+0000 7f0f15f42700 1 --2- 172.21.15.160:0/2308733068 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f100f6510 0x7f0f10106730 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f0f00013510 tx=0x7f0f0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:29:12.726 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:12.713+0000 7f0f067fc700 1 -- 172.21.15.160:0/2308733068 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f00009320 con 0x7f0f100f6510 2023-11-17T05:29:12.726 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:12.713+0000 7f0f181a6700 1 -- 172.21.15.160:0/2308733068 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f10106e20 con 0x7f0f100f6510 2023-11-17T05:29:12.726 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:12.713+0000 7f0f067fc700 1 -- 172.21.15.160:0/2308733068 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0f00013f00 con 0x7f0f100f6510 2023-11-17T05:29:12.726 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:12.713+0000 7f0f067fc700 1 -- 172.21.15.160:0/2308733068 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f00014d90 con 0x7f0f100f6510 2023-11-17T05:29:12.726 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:12.714+0000 7f0f181a6700 1 -- 172.21.15.160:0/2308733068 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f101072b0 con 0x7f0f100f6510 2023-11-17T05:29:12.726 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:12.714+0000 7f0f067fc700 1 -- 172.21.15.160:0/2308733068 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0f000095f0 con 0x7f0f100f6510 2023-11-17T05:29:12.726 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:12.714+0000 7f0f181a6700 1 -- 172.21.15.160:0/2308733068 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f10045ad0 con 0x7f0f100f6510 2023-11-17T05:29:12.726 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:12.714+0000 7f0f067fc700 1 --2- 172.21.15.160:0/2308733068 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f0efc02ffb0 0x7f0efc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:29:12.726 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:12.715+0000 7f0f067fc700 1 -- 172.21.15.160:0/2308733068 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0f000410c0 con 0x7f0f100f6510 2023-11-17T05:29:12.726 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:12.715+0000 7f0f15741700 1 --2- 172.21.15.160:0/2308733068 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f0efc02ffb0 0x7f0efc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:29:12.727 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:12.715+0000 7f0f15741700 1 --2- 172.21.15.160:0/2308733068 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f0efc02ffb0 0x7f0efc032470 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f0f0c0096a0 tx=0x7f0f0c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:29:12.727 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:12.720+0000 7f0f067fc700 1 -- 172.21.15.160:0/2308733068 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0f00014420 con 0x7f0f100f6510 2023-11-17T05:29:12.925 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:12.924+0000 7f0f181a6700 1 -- 172.21.15.160:0/2308733068 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0f100fbf60 con 0x7f0f100f6510 2023-11-17T05:29:12.926 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:12.925+0000 7f0f067fc700 1 -- 172.21.15.160:0/2308733068 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f0f00010230 con 0x7f0f100f6510 2023-11-17T05:29:12.926 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:29:12.926 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:29:12.928 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:12.927+0000 7f0f181a6700 1 -- 172.21.15.160:0/2308733068 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f0efc02ffb0 msgr2=0x7f0efc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:29:12.928 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:12.927+0000 7f0f181a6700 1 --2- 172.21.15.160:0/2308733068 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f0efc02ffb0 0x7f0efc032470 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f0f0c0096a0 tx=0x7f0f0c006b40).stop 2023-11-17T05:29:12.928 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:12.927+0000 7f0f181a6700 1 -- 172.21.15.160:0/2308733068 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f100f6510 msgr2=0x7f0f10106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:29:12.928 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:12.927+0000 7f0f181a6700 1 --2- 172.21.15.160:0/2308733068 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f100f6510 0x7f0f10106730 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f0f00013510 tx=0x7f0f0000b040).stop 2023-11-17T05:29:12.928 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:12.927+0000 7f0f181a6700 1 -- 172.21.15.160:0/2308733068 shutdown_connections 2023-11-17T05:29:12.928 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:12.927+0000 7f0f181a6700 1 --2- 172.21.15.160:0/2308733068 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f0efc02ffb0 0x7f0efc032470 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:29:12.928 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:12.927+0000 7f0f181a6700 1 --2- 172.21.15.160:0/2308733068 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f100f6510 0x7f0f10106730 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:29:12.928 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:12.927+0000 7f0f181a6700 1 -- 172.21.15.160:0/2308733068 >> 172.21.15.160:0/2308733068 conn(0x7f0f100f1c70 msgr2=0x7f0f100f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:29:12.928 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:12.927+0000 7f0f181a6700 1 -- 172.21.15.160:0/2308733068 shutdown_connections 2023-11-17T05:29:12.928 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:12.927+0000 7f0f181a6700 1 -- 172.21.15.160:0/2308733068 wait complete. 2023-11-17T05:29:12.929 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:29:13.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:29:12 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:29:12.925947+0000 mon.smithi003 (mon.0) 161 : audit [DBG] from='client.? 172.21.15.160:0/2308733068' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:29:14.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:29:13 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:29:13.425155+0000 mgr.smithi003.eddcuc (mgr.14164) 114 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:29:14.309 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:29:14.310 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:29:15.505 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:15.494+0000 7f15540dd700 1 -- 172.21.15.160:0/878401750 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f154c0f6510 msgr2=0x7f154c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:29:15.505 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:15.494+0000 7f15540dd700 1 --2- 172.21.15.160:0/878401750 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f154c0f6510 0x7f154c0f6930 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f153c009960 tx=0x7f153c004d10).stop 2023-11-17T05:29:15.505 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:15.494+0000 7f15540dd700 1 -- 172.21.15.160:0/878401750 shutdown_connections 2023-11-17T05:29:15.506 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:15.494+0000 7f15540dd700 1 --2- 172.21.15.160:0/878401750 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f154c0f6510 0x7f154c0f6930 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:29:15.506 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:15.494+0000 7f15540dd700 1 -- 172.21.15.160:0/878401750 >> 172.21.15.160:0/878401750 conn(0x7f154c0f1c70 msgr2=0x7f154c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:29:15.506 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:15.494+0000 7f15540dd700 1 -- 172.21.15.160:0/878401750 shutdown_connections 2023-11-17T05:29:15.506 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:15.494+0000 7f15540dd700 1 -- 172.21.15.160:0/878401750 wait complete. 2023-11-17T05:29:15.506 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:15.495+0000 7f15540dd700 1 Processor -- start 2023-11-17T05:29:15.506 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:15.495+0000 7f15540dd700 1 -- start start 2023-11-17T05:29:15.506 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:15.495+0000 7f15540dd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f154c0f6510 0x7f154c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:29:15.506 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:15.495+0000 7f15540dd700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f154c106c20 con 0x7f154c0f6510 2023-11-17T05:29:15.506 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:15.495+0000 7f1551e79700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f154c0f6510 0x7f154c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:29:15.506 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:15.495+0000 7f1551e79700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f154c0f6510 0x7f154c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:58296/0 (socket says 172.21.15.160:58296) 2023-11-17T05:29:15.506 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:15.495+0000 7f1551e79700 1 -- 172.21.15.160:0/1084938270 learned_addr learned my addr 172.21.15.160:0/1084938270 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:29:15.506 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:15.495+0000 7f1551e79700 1 -- 172.21.15.160:0/1084938270 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f153c009640 con 0x7f154c0f6510 2023-11-17T05:29:15.506 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:15.495+0000 7f1551e79700 1 --2- 172.21.15.160:0/1084938270 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f154c0f6510 0x7f154c106730 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f153c004b70 tx=0x7f153c004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:29:15.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:15.496+0000 7f154a7fc700 1 -- 172.21.15.160:0/1084938270 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f153c018070 con 0x7f154c0f6510 2023-11-17T05:29:15.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:15.496+0000 7f15540dd700 1 -- 172.21.15.160:0/1084938270 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f154c106e20 con 0x7f154c0f6510 2023-11-17T05:29:15.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:15.496+0000 7f154a7fc700 1 -- 172.21.15.160:0/1084938270 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f153c00b870 con 0x7f154c0f6510 2023-11-17T05:29:15.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:15.496+0000 7f154a7fc700 1 -- 172.21.15.160:0/1084938270 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f153c01c560 con 0x7f154c0f6510 2023-11-17T05:29:15.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:15.496+0000 7f15540dd700 1 -- 172.21.15.160:0/1084938270 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f154c1072b0 con 0x7f154c0f6510 2023-11-17T05:29:15.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:15.496+0000 7f154a7fc700 1 -- 172.21.15.160:0/1084938270 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f153c013630 con 0x7f154c0f6510 2023-11-17T05:29:15.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:15.496+0000 7f15540dd700 1 -- 172.21.15.160:0/1084938270 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f154c045ad0 con 0x7f154c0f6510 2023-11-17T05:29:15.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:15.497+0000 7f154a7fc700 1 --2- 172.21.15.160:0/1084938270 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f153402ff60 0x7f1534032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:29:15.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:15.497+0000 7f154a7fc700 1 -- 172.21.15.160:0/1084938270 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f153c041290 con 0x7f154c0f6510 2023-11-17T05:29:15.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:15.497+0000 7f1551678700 1 --2- 172.21.15.160:0/1084938270 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f153402ff60 0x7f1534032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:29:15.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:15.497+0000 7f1551678700 1 --2- 172.21.15.160:0/1084938270 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f153402ff60 0x7f1534032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f15400096a0 tx=0x7f1540006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:29:15.508 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:15.502+0000 7f154a7fc700 1 -- 172.21.15.160:0/1084938270 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f153c004810 con 0x7f154c0f6510 2023-11-17T05:29:15.708 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:15.707+0000 7f15540dd700 1 -- 172.21.15.160:0/1084938270 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f154c0fbf60 con 0x7f154c0f6510 2023-11-17T05:29:15.708 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:15.708+0000 7f154a7fc700 1 -- 172.21.15.160:0/1084938270 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f153c012380 con 0x7f154c0f6510 2023-11-17T05:29:15.709 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:29:15.709 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:29:15.710 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:15.709+0000 7f15540dd700 1 -- 172.21.15.160:0/1084938270 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f153402ff60 msgr2=0x7f1534032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:29:15.710 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:15.709+0000 7f15540dd700 1 --2- 172.21.15.160:0/1084938270 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f153402ff60 0x7f1534032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f15400096a0 tx=0x7f1540006b40).stop 2023-11-17T05:29:15.711 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:15.709+0000 7f15540dd700 1 -- 172.21.15.160:0/1084938270 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f154c0f6510 msgr2=0x7f154c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:29:15.711 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:15.709+0000 7f15540dd700 1 --2- 172.21.15.160:0/1084938270 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f154c0f6510 0x7f154c106730 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f153c004b70 tx=0x7f153c004d10).stop 2023-11-17T05:29:15.711 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:15.709+0000 7f15540dd700 1 -- 172.21.15.160:0/1084938270 shutdown_connections 2023-11-17T05:29:15.711 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:15.709+0000 7f15540dd700 1 --2- 172.21.15.160:0/1084938270 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f153402ff60 0x7f1534032420 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:29:15.711 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:15.709+0000 7f15540dd700 1 --2- 172.21.15.160:0/1084938270 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f154c0f6510 0x7f154c106730 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:29:15.711 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:15.709+0000 7f15540dd700 1 -- 172.21.15.160:0/1084938270 >> 172.21.15.160:0/1084938270 conn(0x7f154c0f1c70 msgr2=0x7f154c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:29:15.711 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:15.709+0000 7f15540dd700 1 -- 172.21.15.160:0/1084938270 shutdown_connections 2023-11-17T05:29:15.711 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:15.709+0000 7f15540dd700 1 -- 172.21.15.160:0/1084938270 wait complete. 2023-11-17T05:29:15.711 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:29:16.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:29:16 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:29:15.425477+0000 mgr.smithi003.eddcuc (mgr.14164) 115 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:29:16.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:29:16 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:29:15.708968+0000 mon.smithi003 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.160:0/1084938270' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:29:17.125 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:29:17.125 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:29:18.311 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:18.307+0000 7f19862d8700 1 -- 172.21.15.160:0/2185216721 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19800f5410 msgr2=0x7f19800f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:29:18.311 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:18.307+0000 7f19862d8700 1 --2- 172.21.15.160:0/2185216721 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19800f5410 0x7f19800f5830 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f1974009fb0 tx=0x7f1974009b30).stop 2023-11-17T05:29:18.312 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:18.307+0000 7f19862d8700 1 -- 172.21.15.160:0/2185216721 shutdown_connections 2023-11-17T05:29:18.312 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:18.307+0000 7f19862d8700 1 --2- 172.21.15.160:0/2185216721 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19800f5410 0x7f19800f5830 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:29:18.312 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:18.307+0000 7f19862d8700 1 -- 172.21.15.160:0/2185216721 >> 172.21.15.160:0/2185216721 conn(0x7f19800f0b70 msgr2=0x7f19800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:29:18.312 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:18.307+0000 7f19862d8700 1 -- 172.21.15.160:0/2185216721 shutdown_connections 2023-11-17T05:29:18.312 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:18.307+0000 7f19862d8700 1 -- 172.21.15.160:0/2185216721 wait complete. 2023-11-17T05:29:18.312 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:18.307+0000 7f19862d8700 1 Processor -- start 2023-11-17T05:29:18.312 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:18.308+0000 7f19862d8700 1 -- start start 2023-11-17T05:29:18.312 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:18.308+0000 7f19862d8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19800f5410 0x7f19800fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:29:18.312 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:18.308+0000 7f19862d8700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19800fb820 con 0x7f19800f5410 2023-11-17T05:29:18.312 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:18.308+0000 7f197ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19800f5410 0x7f19800fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:29:18.327 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:18.308+0000 7f197ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19800f5410 0x7f19800fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:58320/0 (socket says 172.21.15.160:58320) 2023-11-17T05:29:18.327 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:18.308+0000 7f197ffff700 1 -- 172.21.15.160:0/2106951637 learned_addr learned my addr 172.21.15.160:0/2106951637 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:29:18.327 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:18.308+0000 7f197ffff700 1 -- 172.21.15.160:0/2106951637 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1974005040 con 0x7f19800f5410 2023-11-17T05:29:18.327 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:18.308+0000 7f197ffff700 1 --2- 172.21.15.160:0/2106951637 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19800f5410 0x7f19800fd3a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f1974009f80 tx=0x7f197400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:29:18.327 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:18.309+0000 7f197cff9700 1 -- 172.21.15.160:0/2106951637 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1974009320 con 0x7f19800f5410 2023-11-17T05:29:18.327 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:18.309+0000 7f19862d8700 1 -- 172.21.15.160:0/2106951637 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19800fb9c0 con 0x7f19800f5410 2023-11-17T05:29:18.327 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:18.309+0000 7f197cff9700 1 -- 172.21.15.160:0/2106951637 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1974013ec0 con 0x7f19800f5410 2023-11-17T05:29:18.327 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:18.309+0000 7f197cff9700 1 -- 172.21.15.160:0/2106951637 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f197401ece0 con 0x7f19800f5410 2023-11-17T05:29:18.327 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:18.309+0000 7f19862d8700 1 -- 172.21.15.160:0/2106951637 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19800fbe50 con 0x7f19800f5410 2023-11-17T05:29:18.327 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:18.310+0000 7f197cff9700 1 -- 172.21.15.160:0/2106951637 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f197401e420 con 0x7f19800f5410 2023-11-17T05:29:18.327 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:18.310+0000 7f19862d8700 1 -- 172.21.15.160:0/2106951637 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1980059100 con 0x7f19800f5410 2023-11-17T05:29:18.328 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:18.310+0000 7f197cff9700 1 --2- 172.21.15.160:0/2106951637 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f196802ff60 0x7f1968032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:29:18.328 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:18.310+0000 7f197cff9700 1 -- 172.21.15.160:0/2106951637 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1974040d70 con 0x7f19800f5410 2023-11-17T05:29:18.328 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:18.310+0000 7f197f7fe700 1 --2- 172.21.15.160:0/2106951637 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f196802ff60 0x7f1968032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:29:18.328 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:18.311+0000 7f197f7fe700 1 --2- 172.21.15.160:0/2106951637 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f196802ff60 0x7f1968032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f19700096a0 tx=0x7f1970006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:29:18.328 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:18.315+0000 7f197cff9700 1 -- 172.21.15.160:0/2106951637 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f197401e730 con 0x7f19800f5410 2023-11-17T05:29:18.522 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:18.521+0000 7f19862d8700 1 -- 172.21.15.160:0/2106951637 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f19800fae60 con 0x7f19800f5410 2023-11-17T05:29:18.523 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:18.522+0000 7f197cff9700 1 -- 172.21.15.160:0/2106951637 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f197402c2c0 con 0x7f19800f5410 2023-11-17T05:29:18.523 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:29:18.523 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:29:18.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:18.523+0000 7f19862d8700 1 -- 172.21.15.160:0/2106951637 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f196802ff60 msgr2=0x7f1968032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:29:18.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:18.523+0000 7f19862d8700 1 --2- 172.21.15.160:0/2106951637 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f196802ff60 0x7f1968032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f19700096a0 tx=0x7f1970006b40).stop 2023-11-17T05:29:18.525 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:18.523+0000 7f19862d8700 1 -- 172.21.15.160:0/2106951637 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19800f5410 msgr2=0x7f19800fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:29:18.525 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:18.523+0000 7f19862d8700 1 --2- 172.21.15.160:0/2106951637 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19800f5410 0x7f19800fd3a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f1974009f80 tx=0x7f197400b040).stop 2023-11-17T05:29:18.525 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:18.523+0000 7f19862d8700 1 -- 172.21.15.160:0/2106951637 shutdown_connections 2023-11-17T05:29:18.525 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:18.523+0000 7f19862d8700 1 --2- 172.21.15.160:0/2106951637 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f196802ff60 0x7f1968032420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:29:18.525 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:18.523+0000 7f19862d8700 1 --2- 172.21.15.160:0/2106951637 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19800f5410 0x7f19800fd3a0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:29:18.525 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:18.523+0000 7f19862d8700 1 -- 172.21.15.160:0/2106951637 >> 172.21.15.160:0/2106951637 conn(0x7f19800f0b70 msgr2=0x7f198005fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:29:18.525 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:18.524+0000 7f19862d8700 1 -- 172.21.15.160:0/2106951637 shutdown_connections 2023-11-17T05:29:18.525 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:18.524+0000 7f19862d8700 1 -- 172.21.15.160:0/2106951637 wait complete. 2023-11-17T05:29:18.525 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:29:18.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:29:18 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:29:17.425833+0000 mgr.smithi003.eddcuc (mgr.14164) 116 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:29:19.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:29:19 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:29:18.523160+0000 mon.smithi003 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.160:0/2106951637' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:29:19.908 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:29:19.909 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:29:20.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:29:20 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:29:19.426149+0000 mgr.smithi003.eddcuc (mgr.14164) 117 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:29:21.085 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:21.084+0000 7fa40d1ba700 1 -- 172.21.15.160:0/2876352930 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4080f5410 msgr2=0x7fa4080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:29:21.085 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:21.084+0000 7fa40d1ba700 1 --2- 172.21.15.160:0/2876352930 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4080f5410 0x7fa4080f5830 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fa3f0009fb0 tx=0x7fa3f0009b30).stop 2023-11-17T05:29:21.086 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:21.084+0000 7fa40d1ba700 1 -- 172.21.15.160:0/2876352930 shutdown_connections 2023-11-17T05:29:21.086 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:21.084+0000 7fa40d1ba700 1 --2- 172.21.15.160:0/2876352930 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4080f5410 0x7fa4080f5830 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:29:21.086 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:21.084+0000 7fa40d1ba700 1 -- 172.21.15.160:0/2876352930 >> 172.21.15.160:0/2876352930 conn(0x7fa4080f0b70 msgr2=0x7fa4080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:29:21.086 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:21.084+0000 7fa40d1ba700 1 -- 172.21.15.160:0/2876352930 shutdown_connections 2023-11-17T05:29:21.086 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:21.084+0000 7fa40d1ba700 1 -- 172.21.15.160:0/2876352930 wait complete. 2023-11-17T05:29:21.088 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:21.085+0000 7fa40d1ba700 1 Processor -- start 2023-11-17T05:29:21.088 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:21.085+0000 7fa40d1ba700 1 -- start start 2023-11-17T05:29:21.088 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:21.085+0000 7fa40d1ba700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4080f5410 0x7fa408109390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:29:21.088 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:21.085+0000 7fa40d1ba700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa408060280 con 0x7fa4080f5410 2023-11-17T05:29:21.088 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:21.086+0000 7fa406d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4080f5410 0x7fa408109390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:29:21.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:21.086+0000 7fa406d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4080f5410 0x7fa408109390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:43146/0 (socket says 172.21.15.160:43146) 2023-11-17T05:29:21.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:21.086+0000 7fa406d9d700 1 -- 172.21.15.160:0/279260075 learned_addr learned my addr 172.21.15.160:0/279260075 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:29:21.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:21.086+0000 7fa406d9d700 1 -- 172.21.15.160:0/279260075 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f0005040 con 0x7fa4080f5410 2023-11-17T05:29:21.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:21.086+0000 7fa406d9d700 1 --2- 172.21.15.160:0/279260075 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4080f5410 0x7fa408109390 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fa3f0013510 tx=0x7fa3f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:29:21.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:21.086+0000 7fa3ff7fe700 1 -- 172.21.15.160:0/279260075 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3f0009320 con 0x7fa4080f5410 2023-11-17T05:29:21.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:21.086+0000 7fa40d1ba700 1 -- 172.21.15.160:0/279260075 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa408060480 con 0x7fa4080f5410 2023-11-17T05:29:21.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:21.086+0000 7fa3ff7fe700 1 -- 172.21.15.160:0/279260075 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa3f0013f00 con 0x7fa4080f5410 2023-11-17T05:29:21.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:21.086+0000 7fa3ff7fe700 1 -- 172.21.15.160:0/279260075 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3f0014d90 con 0x7fa4080f5410 2023-11-17T05:29:21.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:21.086+0000 7fa40d1ba700 1 -- 172.21.15.160:0/279260075 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa408109ac0 con 0x7fa4080f5410 2023-11-17T05:29:21.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:21.087+0000 7fa3ff7fe700 1 -- 172.21.15.160:0/279260075 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa3f00263a0 con 0x7fa4080f5410 2023-11-17T05:29:21.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:21.087+0000 7fa40d1ba700 1 -- 172.21.15.160:0/279260075 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa408045ad0 con 0x7fa4080f5410 2023-11-17T05:29:21.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:21.087+0000 7fa3ff7fe700 1 --2- 172.21.15.160:0/279260075 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa3f402ff60 0x7fa3f4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:29:21.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:21.087+0000 7fa3ff7fe700 1 -- 172.21.15.160:0/279260075 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa3f0041040 con 0x7fa4080f5410 2023-11-17T05:29:21.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:21.087+0000 7fa40659c700 1 --2- 172.21.15.160:0/279260075 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa3f402ff60 0x7fa3f4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:29:21.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:21.088+0000 7fa40659c700 1 --2- 172.21.15.160:0/279260075 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa3f402ff60 0x7fa3f4032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fa3f80096a0 tx=0x7fa3f8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:29:21.093 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:21.092+0000 7fa3ff7fe700 1 -- 172.21.15.160:0/279260075 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa3f00095f0 con 0x7fa4080f5410 2023-11-17T05:29:21.300 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:21.298+0000 7fa40d1ba700 1 -- 172.21.15.160:0/279260075 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa4080fae60 con 0x7fa4080f5410 2023-11-17T05:29:21.300 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:21.299+0000 7fa3ff7fe700 1 -- 172.21.15.160:0/279260075 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fa3f001db70 con 0x7fa4080f5410 2023-11-17T05:29:21.300 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:29:21.300 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:29:21.301 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:21.300+0000 7fa40d1ba700 1 -- 172.21.15.160:0/279260075 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa3f402ff60 msgr2=0x7fa3f4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:29:21.301 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:21.300+0000 7fa40d1ba700 1 --2- 172.21.15.160:0/279260075 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa3f402ff60 0x7fa3f4032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fa3f80096a0 tx=0x7fa3f8006b40).stop 2023-11-17T05:29:21.301 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:21.300+0000 7fa40d1ba700 1 -- 172.21.15.160:0/279260075 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4080f5410 msgr2=0x7fa408109390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:29:21.302 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:21.300+0000 7fa40d1ba700 1 --2- 172.21.15.160:0/279260075 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4080f5410 0x7fa408109390 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fa3f0013510 tx=0x7fa3f000b040).stop 2023-11-17T05:29:21.302 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:21.300+0000 7fa40d1ba700 1 -- 172.21.15.160:0/279260075 shutdown_connections 2023-11-17T05:29:21.302 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:21.300+0000 7fa40d1ba700 1 --2- 172.21.15.160:0/279260075 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa3f402ff60 0x7fa3f4032420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:29:21.302 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:21.300+0000 7fa40d1ba700 1 --2- 172.21.15.160:0/279260075 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4080f5410 0x7fa408109390 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:29:21.302 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:21.300+0000 7fa40d1ba700 1 -- 172.21.15.160:0/279260075 >> 172.21.15.160:0/279260075 conn(0x7fa4080f0b70 msgr2=0x7fa4080f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:29:21.302 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:21.300+0000 7fa40d1ba700 1 -- 172.21.15.160:0/279260075 shutdown_connections 2023-11-17T05:29:21.302 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:21.300+0000 7fa40d1ba700 1 -- 172.21.15.160:0/279260075 wait complete. 2023-11-17T05:29:21.303 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:29:21.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:29:21 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:29:21.300076+0000 mon.smithi003 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.160:0/279260075' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:29:22.694 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:29:22.695 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:29:22.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:29:22 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:29:21.426516+0000 mgr.smithi003.eddcuc (mgr.14164) 118 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:29:23.857 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:23.855+0000 7f3abe407700 1 -- 172.21.15.160:0/1502468226 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ab80f5410 msgr2=0x7f3ab80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:29:23.857 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:23.855+0000 7f3abe407700 1 --2- 172.21.15.160:0/1502468226 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ab80f5410 0x7f3ab80f5830 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f3aa8009fb0 tx=0x7f3aa8009b30).stop 2023-11-17T05:29:23.857 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:23.855+0000 7f3abe407700 1 -- 172.21.15.160:0/1502468226 shutdown_connections 2023-11-17T05:29:23.857 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:23.855+0000 7f3abe407700 1 --2- 172.21.15.160:0/1502468226 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ab80f5410 0x7f3ab80f5830 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:29:23.857 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:23.855+0000 7f3abe407700 1 -- 172.21.15.160:0/1502468226 >> 172.21.15.160:0/1502468226 conn(0x7f3ab80f0b70 msgr2=0x7f3ab80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:29:23.914 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:23.855+0000 7f3abe407700 1 -- 172.21.15.160:0/1502468226 shutdown_connections 2023-11-17T05:29:23.914 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:23.855+0000 7f3abe407700 1 -- 172.21.15.160:0/1502468226 wait complete. 2023-11-17T05:29:23.914 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:23.856+0000 7f3abe407700 1 Processor -- start 2023-11-17T05:29:23.945 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:23.856+0000 7f3abe407700 1 -- start start 2023-11-17T05:29:23.945 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:23.856+0000 7f3abe407700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ab80f5410 0x7f3ab80fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:29:23.945 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:23.856+0000 7f3abe407700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab80fb820 con 0x7f3ab80f5410 2023-11-17T05:29:23.945 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:23.856+0000 7f3ab7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ab80f5410 0x7f3ab80fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:29:23.945 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:23.856+0000 7f3ab7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ab80f5410 0x7f3ab80fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:43168/0 (socket says 172.21.15.160:43168) 2023-11-17T05:29:23.945 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:23.856+0000 7f3ab7fff700 1 -- 172.21.15.160:0/941621714 learned_addr learned my addr 172.21.15.160:0/941621714 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:29:23.947 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:23.857+0000 7f3ab7fff700 1 -- 172.21.15.160:0/941621714 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aa8005040 con 0x7f3ab80f5410 2023-11-17T05:29:23.947 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:23.857+0000 7f3ab7fff700 1 --2- 172.21.15.160:0/941621714 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ab80f5410 0x7f3ab80fd3a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f3aa8013510 tx=0x7f3aa800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:29:23.947 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:23.857+0000 7f3ab4ff9700 1 -- 172.21.15.160:0/941621714 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3aa8009320 con 0x7f3ab80f5410 2023-11-17T05:29:23.947 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:23.857+0000 7f3abe407700 1 -- 172.21.15.160:0/941621714 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ab80fb9c0 con 0x7f3ab80f5410 2023-11-17T05:29:23.947 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:23.857+0000 7f3ab4ff9700 1 -- 172.21.15.160:0/941621714 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3aa8013f00 con 0x7f3ab80f5410 2023-11-17T05:29:23.947 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:23.857+0000 7f3ab4ff9700 1 -- 172.21.15.160:0/941621714 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3aa8014d90 con 0x7f3ab80f5410 2023-11-17T05:29:23.947 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:23.857+0000 7f3abe407700 1 -- 172.21.15.160:0/941621714 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ab80fbe50 con 0x7f3ab80f5410 2023-11-17T05:29:23.947 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:23.858+0000 7f3ab4ff9700 1 -- 172.21.15.160:0/941621714 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3aa8014420 con 0x7f3ab80f5410 2023-11-17T05:29:23.947 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:23.858+0000 7f3abe407700 1 -- 172.21.15.160:0/941621714 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ab8059100 con 0x7f3ab80f5410 2023-11-17T05:29:23.948 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:23.858+0000 7f3ab4ff9700 1 --2- 172.21.15.160:0/941621714 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f3aa002ff60 0x7f3aa0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:29:23.948 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:23.858+0000 7f3ab4ff9700 1 -- 172.21.15.160:0/941621714 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3aa8041110 con 0x7f3ab80f5410 2023-11-17T05:29:23.948 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:23.858+0000 7f3ab77fe700 1 --2- 172.21.15.160:0/941621714 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f3aa002ff60 0x7f3aa0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:29:23.948 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:23.858+0000 7f3ab77fe700 1 --2- 172.21.15.160:0/941621714 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f3aa002ff60 0x7f3aa0032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f3aac0096a0 tx=0x7f3aac006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:29:23.948 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:23.863+0000 7f3ab4ff9700 1 -- 172.21.15.160:0/941621714 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3aa80095f0 con 0x7f3ab80f5410 2023-11-17T05:29:24.071 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:24.070+0000 7f3abe407700 1 -- 172.21.15.160:0/941621714 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3ab8045ad0 con 0x7f3ab80f5410 2023-11-17T05:29:24.072 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:24.071+0000 7f3ab4ff9700 1 -- 172.21.15.160:0/941621714 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f3aa801db70 con 0x7f3ab80f5410 2023-11-17T05:29:24.072 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:29:24.072 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:29:24.074 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:24.072+0000 7f3abe407700 1 -- 172.21.15.160:0/941621714 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f3aa002ff60 msgr2=0x7f3aa0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:29:24.074 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:24.072+0000 7f3abe407700 1 --2- 172.21.15.160:0/941621714 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f3aa002ff60 0x7f3aa0032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f3aac0096a0 tx=0x7f3aac006b40).stop 2023-11-17T05:29:24.074 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:24.072+0000 7f3abe407700 1 -- 172.21.15.160:0/941621714 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ab80f5410 msgr2=0x7f3ab80fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:29:24.074 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:24.072+0000 7f3abe407700 1 --2- 172.21.15.160:0/941621714 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ab80f5410 0x7f3ab80fd3a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f3aa8013510 tx=0x7f3aa800b040).stop 2023-11-17T05:29:24.074 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:24.072+0000 7f3abe407700 1 -- 172.21.15.160:0/941621714 shutdown_connections 2023-11-17T05:29:24.074 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:24.072+0000 7f3abe407700 1 --2- 172.21.15.160:0/941621714 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f3aa002ff60 0x7f3aa0032420 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:29:24.074 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:24.072+0000 7f3abe407700 1 --2- 172.21.15.160:0/941621714 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ab80f5410 0x7f3ab80fd3a0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:29:24.074 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:24.072+0000 7f3abe407700 1 -- 172.21.15.160:0/941621714 >> 172.21.15.160:0/941621714 conn(0x7f3ab80f0b70 msgr2=0x7f3ab805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:29:24.074 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:24.073+0000 7f3abe407700 1 -- 172.21.15.160:0/941621714 shutdown_connections 2023-11-17T05:29:24.074 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:29:24.073+0000 7f3abe407700 1 -- 172.21.15.160:0/941621714 wait complete. 2023-11-17T05:29:24.075 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:29:24.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:29:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:29:23.426867+0000 mgr.smithi003.eddcuc (mgr.14164) 119 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:29:24.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:29:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:29:24.072187+0000 mon.smithi003 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.160:0/941621714' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:29:25.466 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:30:23.790 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:30:23.821 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:29:26 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:29:25.427210+0000 mgr.smithi003.eddcuc (mgr.14164) 120 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:23.821 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:29:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:29:27.427555+0000 mgr.smithi003.eddcuc (mgr.14164) 121 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:23.821 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:29:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:29:27.549505+0000 mon.smithi003 (mon.0) 166 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.eddcuc/mirror_snapshot_schedule"}]: dispatch 2023-11-17T05:30:23.822 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:29:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:29:27.553336+0000 mon.smithi003 (mon.0) 167 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.eddcuc/trash_purge_schedule"}]: dispatch 2023-11-17T05:30:23.822 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:29:29 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:29:29.427876+0000 mgr.smithi003.eddcuc (mgr.14164) 122 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:23.822 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:29:32 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:29:31.428224+0000 mgr.smithi003.eddcuc (mgr.14164) 123 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:23.822 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:29:34 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:29:33.428514+0000 mgr.smithi003.eddcuc (mgr.14164) 124 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:23.822 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:29:36 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:29:35.428856+0000 mgr.smithi003.eddcuc (mgr.14164) 125 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:23.822 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:29:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:29:37.429212+0000 mgr.smithi003.eddcuc (mgr.14164) 126 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:23.822 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:29:39 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:29:39.429497+0000 mgr.smithi003.eddcuc (mgr.14164) 127 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:23.822 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:29:42 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:29:41.429758+0000 mgr.smithi003.eddcuc (mgr.14164) 128 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:23.822 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:29:44 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:29:43.430075+0000 mgr.smithi003.eddcuc (mgr.14164) 129 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:23.822 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:29:46 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:29:45.430345+0000 mgr.smithi003.eddcuc (mgr.14164) 130 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:23.822 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:29:48 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:29:47.430600+0000 mgr.smithi003.eddcuc (mgr.14164) 131 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:23.822 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:29:49 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:29:49.430904+0000 mgr.smithi003.eddcuc (mgr.14164) 132 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:23.823 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:29:52 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:29:51.431206+0000 mgr.smithi003.eddcuc (mgr.14164) 133 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:23.823 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:29:54 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:29:53.431522+0000 mgr.smithi003.eddcuc (mgr.14164) 134 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:23.823 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:29:56 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:29:55.431777+0000 mgr.smithi003.eddcuc (mgr.14164) 135 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:23.823 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:29:58 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:29:57.432102+0000 mgr.smithi003.eddcuc (mgr.14164) 136 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:23.823 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:29:59 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:29:59.432516+0000 mgr.smithi003.eddcuc (mgr.14164) 137 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:23.823 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:02 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:30:01.432893+0000 mgr.smithi003.eddcuc (mgr.14164) 138 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:23.823 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:04 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:30:03.433269+0000 mgr.smithi003.eddcuc (mgr.14164) 139 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:23.823 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:06 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:30:05.433704+0000 mgr.smithi003.eddcuc (mgr.14164) 140 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:23.823 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:08 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:30:07.434078+0000 mgr.smithi003.eddcuc (mgr.14164) 141 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:23.823 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:09 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:30:09.434451+0000 mgr.smithi003.eddcuc (mgr.14164) 142 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:23.823 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:12 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:30:11.434810+0000 mgr.smithi003.eddcuc (mgr.14164) 143 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:23.823 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:14 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:30:13.435127+0000 mgr.smithi003.eddcuc (mgr.14164) 144 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:23.823 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:16 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:30:15.435449+0000 mgr.smithi003.eddcuc (mgr.14164) 145 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:23.824 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:18 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:30:17.435831+0000 mgr.smithi003.eddcuc (mgr.14164) 146 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:23.824 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:19 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:30:19.436185+0000 mgr.smithi003.eddcuc (mgr.14164) 147 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:23.824 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:22 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:30:21.436541+0000 mgr.smithi003.eddcuc (mgr.14164) 148 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:24.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:30:23.436854+0000 mgr.smithi003.eddcuc (mgr.14164) 149 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:25.020 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:25.019+0000 7f71c7fff700 1 -- 172.21.15.160:0/1740920587 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71c80f62a0 msgr2=0x7f71c80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:30:25.020 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:25.019+0000 7f71c7fff700 1 --2- 172.21.15.160:0/1740920587 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71c80f62a0 0x7f71c80f66c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f71c0009960 tx=0x7f71c0004d10).stop 2023-11-17T05:30:25.020 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:25.019+0000 7f71c7fff700 1 -- 172.21.15.160:0/1740920587 shutdown_connections 2023-11-17T05:30:25.021 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:25.019+0000 7f71c7fff700 1 --2- 172.21.15.160:0/1740920587 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71c80f62a0 0x7f71c80f66c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:30:25.021 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:25.019+0000 7f71c7fff700 1 -- 172.21.15.160:0/1740920587 >> 172.21.15.160:0/1740920587 conn(0x7f71c80f1aa0 msgr2=0x7f71c80f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:30:25.021 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:25.019+0000 7f71c7fff700 1 -- 172.21.15.160:0/1740920587 shutdown_connections 2023-11-17T05:30:25.021 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:25.019+0000 7f71c7fff700 1 -- 172.21.15.160:0/1740920587 wait complete. 2023-11-17T05:30:25.022 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:25.020+0000 7f71c7fff700 1 Processor -- start 2023-11-17T05:30:25.022 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:25.020+0000 7f71c7fff700 1 -- start start 2023-11-17T05:30:25.022 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:25.020+0000 7f71c7fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71c80f62a0 0x7f71c81064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:30:25.022 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:25.020+0000 7f71c7fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c81069a0 con 0x7f71c80f62a0 2023-11-17T05:30:25.022 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:25.021+0000 7f71c6ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71c80f62a0 0x7f71c81064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:30:25.022 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:25.021+0000 7f71c6ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71c80f62a0 0x7f71c81064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:54044/0 (socket says 172.21.15.160:54044) 2023-11-17T05:30:25.023 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:25.021+0000 7f71c6ffd700 1 -- 172.21.15.160:0/1550529463 learned_addr learned my addr 172.21.15.160:0/1550529463 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:30:25.023 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:25.021+0000 7f71c6ffd700 1 -- 172.21.15.160:0/1550529463 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c0009640 con 0x7f71c80f62a0 2023-11-17T05:30:25.023 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:25.021+0000 7f71c6ffd700 1 --2- 172.21.15.160:0/1550529463 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71c80f62a0 0x7f71c81064b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f71c0007bd0 tx=0x7f71c0004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:30:25.023 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:25.021+0000 7f71bf7fe700 1 -- 172.21.15.160:0/1550529463 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71c0018070 con 0x7f71c80f62a0 2023-11-17T05:30:25.023 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:25.021+0000 7f71c7fff700 1 -- 172.21.15.160:0/1550529463 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71c8106ba0 con 0x7f71c80f62a0 2023-11-17T05:30:25.023 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:25.021+0000 7f71bf7fe700 1 -- 172.21.15.160:0/1550529463 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f71c000b870 con 0x7f71c80f62a0 2023-11-17T05:30:25.023 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:25.021+0000 7f71bf7fe700 1 -- 172.21.15.160:0/1550529463 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71c001d420 con 0x7f71c80f62a0 2023-11-17T05:30:25.023 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:25.021+0000 7f71c7fff700 1 -- 172.21.15.160:0/1550529463 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71c8107030 con 0x7f71c80f62a0 2023-11-17T05:30:25.024 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:25.022+0000 7f71bf7fe700 1 -- 172.21.15.160:0/1550529463 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f71c0024060 con 0x7f71c80f62a0 2023-11-17T05:30:25.024 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:25.022+0000 7f71c7fff700 1 -- 172.21.15.160:0/1550529463 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71c8045ad0 con 0x7f71c80f62a0 2023-11-17T05:30:25.024 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:25.022+0000 7f71bf7fe700 1 --2- 172.21.15.160:0/1550529463 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f71b402ff60 0x7f71b4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:30:25.024 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:25.022+0000 7f71bf7fe700 1 -- 172.21.15.160:0/1550529463 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f71c0041230 con 0x7f71c80f62a0 2023-11-17T05:30:25.024 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:25.023+0000 7f71c67fc700 1 --2- 172.21.15.160:0/1550529463 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f71b402ff60 0x7f71b4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:30:25.024 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:25.023+0000 7f71c67fc700 1 --2- 172.21.15.160:0/1550529463 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f71b402ff60 0x7f71b4032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f71b00096a0 tx=0x7f71b0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:30:25.033 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:25.027+0000 7f71bf7fe700 1 -- 172.21.15.160:0/1550529463 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f71c0025520 con 0x7f71c80f62a0 2023-11-17T05:30:25.233 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:25.232+0000 7f71c7fff700 1 -- 172.21.15.160:0/1550529463 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f71c8107e40 con 0x7f71c80f62a0 2023-11-17T05:30:25.234 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:25.233+0000 7f71bf7fe700 1 -- 172.21.15.160:0/1550529463 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f71c001a2d0 con 0x7f71c80f62a0 2023-11-17T05:30:25.234 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:30:25.234 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:30:25.236 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:25.235+0000 7f71c7fff700 1 -- 172.21.15.160:0/1550529463 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f71b402ff60 msgr2=0x7f71b4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:30:25.236 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:25.235+0000 7f71c7fff700 1 --2- 172.21.15.160:0/1550529463 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f71b402ff60 0x7f71b4032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f71b00096a0 tx=0x7f71b0006b40).stop 2023-11-17T05:30:25.236 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:25.235+0000 7f71c7fff700 1 -- 172.21.15.160:0/1550529463 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71c80f62a0 msgr2=0x7f71c81064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:30:25.236 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:25.235+0000 7f71c7fff700 1 --2- 172.21.15.160:0/1550529463 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71c80f62a0 0x7f71c81064b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f71c0007bd0 tx=0x7f71c0004d10).stop 2023-11-17T05:30:25.236 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:25.235+0000 7f71c7fff700 1 -- 172.21.15.160:0/1550529463 shutdown_connections 2023-11-17T05:30:25.236 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:25.235+0000 7f71c7fff700 1 --2- 172.21.15.160:0/1550529463 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f71b402ff60 0x7f71b4032420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:30:25.236 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:25.235+0000 7f71c7fff700 1 --2- 172.21.15.160:0/1550529463 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71c80f62a0 0x7f71c81064b0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:30:25.236 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:25.235+0000 7f71c7fff700 1 -- 172.21.15.160:0/1550529463 >> 172.21.15.160:0/1550529463 conn(0x7f71c80f1aa0 msgr2=0x7f71c80f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:30:25.236 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:25.235+0000 7f71c7fff700 1 -- 172.21.15.160:0/1550529463 shutdown_connections 2023-11-17T05:30:25.236 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:25.235+0000 7f71c7fff700 1 -- 172.21.15.160:0/1550529463 wait complete. 2023-11-17T05:30:25.237 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:30:25.771 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:25 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:30:25.234416+0000 mon.smithi003 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.160:0/1550529463' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:30:26.655 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:30:26.656 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:30:26.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:26 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:30:25.437187+0000 mgr.smithi003.eddcuc (mgr.14164) 150 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:27.814 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:27.799+0000 7fde257d3700 1 -- 172.21.15.160:0/328644377 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde200f6510 msgr2=0x7fde200f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:30:27.815 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:27.799+0000 7fde257d3700 1 --2- 172.21.15.160:0/328644377 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde200f6510 0x7fde200f6930 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fde08009fb0 tx=0x7fde08009b30).stop 2023-11-17T05:30:27.816 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:27.799+0000 7fde257d3700 1 -- 172.21.15.160:0/328644377 shutdown_connections 2023-11-17T05:30:27.816 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:27.799+0000 7fde257d3700 1 --2- 172.21.15.160:0/328644377 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde200f6510 0x7fde200f6930 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:30:27.817 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:27.799+0000 7fde257d3700 1 -- 172.21.15.160:0/328644377 >> 172.21.15.160:0/328644377 conn(0x7fde200f1c70 msgr2=0x7fde200f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:30:27.817 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:27.799+0000 7fde257d3700 1 -- 172.21.15.160:0/328644377 shutdown_connections 2023-11-17T05:30:27.818 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:27.799+0000 7fde257d3700 1 -- 172.21.15.160:0/328644377 wait complete. 2023-11-17T05:30:27.819 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:27.800+0000 7fde257d3700 1 Processor -- start 2023-11-17T05:30:27.820 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:27.800+0000 7fde257d3700 1 -- start start 2023-11-17T05:30:27.820 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:27.800+0000 7fde257d3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde200f6510 0x7fde2006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:30:27.821 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:27.800+0000 7fde257d3700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde20069e20 con 0x7fde200f6510 2023-11-17T05:30:27.821 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:27.801+0000 7fde1effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde200f6510 0x7fde2006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:30:27.822 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:27.801+0000 7fde1effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde200f6510 0x7fde2006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:54058/0 (socket says 172.21.15.160:54058) 2023-11-17T05:30:27.822 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:27.801+0000 7fde1effd700 1 -- 172.21.15.160:0/1797527649 learned_addr learned my addr 172.21.15.160:0/1797527649 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:30:27.823 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:27.801+0000 7fde1effd700 1 -- 172.21.15.160:0/1797527649 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde08005040 con 0x7fde200f6510 2023-11-17T05:30:27.823 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:27.801+0000 7fde1effd700 1 --2- 172.21.15.160:0/1797527649 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde200f6510 0x7fde2006b9a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fde08013510 tx=0x7fde0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:30:27.824 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:27.801+0000 7fde177fe700 1 -- 172.21.15.160:0/1797527649 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde08009320 con 0x7fde200f6510 2023-11-17T05:30:27.824 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:27.801+0000 7fde177fe700 1 -- 172.21.15.160:0/1797527649 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fde08013f00 con 0x7fde200f6510 2023-11-17T05:30:27.824 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:27.801+0000 7fde257d3700 1 -- 172.21.15.160:0/1797527649 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde20069fc0 con 0x7fde200f6510 2023-11-17T05:30:27.825 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:27.801+0000 7fde177fe700 1 -- 172.21.15.160:0/1797527649 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde08014d90 con 0x7fde200f6510 2023-11-17T05:30:27.826 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:27.801+0000 7fde257d3700 1 -- 172.21.15.160:0/1797527649 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde2006a450 con 0x7fde200f6510 2023-11-17T05:30:27.826 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:27.802+0000 7fde177fe700 1 -- 172.21.15.160:0/1797527649 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fde080095f0 con 0x7fde200f6510 2023-11-17T05:30:27.827 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:27.802+0000 7fde257d3700 1 -- 172.21.15.160:0/1797527649 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde20059100 con 0x7fde200f6510 2023-11-17T05:30:27.828 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:27.802+0000 7fde177fe700 1 --2- 172.21.15.160:0/1797527649 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fde0c02ff60 0x7fde0c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:30:27.828 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:27.802+0000 7fde177fe700 1 -- 172.21.15.160:0/1797527649 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fde080410a0 con 0x7fde200f6510 2023-11-17T05:30:27.834 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:27.803+0000 7fde1e7fc700 1 --2- 172.21.15.160:0/1797527649 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fde0c02ff60 0x7fde0c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:30:27.834 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:27.803+0000 7fde1e7fc700 1 --2- 172.21.15.160:0/1797527649 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fde0c02ff60 0x7fde0c032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fde100096a0 tx=0x7fde10006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:30:27.834 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:27.807+0000 7fde177fe700 1 -- 172.21.15.160:0/1797527649 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fde0801d7f0 con 0x7fde200f6510 2023-11-17T05:30:28.014 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:28.013+0000 7fde257d3700 1 -- 172.21.15.160:0/1797527649 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fde200fbf60 con 0x7fde200f6510 2023-11-17T05:30:28.015 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:28.014+0000 7fde177fe700 1 -- 172.21.15.160:0/1797527649 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fde08010230 con 0x7fde200f6510 2023-11-17T05:30:28.015 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:30:28.016 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:30:28.017 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:28.016+0000 7fde257d3700 1 -- 172.21.15.160:0/1797527649 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fde0c02ff60 msgr2=0x7fde0c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:30:28.017 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:28.016+0000 7fde257d3700 1 --2- 172.21.15.160:0/1797527649 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fde0c02ff60 0x7fde0c032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fde100096a0 tx=0x7fde10006b40).stop 2023-11-17T05:30:28.017 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:28.016+0000 7fde257d3700 1 -- 172.21.15.160:0/1797527649 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde200f6510 msgr2=0x7fde2006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:30:28.017 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:28.016+0000 7fde257d3700 1 --2- 172.21.15.160:0/1797527649 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde200f6510 0x7fde2006b9a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fde08013510 tx=0x7fde0800b040).stop 2023-11-17T05:30:28.017 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:28.016+0000 7fde257d3700 1 -- 172.21.15.160:0/1797527649 shutdown_connections 2023-11-17T05:30:28.017 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:28.016+0000 7fde257d3700 1 --2- 172.21.15.160:0/1797527649 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fde0c02ff60 0x7fde0c032420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:30:28.017 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:28.016+0000 7fde257d3700 1 --2- 172.21.15.160:0/1797527649 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fde200f6510 0x7fde2006b9a0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:30:28.017 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:28.016+0000 7fde257d3700 1 -- 172.21.15.160:0/1797527649 >> 172.21.15.160:0/1797527649 conn(0x7fde200f1c70 msgr2=0x7fde200f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:30:28.017 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:28.016+0000 7fde257d3700 1 -- 172.21.15.160:0/1797527649 shutdown_connections 2023-11-17T05:30:28.017 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:28.016+0000 7fde257d3700 1 -- 172.21.15.160:0/1797527649 wait complete. 2023-11-17T05:30:28.018 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:30:28.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:30:27.437563+0000 mgr.smithi003.eddcuc (mgr.14164) 151 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:28.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:30:27.550651+0000 mon.smithi003 (mon.0) 169 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.eddcuc/mirror_snapshot_schedule"}]: dispatch 2023-11-17T05:30:28.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:30:27.554361+0000 mon.smithi003 (mon.0) 170 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.eddcuc/trash_purge_schedule"}]: dispatch 2023-11-17T05:30:28.771 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:30:28.015224+0000 mon.smithi003 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.160:0/1797527649' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:30:29.406 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:30:29.406 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:30:30.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:29 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:30:29.437816+0000 mgr.smithi003.eddcuc (mgr.14164) 152 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:30.665 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:30.664+0000 7f7428df0700 1 -- 172.21.15.160:0/1039578610 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7424060280 msgr2=0x7f74240f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:30:30.666 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:30.664+0000 7f7428df0700 1 --2- 172.21.15.160:0/1039578610 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7424060280 0x7f74240f90f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f740c009fb0 tx=0x7f740c009b30).stop 2023-11-17T05:30:30.666 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:30.664+0000 7f7428df0700 1 -- 172.21.15.160:0/1039578610 shutdown_connections 2023-11-17T05:30:30.666 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:30.664+0000 7f7428df0700 1 --2- 172.21.15.160:0/1039578610 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7424060280 0x7f74240f90f0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:30:30.666 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:30.664+0000 7f7428df0700 1 -- 172.21.15.160:0/1039578610 >> 172.21.15.160:0/1039578610 conn(0x7f74240f0b70 msgr2=0x7f74240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:30:30.666 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:30.664+0000 7f7428df0700 1 -- 172.21.15.160:0/1039578610 shutdown_connections 2023-11-17T05:30:30.666 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:30.664+0000 7f7428df0700 1 -- 172.21.15.160:0/1039578610 wait complete. 2023-11-17T05:30:30.666 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:30.665+0000 7f7428df0700 1 Processor -- start 2023-11-17T05:30:30.666 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:30.665+0000 7f7428df0700 1 -- start start 2023-11-17T05:30:30.668 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:30.665+0000 7f7428df0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7424060280 0x7f74241066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:30:30.669 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:30.665+0000 7f7428df0700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7424106bd0 con 0x7f7424060280 2023-11-17T05:30:30.669 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:30.665+0000 7f742259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7424060280 0x7f74241066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:30:30.669 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:30.665+0000 7f742259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7424060280 0x7f74241066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:59472/0 (socket says 172.21.15.160:59472) 2023-11-17T05:30:30.669 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:30.665+0000 7f742259c700 1 -- 172.21.15.160:0/562733631 learned_addr learned my addr 172.21.15.160:0/562733631 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:30:30.669 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:30.666+0000 7f742259c700 1 -- 172.21.15.160:0/562733631 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f740c005040 con 0x7f7424060280 2023-11-17T05:30:30.669 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:30.666+0000 7f742259c700 1 --2- 172.21.15.160:0/562733631 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7424060280 0x7f74241066e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f740c009830 tx=0x7f740c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:30:30.669 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:30.666+0000 7f741affd700 1 -- 172.21.15.160:0/562733631 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f740c009320 con 0x7f7424060280 2023-11-17T05:30:30.669 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:30.666+0000 7f7428df0700 1 -- 172.21.15.160:0/562733631 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7424106dd0 con 0x7f7424060280 2023-11-17T05:30:30.669 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:30.666+0000 7f741affd700 1 -- 172.21.15.160:0/562733631 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f740c0136c0 con 0x7f7424060280 2023-11-17T05:30:30.669 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:30.666+0000 7f741affd700 1 -- 172.21.15.160:0/562733631 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f740c01ee90 con 0x7f7424060280 2023-11-17T05:30:30.670 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:30.666+0000 7f7428df0700 1 -- 172.21.15.160:0/562733631 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7424107260 con 0x7f7424060280 2023-11-17T05:30:30.670 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:30.667+0000 7f741affd700 1 -- 172.21.15.160:0/562733631 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f740c01e420 con 0x7f7424060280 2023-11-17T05:30:30.670 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:30.667+0000 7f7428df0700 1 -- 172.21.15.160:0/562733631 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7424045ad0 con 0x7f7424060280 2023-11-17T05:30:30.670 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:30.667+0000 7f741affd700 1 --2- 172.21.15.160:0/562733631 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f741002ff10 0x7f74100323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:30:30.670 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:30.667+0000 7f741affd700 1 -- 172.21.15.160:0/562733631 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f740c041010 con 0x7f7424060280 2023-11-17T05:30:30.670 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:30.667+0000 7f7421d9b700 1 --2- 172.21.15.160:0/562733631 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f741002ff10 0x7f74100323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:30:30.670 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:30.668+0000 7f7421d9b700 1 --2- 172.21.15.160:0/562733631 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f741002ff10 0x7f74100323d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f74140096a0 tx=0x7f7414006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:30:30.672 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:30.672+0000 7f741affd700 1 -- 172.21.15.160:0/562733631 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f740c0095f0 con 0x7f7424060280 2023-11-17T05:30:30.876 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:30.875+0000 7f7428df0700 1 -- 172.21.15.160:0/562733631 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7424108110 con 0x7f7424060280 2023-11-17T05:30:30.876 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:30.875+0000 7f741affd700 1 -- 172.21.15.160:0/562733631 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f740c01cb10 con 0x7f7424060280 2023-11-17T05:30:30.877 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:30:30.877 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:30:30.878 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:30.877+0000 7f7428df0700 1 -- 172.21.15.160:0/562733631 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f741002ff10 msgr2=0x7f74100323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:30:30.878 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:30.877+0000 7f7428df0700 1 --2- 172.21.15.160:0/562733631 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f741002ff10 0x7f74100323d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f74140096a0 tx=0x7f7414006b40).stop 2023-11-17T05:30:30.878 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:30.877+0000 7f7428df0700 1 -- 172.21.15.160:0/562733631 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7424060280 msgr2=0x7f74241066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:30:30.878 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:30.877+0000 7f7428df0700 1 --2- 172.21.15.160:0/562733631 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7424060280 0x7f74241066e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f740c009830 tx=0x7f740c00b040).stop 2023-11-17T05:30:30.878 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:30.877+0000 7f7428df0700 1 -- 172.21.15.160:0/562733631 shutdown_connections 2023-11-17T05:30:30.878 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:30.877+0000 7f7428df0700 1 --2- 172.21.15.160:0/562733631 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f741002ff10 0x7f74100323d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:30:30.878 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:30.877+0000 7f7428df0700 1 --2- 172.21.15.160:0/562733631 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7424060280 0x7f74241066e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:30:30.878 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:30.877+0000 7f7428df0700 1 -- 172.21.15.160:0/562733631 >> 172.21.15.160:0/562733631 conn(0x7f74240f0b70 msgr2=0x7f74240f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:30:30.879 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:30.877+0000 7f7428df0700 1 -- 172.21.15.160:0/562733631 shutdown_connections 2023-11-17T05:30:30.879 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:30.877+0000 7f7428df0700 1 -- 172.21.15.160:0/562733631 wait complete. 2023-11-17T05:30:30.879 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:30:31.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:30 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:30:30.876704+0000 mon.smithi003 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.160:0/562733631' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:30:32.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:31 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:30:31.438159+0000 mgr.smithi003.eddcuc (mgr.14164) 153 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:32.292 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:30:32.292 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:30:33.458 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:33.457+0000 7f9b4a2c1700 1 -- 172.21.15.160:0/1912617957 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b440f5410 msgr2=0x7f9b440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:30:33.459 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:33.457+0000 7f9b4a2c1700 1 --2- 172.21.15.160:0/1912617957 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b440f5410 0x7f9b440f5830 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f9b34009fb0 tx=0x7f9b34009b30).stop 2023-11-17T05:30:33.459 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:33.457+0000 7f9b4a2c1700 1 -- 172.21.15.160:0/1912617957 shutdown_connections 2023-11-17T05:30:33.459 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:33.457+0000 7f9b4a2c1700 1 --2- 172.21.15.160:0/1912617957 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b440f5410 0x7f9b440f5830 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:30:33.459 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:33.457+0000 7f9b4a2c1700 1 -- 172.21.15.160:0/1912617957 >> 172.21.15.160:0/1912617957 conn(0x7f9b440f0b70 msgr2=0x7f9b440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:30:33.459 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:33.457+0000 7f9b4a2c1700 1 -- 172.21.15.160:0/1912617957 shutdown_connections 2023-11-17T05:30:33.459 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:33.457+0000 7f9b4a2c1700 1 -- 172.21.15.160:0/1912617957 wait complete. 2023-11-17T05:30:33.460 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:33.458+0000 7f9b4a2c1700 1 Processor -- start 2023-11-17T05:30:33.460 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:33.458+0000 7f9b4a2c1700 1 -- start start 2023-11-17T05:30:33.460 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:33.459+0000 7f9b4a2c1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b440f5410 0x7f9b44105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:30:33.460 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:33.459+0000 7f9b4a2c1700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b44105b20 con 0x7f9b440f5410 2023-11-17T05:30:33.460 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:33.459+0000 7f9b43fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b440f5410 0x7f9b44105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:30:33.460 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:33.459+0000 7f9b43fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b440f5410 0x7f9b44105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:59476/0 (socket says 172.21.15.160:59476) 2023-11-17T05:30:33.460 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:33.459+0000 7f9b43fff700 1 -- 172.21.15.160:0/2413300391 learned_addr learned my addr 172.21.15.160:0/2413300391 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:30:33.460 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:33.459+0000 7f9b43fff700 1 -- 172.21.15.160:0/2413300391 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b34005040 con 0x7f9b440f5410 2023-11-17T05:30:33.460 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:33.459+0000 7f9b43fff700 1 --2- 172.21.15.160:0/2413300391 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b440f5410 0x7f9b44105630 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f9b34013510 tx=0x7f9b3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:30:33.461 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:33.460+0000 7f9b40ff9700 1 -- 172.21.15.160:0/2413300391 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b34009320 con 0x7f9b440f5410 2023-11-17T05:30:33.462 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:33.460+0000 7f9b4a2c1700 1 -- 172.21.15.160:0/2413300391 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b44105d20 con 0x7f9b440f5410 2023-11-17T05:30:33.462 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:33.460+0000 7f9b40ff9700 1 -- 172.21.15.160:0/2413300391 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9b34013f00 con 0x7f9b440f5410 2023-11-17T05:30:33.462 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:33.460+0000 7f9b40ff9700 1 -- 172.21.15.160:0/2413300391 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b34014d90 con 0x7f9b440f5410 2023-11-17T05:30:33.462 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:33.460+0000 7f9b4a2c1700 1 -- 172.21.15.160:0/2413300391 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b441061b0 con 0x7f9b440f5410 2023-11-17T05:30:33.462 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:33.460+0000 7f9b40ff9700 1 -- 172.21.15.160:0/2413300391 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9b340095f0 con 0x7f9b440f5410 2023-11-17T05:30:33.462 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:33.460+0000 7f9b4a2c1700 1 -- 172.21.15.160:0/2413300391 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b44045ad0 con 0x7f9b440f5410 2023-11-17T05:30:33.462 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:33.461+0000 7f9b40ff9700 1 --2- 172.21.15.160:0/2413300391 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f9b2c02ff60 0x7f9b2c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:30:33.462 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:33.461+0000 7f9b40ff9700 1 -- 172.21.15.160:0/2413300391 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9b34042e80 con 0x7f9b440f5410 2023-11-17T05:30:33.462 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:33.461+0000 7f9b437fe700 1 --2- 172.21.15.160:0/2413300391 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f9b2c02ff60 0x7f9b2c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:30:33.463 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:33.461+0000 7f9b437fe700 1 --2- 172.21.15.160:0/2413300391 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f9b2c02ff60 0x7f9b2c032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f9b380096a0 tx=0x7f9b38006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:30:33.466 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:33.466+0000 7f9b40ff9700 1 -- 172.21.15.160:0/2413300391 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9b34014420 con 0x7f9b440f5410 2023-11-17T05:30:33.674 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:33.673+0000 7f9b4a2c1700 1 -- 172.21.15.160:0/2413300391 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9b440fae60 con 0x7f9b440f5410 2023-11-17T05:30:33.676 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:33.674+0000 7f9b40ff9700 1 -- 172.21.15.160:0/2413300391 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f9b34011290 con 0x7f9b440f5410 2023-11-17T05:30:33.676 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:30:33.676 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:30:33.677 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:33.675+0000 7f9b4a2c1700 1 -- 172.21.15.160:0/2413300391 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f9b2c02ff60 msgr2=0x7f9b2c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:30:33.677 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:33.675+0000 7f9b4a2c1700 1 --2- 172.21.15.160:0/2413300391 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f9b2c02ff60 0x7f9b2c032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f9b380096a0 tx=0x7f9b38006b40).stop 2023-11-17T05:30:33.677 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:33.675+0000 7f9b4a2c1700 1 -- 172.21.15.160:0/2413300391 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b440f5410 msgr2=0x7f9b44105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:30:33.677 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:33.675+0000 7f9b4a2c1700 1 --2- 172.21.15.160:0/2413300391 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b440f5410 0x7f9b44105630 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f9b34013510 tx=0x7f9b3400b040).stop 2023-11-17T05:30:33.677 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:33.675+0000 7f9b4a2c1700 1 -- 172.21.15.160:0/2413300391 shutdown_connections 2023-11-17T05:30:33.677 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:33.675+0000 7f9b4a2c1700 1 --2- 172.21.15.160:0/2413300391 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f9b2c02ff60 0x7f9b2c032420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:30:33.677 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:33.675+0000 7f9b4a2c1700 1 --2- 172.21.15.160:0/2413300391 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9b440f5410 0x7f9b44105630 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:30:33.677 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:33.675+0000 7f9b4a2c1700 1 -- 172.21.15.160:0/2413300391 >> 172.21.15.160:0/2413300391 conn(0x7f9b440f0b70 msgr2=0x7f9b440f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:30:33.677 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:33.675+0000 7f9b4a2c1700 1 -- 172.21.15.160:0/2413300391 shutdown_connections 2023-11-17T05:30:33.678 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:33.675+0000 7f9b4a2c1700 1 -- 172.21.15.160:0/2413300391 wait complete. 2023-11-17T05:30:33.678 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:30:34.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:34 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:30:33.438469+0000 mgr.smithi003.eddcuc (mgr.14164) 154 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:34.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:34 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:30:33.674989+0000 mon.smithi003 (mon.0) 173 : audit [DBG] from='client.? 172.21.15.160:0/2413300391' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:30:35.062 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:30:35.063 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:30:36.337 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:36.332+0000 7fa2c46a9700 1 -- 172.21.15.160:0/2451741977 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2bc0f6510 msgr2=0x7fa2bc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:30:36.337 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:36.332+0000 7fa2c46a9700 1 --2- 172.21.15.160:0/2451741977 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2bc0f6510 0x7fa2bc0f6930 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fa2ac009fb0 tx=0x7fa2ac009b30).stop 2023-11-17T05:30:36.337 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:36.332+0000 7fa2c46a9700 1 -- 172.21.15.160:0/2451741977 shutdown_connections 2023-11-17T05:30:36.338 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:36.332+0000 7fa2c46a9700 1 --2- 172.21.15.160:0/2451741977 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2bc0f6510 0x7fa2bc0f6930 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:30:36.338 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:36.332+0000 7fa2c46a9700 1 -- 172.21.15.160:0/2451741977 >> 172.21.15.160:0/2451741977 conn(0x7fa2bc0f1c70 msgr2=0x7fa2bc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:30:36.338 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:36.332+0000 7fa2c46a9700 1 -- 172.21.15.160:0/2451741977 shutdown_connections 2023-11-17T05:30:36.338 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:36.332+0000 7fa2c46a9700 1 -- 172.21.15.160:0/2451741977 wait complete. 2023-11-17T05:30:36.338 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:36.333+0000 7fa2c46a9700 1 Processor -- start 2023-11-17T05:30:36.339 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:36.333+0000 7fa2c46a9700 1 -- start start 2023-11-17T05:30:36.339 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:36.333+0000 7fa2c46a9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2bc0f6510 0x7fa2bc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:30:36.339 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:36.333+0000 7fa2c46a9700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc106c20 con 0x7fa2bc0f6510 2023-11-17T05:30:36.339 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:36.333+0000 7fa2c2445700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2bc0f6510 0x7fa2bc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:30:36.340 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:36.333+0000 7fa2c2445700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2bc0f6510 0x7fa2bc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:59496/0 (socket says 172.21.15.160:59496) 2023-11-17T05:30:36.340 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:36.333+0000 7fa2c2445700 1 -- 172.21.15.160:0/3730472676 learned_addr learned my addr 172.21.15.160:0/3730472676 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:30:36.340 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:36.333+0000 7fa2c2445700 1 -- 172.21.15.160:0/3730472676 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2ac005040 con 0x7fa2bc0f6510 2023-11-17T05:30:36.340 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:36.334+0000 7fa2c2445700 1 --2- 172.21.15.160:0/3730472676 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2bc0f6510 0x7fa2bc106730 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fa2ac013510 tx=0x7fa2ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:30:36.340 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:36.334+0000 7fa2b2ffd700 1 -- 172.21.15.160:0/3730472676 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2ac009320 con 0x7fa2bc0f6510 2023-11-17T05:30:36.341 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:36.334+0000 7fa2c46a9700 1 -- 172.21.15.160:0/3730472676 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2bc106e20 con 0x7fa2bc0f6510 2023-11-17T05:30:36.341 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:36.334+0000 7fa2b2ffd700 1 -- 172.21.15.160:0/3730472676 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa2ac013f00 con 0x7fa2bc0f6510 2023-11-17T05:30:36.341 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:36.334+0000 7fa2b2ffd700 1 -- 172.21.15.160:0/3730472676 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2ac014d90 con 0x7fa2bc0f6510 2023-11-17T05:30:36.341 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:36.334+0000 7fa2c46a9700 1 -- 172.21.15.160:0/3730472676 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2bc1072b0 con 0x7fa2bc0f6510 2023-11-17T05:30:36.341 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:36.335+0000 7fa2b2ffd700 1 -- 172.21.15.160:0/3730472676 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa2ac0263a0 con 0x7fa2bc0f6510 2023-11-17T05:30:36.342 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:36.335+0000 7fa2b2ffd700 1 --2- 172.21.15.160:0/3730472676 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa2a802ff60 0x7fa2a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:30:36.342 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:36.335+0000 7fa2b2ffd700 1 -- 172.21.15.160:0/3730472676 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa2ac0410b0 con 0x7fa2bc0f6510 2023-11-17T05:30:36.342 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:36.335+0000 7fa2c46a9700 1 -- 172.21.15.160:0/3730472676 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2bc045ad0 con 0x7fa2bc0f6510 2023-11-17T05:30:36.342 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:36.335+0000 7fa2c1c44700 1 --2- 172.21.15.160:0/3730472676 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa2a802ff60 0x7fa2a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:30:36.342 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:36.335+0000 7fa2c1c44700 1 --2- 172.21.15.160:0/3730472676 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa2a802ff60 0x7fa2a8032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fa2b80096a0 tx=0x7fa2b8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:30:36.343 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:36.340+0000 7fa2b2ffd700 1 -- 172.21.15.160:0/3730472676 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa2ac012920 con 0x7fa2bc0f6510 2023-11-17T05:30:36.549 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:36.548+0000 7fa2c46a9700 1 -- 172.21.15.160:0/3730472676 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa2bc0fc050 con 0x7fa2bc0f6510 2023-11-17T05:30:36.551 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:36.549+0000 7fa2b2ffd700 1 -- 172.21.15.160:0/3730472676 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fa2ac02b020 con 0x7fa2bc0f6510 2023-11-17T05:30:36.552 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:30:36.552 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:30:36.553 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:36.551+0000 7fa2c46a9700 1 -- 172.21.15.160:0/3730472676 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa2a802ff60 msgr2=0x7fa2a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:30:36.554 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:36.551+0000 7fa2c46a9700 1 --2- 172.21.15.160:0/3730472676 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa2a802ff60 0x7fa2a8032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fa2b80096a0 tx=0x7fa2b8006b40).stop 2023-11-17T05:30:36.554 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:36.551+0000 7fa2c46a9700 1 -- 172.21.15.160:0/3730472676 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2bc0f6510 msgr2=0x7fa2bc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:30:36.554 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:36.551+0000 7fa2c46a9700 1 --2- 172.21.15.160:0/3730472676 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2bc0f6510 0x7fa2bc106730 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fa2ac013510 tx=0x7fa2ac00b040).stop 2023-11-17T05:30:36.554 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:36.551+0000 7fa2c46a9700 1 -- 172.21.15.160:0/3730472676 shutdown_connections 2023-11-17T05:30:36.554 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:36.551+0000 7fa2c46a9700 1 --2- 172.21.15.160:0/3730472676 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa2a802ff60 0x7fa2a8032420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:30:36.554 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:36.551+0000 7fa2c46a9700 1 --2- 172.21.15.160:0/3730472676 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2bc0f6510 0x7fa2bc106730 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:30:36.554 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:36.551+0000 7fa2c46a9700 1 -- 172.21.15.160:0/3730472676 >> 172.21.15.160:0/3730472676 conn(0x7fa2bc0f1c70 msgr2=0x7fa2bc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:30:36.555 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:36.551+0000 7fa2c46a9700 1 -- 172.21.15.160:0/3730472676 shutdown_connections 2023-11-17T05:30:36.555 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:36.551+0000 7fa2c46a9700 1 -- 172.21.15.160:0/3730472676 wait complete. 2023-11-17T05:30:36.555 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:30:36.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:36 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:30:35.438810+0000 mgr.smithi003.eddcuc (mgr.14164) 155 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:37.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:37 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:30:36.550190+0000 mon.smithi003 (mon.0) 174 : audit [DBG] from='client.? 172.21.15.160:0/3730472676' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:30:37.771 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:37 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:30:36.824526+0000 mon.smithi003 (mon.0) 175 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi003", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-11-17T05:30:37.771 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:37 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:30:36.827068+0000 mon.smithi003 (mon.0) 176 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi003", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2023-11-17T05:30:37.771 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:37 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:30:36.827984+0000 mon.smithi003 (mon.0) 177 : audit [DBG] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-17T05:30:37.938 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:30:37.938 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:30:38.771 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cephadm 2023-11-17T05:30:36.823256+0000 mgr.smithi003.eddcuc (mgr.14164) 156 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi003 ... 2023-11-17T05:30:38.771 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2023-11-17T05:30:38.771 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-17T05:30:38.771 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: stat: stderr time="2023-11-17T05:26:15Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-17T05:30:38.771 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: stat: stderr time="2023-11-17T05:26:53Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-17T05:30:38.771 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: stat: stderr time="2023-11-17T05:27:30Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-17T05:30:38.771 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-17T05:30:38.771 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2023-11-17T05:30:38.772 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-17T05:30:38.772 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: stat: stderr time="2023-11-17T05:28:44Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-17T05:30:38.772 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: stat: stderr time="2023-11-17T05:29:22Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-17T05:30:38.772 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: stat: stderr time="2023-11-17T05:29:59Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-17T05:30:38.772 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-17T05:30:38.772 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: Traceback (most recent call last): 2023-11-17T05:30:38.772 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: File "/var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-17T05:30:38.772 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: main() 2023-11-17T05:30:38.772 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: File "/var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-17T05:30:38.772 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: r = ctx.func(ctx) 2023-11-17T05:30:38.772 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: File "/var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-17T05:30:38.773 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: return func(ctx) 2023-11-17T05:30:38.773 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: File "/var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-11-17T05:30:38.773 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-11-17T05:30:38.773 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: File "/var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2023-11-17T05:30:38.773 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2023-11-17T05:30:38.773 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: File "/var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-11-17T05:30:38.773 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: raise RuntimeError('uid/gid not found') 2023-11-17T05:30:38.773 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: RuntimeError: uid/gid not found 2023-11-17T05:30:38.773 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: Traceback (most recent call last): 2023-11-17T05:30:38.773 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-11-17T05:30:38.773 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: yield (conn, connr) 2023-11-17T05:30:38.773 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-11-17T05:30:38.773 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: code, '\n'.join(err))) 2023-11-17T05:30:38.774 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi003 ... 2023-11-17T05:30:38.774 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2023-11-17T05:30:38.774 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-17T05:30:38.774 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: stat: stderr time="2023-11-17T05:26:15Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-17T05:30:38.775 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: stat: stderr time="2023-11-17T05:26:53Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-17T05:30:38.775 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: stat: stderr time="2023-11-17T05:27:30Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-17T05:30:38.775 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-17T05:30:38.775 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2023-11-17T05:30:38.775 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-17T05:30:38.775 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: stat: stderr time="2023-11-17T05:28:44Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-17T05:30:38.775 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: stat: stderr time="2023-11-17T05:29:22Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-17T05:30:38.775 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: stat: stderr time="2023-11-17T05:29:59Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-17T05:30:38.775 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-17T05:30:38.776 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: Traceback (most recent call last): 2023-11-17T05:30:38.776 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: File "/var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-17T05:30:38.776 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: main() 2023-11-17T05:30:38.776 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: File "/var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-17T05:30:38.776 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: r = ctx.func(ctx) 2023-11-17T05:30:38.776 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: File "/var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-17T05:30:38.776 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: return func(ctx) 2023-11-17T05:30:38.776 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: File "/var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-11-17T05:30:38.776 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-11-17T05:30:38.776 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: File "/var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2023-11-17T05:30:38.776 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2023-11-17T05:30:38.776 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: File "/var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-11-17T05:30:38.777 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: raise RuntimeError('uid/gid not found') 2023-11-17T05:30:38.777 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: RuntimeError: uid/gid not found 2023-11-17T05:30:38.777 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cephadm 2023-11-17T05:30:36.828631+0000 mgr.smithi003.eddcuc (mgr.14164) 157 : cephadm [INF] Deploying daemon crash.smithi003 on smithi003 2023-11-17T05:30:38.777 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:30:37.439115+0000 mgr.smithi003.eddcuc (mgr.14164) 158 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:38.777 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:30:38.385872+0000 mon.smithi003 (mon.0) 178 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' 2023-11-17T05:30:38.777 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:30:38.470191+0000 mon.smithi003 (mon.0) 179 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' 2023-11-17T05:30:38.777 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:30:38.472714+0000 mon.smithi003 (mon.0) 180 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' 2023-11-17T05:30:38.777 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:30:38.473213+0000 mon.smithi003 (mon.0) 181 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-11-17T05:30:38.777 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:30:38.476232+0000 mon.smithi003 (mon.0) 182 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' 2023-11-17T05:30:39.104 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:39.102+0000 7f46af34b700 1 -- 172.21.15.160:0/2055680725 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46a80f5410 msgr2=0x7f46a80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:30:39.105 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:39.102+0000 7f46af34b700 1 --2- 172.21.15.160:0/2055680725 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46a80f5410 0x7f46a80f5830 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f4698009fb0 tx=0x7f4698009b30).stop 2023-11-17T05:30:39.105 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:39.102+0000 7f46af34b700 1 -- 172.21.15.160:0/2055680725 shutdown_connections 2023-11-17T05:30:39.105 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:39.102+0000 7f46af34b700 1 --2- 172.21.15.160:0/2055680725 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46a80f5410 0x7f46a80f5830 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:30:39.105 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:39.102+0000 7f46af34b700 1 -- 172.21.15.160:0/2055680725 >> 172.21.15.160:0/2055680725 conn(0x7f46a80f0b70 msgr2=0x7f46a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:30:39.105 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:39.102+0000 7f46af34b700 1 -- 172.21.15.160:0/2055680725 shutdown_connections 2023-11-17T05:30:39.105 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:39.102+0000 7f46af34b700 1 -- 172.21.15.160:0/2055680725 wait complete. 2023-11-17T05:30:39.105 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:39.103+0000 7f46af34b700 1 Processor -- start 2023-11-17T05:30:39.105 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:39.103+0000 7f46af34b700 1 -- start start 2023-11-17T05:30:39.105 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:39.103+0000 7f46af34b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46a80f5410 0x7f46a8109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:30:39.105 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:39.103+0000 7f46af34b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a8060220 con 0x7f46a80f5410 2023-11-17T05:30:39.105 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:39.103+0000 7f46ad0e7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46a80f5410 0x7f46a8109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:30:39.106 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:39.103+0000 7f46ad0e7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46a80f5410 0x7f46a8109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:59504/0 (socket says 172.21.15.160:59504) 2023-11-17T05:30:39.106 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:39.103+0000 7f46ad0e7700 1 -- 172.21.15.160:0/2121050862 learned_addr learned my addr 172.21.15.160:0/2121050862 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:30:39.106 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:39.104+0000 7f46ad0e7700 1 -- 172.21.15.160:0/2121050862 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4698005040 con 0x7f46a80f5410 2023-11-17T05:30:39.106 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:39.104+0000 7f46ad0e7700 1 --2- 172.21.15.160:0/2121050862 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46a80f5410 0x7f46a8109450 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f4698013510 tx=0x7f469800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:30:39.107 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:39.104+0000 7f469dffb700 1 -- 172.21.15.160:0/2121050862 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4698009320 con 0x7f46a80f5410 2023-11-17T05:30:39.107 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:39.104+0000 7f46af34b700 1 -- 172.21.15.160:0/2121050862 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46a8060420 con 0x7f46a80f5410 2023-11-17T05:30:39.107 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:39.104+0000 7f469dffb700 1 -- 172.21.15.160:0/2121050862 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4698013f00 con 0x7f46a80f5410 2023-11-17T05:30:39.107 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:39.104+0000 7f469dffb700 1 -- 172.21.15.160:0/2121050862 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4698014d90 con 0x7f46a80f5410 2023-11-17T05:30:39.107 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:39.104+0000 7f46af34b700 1 -- 172.21.15.160:0/2121050862 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46a8109b00 con 0x7f46a80f5410 2023-11-17T05:30:39.107 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:39.105+0000 7f46af34b700 1 -- 172.21.15.160:0/2121050862 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46a8059100 con 0x7f46a80f5410 2023-11-17T05:30:39.107 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:39.105+0000 7f469dffb700 1 -- 172.21.15.160:0/2121050862 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f46980095f0 con 0x7f46a80f5410 2023-11-17T05:30:39.108 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:39.105+0000 7f469dffb700 1 --2- 172.21.15.160:0/2121050862 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f469402ff60 0x7f4694032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:30:39.108 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:39.105+0000 7f469dffb700 1 -- 172.21.15.160:0/2121050862 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f46980410c0 con 0x7f46a80f5410 2023-11-17T05:30:39.109 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:39.105+0000 7f46ac8e6700 1 --2- 172.21.15.160:0/2121050862 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f469402ff60 0x7f4694032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:30:39.109 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:39.106+0000 7f46ac8e6700 1 --2- 172.21.15.160:0/2121050862 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f469402ff60 0x7f4694032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f46a40096a0 tx=0x7f46a4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:30:39.112 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:39.110+0000 7f469dffb700 1 -- 172.21.15.160:0/2121050862 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4698014420 con 0x7f46a80f5410 2023-11-17T05:30:39.326 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:39.324+0000 7f46af34b700 1 -- 172.21.15.160:0/2121050862 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f46a8045ad0 con 0x7f46a80f5410 2023-11-17T05:30:39.326 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:39.325+0000 7f469dffb700 1 -- 172.21.15.160:0/2121050862 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f469801d7f0 con 0x7f46a80f5410 2023-11-17T05:30:39.326 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:30:39.326 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:30:39.328 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:39.326+0000 7f46af34b700 1 -- 172.21.15.160:0/2121050862 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f469402ff60 msgr2=0x7f4694032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:30:39.328 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:39.326+0000 7f46af34b700 1 --2- 172.21.15.160:0/2121050862 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f469402ff60 0x7f4694032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f46a40096a0 tx=0x7f46a4006b40).stop 2023-11-17T05:30:39.328 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:39.326+0000 7f46af34b700 1 -- 172.21.15.160:0/2121050862 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46a80f5410 msgr2=0x7f46a8109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:30:39.328 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:39.326+0000 7f46af34b700 1 --2- 172.21.15.160:0/2121050862 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46a80f5410 0x7f46a8109450 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f4698013510 tx=0x7f469800b040).stop 2023-11-17T05:30:39.328 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:39.327+0000 7f46af34b700 1 -- 172.21.15.160:0/2121050862 shutdown_connections 2023-11-17T05:30:39.328 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:39.327+0000 7f46af34b700 1 --2- 172.21.15.160:0/2121050862 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f469402ff60 0x7f4694032420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:30:39.328 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:39.327+0000 7f46af34b700 1 --2- 172.21.15.160:0/2121050862 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46a80f5410 0x7f46a8109450 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:30:39.328 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:39.327+0000 7f46af34b700 1 -- 172.21.15.160:0/2121050862 >> 172.21.15.160:0/2121050862 conn(0x7f46a80f0b70 msgr2=0x7f46a80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:30:39.328 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:39.327+0000 7f46af34b700 1 -- 172.21.15.160:0/2121050862 shutdown_connections 2023-11-17T05:30:39.328 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:39.327+0000 7f46af34b700 1 -- 172.21.15.160:0/2121050862 wait complete. 2023-11-17T05:30:39.329 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:30:39.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:39 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:30:38.473657+0000 mgr.smithi003.eddcuc (mgr.14164) 159 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-11-17T05:30:39.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:39 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cephadm 2023-11-17T05:30:38.479521+0000 mgr.smithi003.eddcuc (mgr.14164) 160 : cephadm [INF] Deploying daemon grafana.smithi003 on smithi003 2023-11-17T05:30:39.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:39 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:30:39.326079+0000 mon.smithi003 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.160:0/2121050862' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:30:40.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:40 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:30:39.439556+0000 mgr.smithi003.eddcuc (mgr.14164) 161 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:40.781 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:30:40.782 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:30:41.983 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:41.982+0000 7ff8431f2700 1 -- 172.21.15.160:0/1731388499 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff83c060280 msgr2=0x7ff83c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:30:41.983 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:41.982+0000 7ff8431f2700 1 --2- 172.21.15.160:0/1731388499 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff83c060280 0x7ff83c0f90f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7ff830009fb0 tx=0x7ff830009b30).stop 2023-11-17T05:30:41.983 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:41.982+0000 7ff8431f2700 1 -- 172.21.15.160:0/1731388499 shutdown_connections 2023-11-17T05:30:41.984 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:41.982+0000 7ff8431f2700 1 --2- 172.21.15.160:0/1731388499 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff83c060280 0x7ff83c0f90f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:30:41.984 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:41.982+0000 7ff8431f2700 1 -- 172.21.15.160:0/1731388499 >> 172.21.15.160:0/1731388499 conn(0x7ff83c0f0b70 msgr2=0x7ff83c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:30:41.984 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:41.982+0000 7ff8431f2700 1 -- 172.21.15.160:0/1731388499 shutdown_connections 2023-11-17T05:30:41.984 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:41.983+0000 7ff8431f2700 1 -- 172.21.15.160:0/1731388499 wait complete. 2023-11-17T05:30:41.985 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:41.983+0000 7ff8431f2700 1 Processor -- start 2023-11-17T05:30:41.985 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:41.983+0000 7ff8431f2700 1 -- start start 2023-11-17T05:30:41.985 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:41.984+0000 7ff8431f2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff83c060280 0x7ff83c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:30:41.985 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:41.984+0000 7ff8431f2700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff83c106bd0 con 0x7ff83c060280 2023-11-17T05:30:41.985 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:41.984+0000 7ff840f8e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff83c060280 0x7ff83c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:30:41.985 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:41.984+0000 7ff840f8e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff83c060280 0x7ff83c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:46716/0 (socket says 172.21.15.160:46716) 2023-11-17T05:30:41.985 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:41.984+0000 7ff840f8e700 1 -- 172.21.15.160:0/1833185153 learned_addr learned my addr 172.21.15.160:0/1833185153 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:30:41.986 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:41.984+0000 7ff840f8e700 1 -- 172.21.15.160:0/1833185153 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff830005040 con 0x7ff83c060280 2023-11-17T05:30:41.986 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:41.984+0000 7ff840f8e700 1 --2- 172.21.15.160:0/1833185153 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff83c060280 0x7ff83c1066e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7ff830009830 tx=0x7ff83000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:30:41.986 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:41.985+0000 7ff8397fa700 1 -- 172.21.15.160:0/1833185153 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff830009320 con 0x7ff83c060280 2023-11-17T05:30:41.986 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:41.985+0000 7ff8397fa700 1 -- 172.21.15.160:0/1833185153 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff8300136c0 con 0x7ff83c060280 2023-11-17T05:30:41.986 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:41.985+0000 7ff8431f2700 1 -- 172.21.15.160:0/1833185153 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff83c106dd0 con 0x7ff83c060280 2023-11-17T05:30:41.986 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:41.985+0000 7ff8397fa700 1 -- 172.21.15.160:0/1833185153 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff83001ee90 con 0x7ff83c060280 2023-11-17T05:30:41.986 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:41.985+0000 7ff8431f2700 1 -- 172.21.15.160:0/1833185153 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff83c107260 con 0x7ff83c060280 2023-11-17T05:30:41.987 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:41.986+0000 7ff8397fa700 1 -- 172.21.15.160:0/1833185153 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff830014440 con 0x7ff83c060280 2023-11-17T05:30:41.988 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:41.986+0000 7ff8431f2700 1 -- 172.21.15.160:0/1833185153 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff83c045ad0 con 0x7ff83c060280 2023-11-17T05:30:41.988 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:41.986+0000 7ff8397fa700 1 --2- 172.21.15.160:0/1833185153 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7ff82c02ff10 0x7ff82c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:30:41.988 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:41.986+0000 7ff8397fa700 1 -- 172.21.15.160:0/1833185153 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff830041220 con 0x7ff83c060280 2023-11-17T05:30:41.988 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:41.986+0000 7ff83bfff700 1 --2- 172.21.15.160:0/1833185153 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7ff82c02ff10 0x7ff82c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:30:41.988 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:41.986+0000 7ff83bfff700 1 --2- 172.21.15.160:0/1833185153 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7ff82c02ff10 0x7ff82c0323d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7ff8280096a0 tx=0x7ff828006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:30:41.991 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:41.991+0000 7ff8397fa700 1 -- 172.21.15.160:0/1833185153 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff83001c320 con 0x7ff83c060280 2023-11-17T05:30:42.196 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:42.195+0000 7ff8431f2700 1 -- 172.21.15.160:0/1833185153 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff83c1080b0 con 0x7ff83c060280 2023-11-17T05:30:42.197 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:42.196+0000 7ff8397fa700 1 -- 172.21.15.160:0/1833185153 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7ff83001e420 con 0x7ff83c060280 2023-11-17T05:30:42.197 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:30:42.197 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:30:42.200 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:42.198+0000 7ff8431f2700 1 -- 172.21.15.160:0/1833185153 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7ff82c02ff10 msgr2=0x7ff82c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:30:42.200 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:42.198+0000 7ff8431f2700 1 --2- 172.21.15.160:0/1833185153 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7ff82c02ff10 0x7ff82c0323d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7ff8280096a0 tx=0x7ff828006b40).stop 2023-11-17T05:30:42.200 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:42.198+0000 7ff8431f2700 1 -- 172.21.15.160:0/1833185153 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff83c060280 msgr2=0x7ff83c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:30:42.200 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:42.198+0000 7ff8431f2700 1 --2- 172.21.15.160:0/1833185153 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff83c060280 0x7ff83c1066e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7ff830009830 tx=0x7ff83000b040).stop 2023-11-17T05:30:42.200 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:42.198+0000 7ff8431f2700 1 -- 172.21.15.160:0/1833185153 shutdown_connections 2023-11-17T05:30:42.200 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:42.198+0000 7ff8431f2700 1 --2- 172.21.15.160:0/1833185153 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7ff82c02ff10 0x7ff82c0323d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:30:42.200 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:42.198+0000 7ff8431f2700 1 --2- 172.21.15.160:0/1833185153 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff83c060280 0x7ff83c1066e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:30:42.200 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:42.198+0000 7ff8431f2700 1 -- 172.21.15.160:0/1833185153 >> 172.21.15.160:0/1833185153 conn(0x7ff83c0f0b70 msgr2=0x7ff83c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:30:42.200 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:42.198+0000 7ff8431f2700 1 -- 172.21.15.160:0/1833185153 shutdown_connections 2023-11-17T05:30:42.200 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:42.198+0000 7ff8431f2700 1 -- 172.21.15.160:0/1833185153 wait complete. 2023-11-17T05:30:42.201 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:30:42.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:42 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:30:41.439893+0000 mgr.smithi003.eddcuc (mgr.14164) 162 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:42.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:42 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:30:42.197363+0000 mon.smithi003 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.160:0/1833185153' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:30:43.591 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:30:43.591 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:30:44.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:44 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:30:43.440241+0000 mgr.smithi003.eddcuc (mgr.14164) 163 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:44.813 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:44.811+0000 7f60e4461700 1 -- 172.21.15.160:0/1376226841 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60dc0f41a0 msgr2=0x7f60dc0f45c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:30:44.813 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:44.811+0000 7f60e4461700 1 --2- 172.21.15.160:0/1376226841 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60dc0f41a0 0x7f60dc0f45c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f60d8009fb0 tx=0x7f60d8009b30).stop 2023-11-17T05:30:44.813 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:44.811+0000 7f60e4461700 1 -- 172.21.15.160:0/1376226841 shutdown_connections 2023-11-17T05:30:44.813 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:44.811+0000 7f60e4461700 1 --2- 172.21.15.160:0/1376226841 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60dc0f41a0 0x7f60dc0f45c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:30:44.813 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:44.811+0000 7f60e4461700 1 -- 172.21.15.160:0/1376226841 >> 172.21.15.160:0/1376226841 conn(0x7f60dc0efe40 msgr2=0x7f60dc0f2260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:30:44.813 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:44.811+0000 7f60e4461700 1 -- 172.21.15.160:0/1376226841 shutdown_connections 2023-11-17T05:30:44.813 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:44.811+0000 7f60e4461700 1 -- 172.21.15.160:0/1376226841 wait complete. 2023-11-17T05:30:44.814 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:44.812+0000 7f60e4461700 1 Processor -- start 2023-11-17T05:30:44.814 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:44.812+0000 7f60e4461700 1 -- start start 2023-11-17T05:30:44.814 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:44.812+0000 7f60e4461700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60dc0f41a0 0x7f60dc1066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:30:44.814 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:44.812+0000 7f60e4461700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60dc106bc0 con 0x7f60dc0f41a0 2023-11-17T05:30:44.814 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:44.812+0000 7f60e21fd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60dc0f41a0 0x7f60dc1066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:30:44.814 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:44.812+0000 7f60e21fd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60dc0f41a0 0x7f60dc1066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:46748/0 (socket says 172.21.15.160:46748) 2023-11-17T05:30:44.814 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:44.812+0000 7f60e21fd700 1 -- 172.21.15.160:0/2305593929 learned_addr learned my addr 172.21.15.160:0/2305593929 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:30:44.814 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:44.812+0000 7f60e21fd700 1 -- 172.21.15.160:0/2305593929 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d8005040 con 0x7f60dc0f41a0 2023-11-17T05:30:44.815 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:44.813+0000 7f60e21fd700 1 --2- 172.21.15.160:0/2305593929 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60dc0f41a0 0x7f60dc1066d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f60d8013510 tx=0x7f60d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:30:44.815 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:44.813+0000 7f60d2ffd700 1 -- 172.21.15.160:0/2305593929 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60d8009320 con 0x7f60dc0f41a0 2023-11-17T05:30:44.816 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:44.813+0000 7f60e4461700 1 -- 172.21.15.160:0/2305593929 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60dc106dc0 con 0x7f60dc0f41a0 2023-11-17T05:30:44.816 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:44.813+0000 7f60e4461700 1 -- 172.21.15.160:0/2305593929 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60dc107250 con 0x7f60dc0f41a0 2023-11-17T05:30:44.816 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:44.813+0000 7f60d2ffd700 1 -- 172.21.15.160:0/2305593929 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f60d8013f00 con 0x7f60dc0f41a0 2023-11-17T05:30:44.816 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:44.813+0000 7f60d2ffd700 1 -- 172.21.15.160:0/2305593929 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60d8014d90 con 0x7f60dc0f41a0 2023-11-17T05:30:44.816 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:44.814+0000 7f60d2ffd700 1 -- 172.21.15.160:0/2305593929 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f60d80095f0 con 0x7f60dc0f41a0 2023-11-17T05:30:44.816 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:44.814+0000 7f60e4461700 1 -- 172.21.15.160:0/2305593929 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60dc045ad0 con 0x7f60dc0f41a0 2023-11-17T05:30:44.816 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:44.814+0000 7f60d2ffd700 1 --2- 172.21.15.160:0/2305593929 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f60c802ff60 0x7f60c8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:30:44.816 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:44.814+0000 7f60d2ffd700 1 -- 172.21.15.160:0/2305593929 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f60d80410e0 con 0x7f60dc0f41a0 2023-11-17T05:30:44.816 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:44.814+0000 7f60e19fc700 1 --2- 172.21.15.160:0/2305593929 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f60c802ff60 0x7f60c8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:30:44.816 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:44.814+0000 7f60e19fc700 1 --2- 172.21.15.160:0/2305593929 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f60c802ff60 0x7f60c8032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f60cc0096a0 tx=0x7f60cc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:30:44.819 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:44.819+0000 7f60d2ffd700 1 -- 172.21.15.160:0/2305593929 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f60d8014420 con 0x7f60dc0f41a0 2023-11-17T05:30:45.027 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:45.026+0000 7f60e4461700 1 -- 172.21.15.160:0/2305593929 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f60dc108100 con 0x7f60dc0f41a0 2023-11-17T05:30:45.028 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:45.027+0000 7f60d2ffd700 1 -- 172.21.15.160:0/2305593929 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f60d8012980 con 0x7f60dc0f41a0 2023-11-17T05:30:45.028 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:30:45.028 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:30:45.030 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:45.029+0000 7f60e4461700 1 -- 172.21.15.160:0/2305593929 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f60c802ff60 msgr2=0x7f60c8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:30:45.030 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:45.029+0000 7f60e4461700 1 --2- 172.21.15.160:0/2305593929 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f60c802ff60 0x7f60c8032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f60cc0096a0 tx=0x7f60cc006b40).stop 2023-11-17T05:30:45.030 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:45.029+0000 7f60e4461700 1 -- 172.21.15.160:0/2305593929 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60dc0f41a0 msgr2=0x7f60dc1066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:30:45.030 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:45.029+0000 7f60e4461700 1 --2- 172.21.15.160:0/2305593929 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60dc0f41a0 0x7f60dc1066d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f60d8013510 tx=0x7f60d800b040).stop 2023-11-17T05:30:45.030 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:45.029+0000 7f60e4461700 1 -- 172.21.15.160:0/2305593929 shutdown_connections 2023-11-17T05:30:45.030 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:45.029+0000 7f60e4461700 1 --2- 172.21.15.160:0/2305593929 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f60c802ff60 0x7f60c8032420 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:30:45.030 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:45.029+0000 7f60e4461700 1 --2- 172.21.15.160:0/2305593929 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60dc0f41a0 0x7f60dc1066d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:30:45.030 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:45.029+0000 7f60e4461700 1 -- 172.21.15.160:0/2305593929 >> 172.21.15.160:0/2305593929 conn(0x7f60dc0efe40 msgr2=0x7f60dc0f0950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:30:45.030 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:45.029+0000 7f60e4461700 1 -- 172.21.15.160:0/2305593929 shutdown_connections 2023-11-17T05:30:45.030 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:45.029+0000 7f60e4461700 1 -- 172.21.15.160:0/2305593929 wait complete. 2023-11-17T05:30:45.031 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:30:45.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:45 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:30:45.028123+0000 mon.smithi003 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.160:0/2305593929' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:30:46.446 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:30:46.446 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:30:46.771 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:46 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:30:45.440553+0000 mgr.smithi003.eddcuc (mgr.14164) 164 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:47.610 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:47.608+0000 7fad16d6e700 1 -- 172.21.15.160:0/3937643578 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad100f53f0 msgr2=0x7fad100f5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:30:47.610 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:47.608+0000 7fad16d6e700 1 --2- 172.21.15.160:0/3937643578 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad100f53f0 0x7fad100f5810 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fad04009fb0 tx=0x7fad04009b30).stop 2023-11-17T05:30:47.610 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:47.608+0000 7fad16d6e700 1 -- 172.21.15.160:0/3937643578 shutdown_connections 2023-11-17T05:30:47.610 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:47.608+0000 7fad16d6e700 1 --2- 172.21.15.160:0/3937643578 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad100f53f0 0x7fad100f5810 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:30:47.610 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:47.608+0000 7fad16d6e700 1 -- 172.21.15.160:0/3937643578 >> 172.21.15.160:0/3937643578 conn(0x7fad100f0b70 msgr2=0x7fad100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:30:47.610 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:47.608+0000 7fad16d6e700 1 -- 172.21.15.160:0/3937643578 shutdown_connections 2023-11-17T05:30:47.611 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:47.608+0000 7fad16d6e700 1 -- 172.21.15.160:0/3937643578 wait complete. 2023-11-17T05:30:47.611 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:47.609+0000 7fad16d6e700 1 Processor -- start 2023-11-17T05:30:47.611 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:47.609+0000 7fad16d6e700 1 -- start start 2023-11-17T05:30:47.611 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:47.609+0000 7fad16d6e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad100f53f0 0x7fad101066c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:30:47.611 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:47.609+0000 7fad16d6e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad10106bb0 con 0x7fad100f53f0 2023-11-17T05:30:47.611 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:47.609+0000 7fad14b0a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad100f53f0 0x7fad101066c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:30:47.611 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:47.609+0000 7fad14b0a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad100f53f0 0x7fad101066c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:46766/0 (socket says 172.21.15.160:46766) 2023-11-17T05:30:47.611 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:47.609+0000 7fad14b0a700 1 -- 172.21.15.160:0/856955237 learned_addr learned my addr 172.21.15.160:0/856955237 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:30:47.611 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:47.609+0000 7fad14b0a700 1 -- 172.21.15.160:0/856955237 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad04005040 con 0x7fad100f53f0 2023-11-17T05:30:47.611 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:47.609+0000 7fad14b0a700 1 --2- 172.21.15.160:0/856955237 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad100f53f0 0x7fad101066c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fad04013510 tx=0x7fad0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:30:47.612 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:47.610+0000 7fad0d7fa700 1 -- 172.21.15.160:0/856955237 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad04009320 con 0x7fad100f53f0 2023-11-17T05:30:47.612 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:47.610+0000 7fad0d7fa700 1 -- 172.21.15.160:0/856955237 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fad04013f00 con 0x7fad100f53f0 2023-11-17T05:30:47.612 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:47.610+0000 7fad16d6e700 1 -- 172.21.15.160:0/856955237 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad10106db0 con 0x7fad100f53f0 2023-11-17T05:30:47.613 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:47.610+0000 7fad0d7fa700 1 -- 172.21.15.160:0/856955237 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad04014d90 con 0x7fad100f53f0 2023-11-17T05:30:47.613 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:47.610+0000 7fad16d6e700 1 -- 172.21.15.160:0/856955237 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad10107240 con 0x7fad100f53f0 2023-11-17T05:30:47.613 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:47.610+0000 7fad0d7fa700 1 -- 172.21.15.160:0/856955237 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fad040263a0 con 0x7fad100f53f0 2023-11-17T05:30:47.613 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:47.611+0000 7fad0d7fa700 1 --2- 172.21.15.160:0/856955237 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fad0002ff60 0x7fad00032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:30:47.613 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:47.611+0000 7fad0d7fa700 1 -- 172.21.15.160:0/856955237 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fad040410b0 con 0x7fad100f53f0 2023-11-17T05:30:47.613 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:47.611+0000 7fad16d6e700 1 -- 172.21.15.160:0/856955237 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad10045ad0 con 0x7fad100f53f0 2023-11-17T05:30:47.613 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:47.611+0000 7fad0ffff700 1 --2- 172.21.15.160:0/856955237 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fad0002ff60 0x7fad00032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:30:47.613 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:47.611+0000 7fad0ffff700 1 --2- 172.21.15.160:0/856955237 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fad0002ff60 0x7fad00032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7facfc0096a0 tx=0x7facfc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:30:47.622 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:47.616+0000 7fad0d7fa700 1 -- 172.21.15.160:0/856955237 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fad04012920 con 0x7fad100f53f0 2023-11-17T05:30:47.822 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:47.821+0000 7fad16d6e700 1 -- 172.21.15.160:0/856955237 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fad101080f0 con 0x7fad100f53f0 2023-11-17T05:30:47.823 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:47.822+0000 7fad0d7fa700 1 -- 172.21.15.160:0/856955237 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fad0402b020 con 0x7fad100f53f0 2023-11-17T05:30:47.823 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:30:47.823 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:30:47.824 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:47.823+0000 7fad16d6e700 1 -- 172.21.15.160:0/856955237 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fad0002ff60 msgr2=0x7fad00032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:30:47.824 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:47.823+0000 7fad16d6e700 1 --2- 172.21.15.160:0/856955237 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fad0002ff60 0x7fad00032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7facfc0096a0 tx=0x7facfc006b40).stop 2023-11-17T05:30:47.824 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:47.823+0000 7fad16d6e700 1 -- 172.21.15.160:0/856955237 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad100f53f0 msgr2=0x7fad101066c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:30:47.840 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:47.823+0000 7fad16d6e700 1 --2- 172.21.15.160:0/856955237 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad100f53f0 0x7fad101066c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fad04013510 tx=0x7fad0400b040).stop 2023-11-17T05:30:47.840 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:47.823+0000 7fad16d6e700 1 -- 172.21.15.160:0/856955237 shutdown_connections 2023-11-17T05:30:47.840 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:47.823+0000 7fad16d6e700 1 --2- 172.21.15.160:0/856955237 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fad0002ff60 0x7fad00032420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:30:47.840 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:47.823+0000 7fad16d6e700 1 --2- 172.21.15.160:0/856955237 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fad100f53f0 0x7fad101066c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:30:47.840 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:47.823+0000 7fad16d6e700 1 -- 172.21.15.160:0/856955237 >> 172.21.15.160:0/856955237 conn(0x7fad100f0b70 msgr2=0x7fad100f1640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:30:47.840 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:47.823+0000 7fad16d6e700 1 -- 172.21.15.160:0/856955237 shutdown_connections 2023-11-17T05:30:47.840 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:47.823+0000 7fad16d6e700 1 -- 172.21.15.160:0/856955237 wait complete. 2023-11-17T05:30:47.841 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:30:48.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:48 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:30:47.440769+0000 mgr.smithi003.eddcuc (mgr.14164) 165 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:48.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:48 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:30:47.822906+0000 mon.smithi003 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.160:0/856955237' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:30:49.211 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:30:49.212 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:30:50.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:49 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:30:49.441016+0000 mgr.smithi003.eddcuc (mgr.14164) 166 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:50.471 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:50.469+0000 7f9cf2e7e700 1 -- 172.21.15.160:0/145783405 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9cec0f51a0 msgr2=0x7f9cec0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:30:50.471 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:50.469+0000 7f9cf2e7e700 1 --2- 172.21.15.160:0/145783405 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9cec0f51a0 0x7f9cec0f55c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f9cdc009fb0 tx=0x7f9cdc009b30).stop 2023-11-17T05:30:50.471 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:50.469+0000 7f9cf2e7e700 1 -- 172.21.15.160:0/145783405 shutdown_connections 2023-11-17T05:30:50.471 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:50.469+0000 7f9cf2e7e700 1 --2- 172.21.15.160:0/145783405 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9cec0f51a0 0x7f9cec0f55c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:30:50.471 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:50.469+0000 7f9cf2e7e700 1 -- 172.21.15.160:0/145783405 >> 172.21.15.160:0/145783405 conn(0x7f9cec0f09a0 msgr2=0x7f9cec0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:30:50.471 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:50.469+0000 7f9cf2e7e700 1 -- 172.21.15.160:0/145783405 shutdown_connections 2023-11-17T05:30:50.471 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:50.469+0000 7f9cf2e7e700 1 -- 172.21.15.160:0/145783405 wait complete. 2023-11-17T05:30:50.471 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:50.469+0000 7f9cf2e7e700 1 Processor -- start 2023-11-17T05:30:50.471 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:50.469+0000 7f9cf2e7e700 1 -- start start 2023-11-17T05:30:50.471 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:50.470+0000 7f9cf2e7e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9cec0f51a0 0x7f9cec1053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:30:50.471 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:50.470+0000 7f9cf2e7e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cec1058a0 con 0x7f9cec0f51a0 2023-11-17T05:30:50.471 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:50.470+0000 7f9cf1e7c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9cec0f51a0 0x7f9cec1053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:30:50.472 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:50.470+0000 7f9cf1e7c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9cec0f51a0 0x7f9cec1053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:43658/0 (socket says 172.21.15.160:43658) 2023-11-17T05:30:50.472 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:50.470+0000 7f9cf1e7c700 1 -- 172.21.15.160:0/4187826538 learned_addr learned my addr 172.21.15.160:0/4187826538 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:30:50.473 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:50.470+0000 7f9cf1e7c700 1 -- 172.21.15.160:0/4187826538 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cdc005040 con 0x7f9cec0f51a0 2023-11-17T05:30:50.473 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:50.470+0000 7f9cf1e7c700 1 --2- 172.21.15.160:0/4187826538 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9cec0f51a0 0x7f9cec1053b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f9cdc013510 tx=0x7f9cdc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:30:50.473 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:50.470+0000 7f9ce27fc700 1 -- 172.21.15.160:0/4187826538 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cdc009320 con 0x7f9cec0f51a0 2023-11-17T05:30:50.473 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:50.471+0000 7f9cf2e7e700 1 -- 172.21.15.160:0/4187826538 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cec105aa0 con 0x7f9cec0f51a0 2023-11-17T05:30:50.473 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:50.471+0000 7f9ce27fc700 1 -- 172.21.15.160:0/4187826538 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9cdc013f00 con 0x7f9cec0f51a0 2023-11-17T05:30:50.473 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:50.471+0000 7f9ce27fc700 1 -- 172.21.15.160:0/4187826538 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cdc014dc0 con 0x7f9cec0f51a0 2023-11-17T05:30:50.473 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:50.471+0000 7f9cf2e7e700 1 -- 172.21.15.160:0/4187826538 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cec105f30 con 0x7f9cec0f51a0 2023-11-17T05:30:50.474 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:50.471+0000 7f9ce27fc700 1 -- 172.21.15.160:0/4187826538 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9cdc0095f0 con 0x7f9cec0f51a0 2023-11-17T05:30:50.474 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:50.471+0000 7f9cf2e7e700 1 -- 172.21.15.160:0/4187826538 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cec045ad0 con 0x7f9cec0f51a0 2023-11-17T05:30:50.474 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:50.472+0000 7f9ce27fc700 1 --2- 172.21.15.160:0/4187826538 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f9cd802ff60 0x7f9cd8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:30:50.474 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:50.472+0000 7f9ce27fc700 1 -- 172.21.15.160:0/4187826538 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9cdc041000 con 0x7f9cec0f51a0 2023-11-17T05:30:50.474 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:50.472+0000 7f9cf167b700 1 --2- 172.21.15.160:0/4187826538 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f9cd802ff60 0x7f9cd8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:30:50.474 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:50.472+0000 7f9cf167b700 1 --2- 172.21.15.160:0/4187826538 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f9cd802ff60 0x7f9cd8032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f9ce80096a0 tx=0x7f9ce8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:30:50.477 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:50.477+0000 7f9ce27fc700 1 -- 172.21.15.160:0/4187826538 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9cdc014420 con 0x7f9cec0f51a0 2023-11-17T05:30:50.682 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:50.681+0000 7f9cf2e7e700 1 -- 172.21.15.160:0/4187826538 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9cec106de0 con 0x7f9cec0f51a0 2023-11-17T05:30:50.682 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:50.681+0000 7f9ce27fc700 1 -- 172.21.15.160:0/4187826538 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f9cdc010230 con 0x7f9cec0f51a0 2023-11-17T05:30:50.682 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:30:50.682 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:30:50.684 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:50.683+0000 7f9cf2e7e700 1 -- 172.21.15.160:0/4187826538 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f9cd802ff60 msgr2=0x7f9cd8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:30:50.684 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:50.683+0000 7f9cf2e7e700 1 --2- 172.21.15.160:0/4187826538 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f9cd802ff60 0x7f9cd8032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f9ce80096a0 tx=0x7f9ce8006b40).stop 2023-11-17T05:30:50.684 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:50.683+0000 7f9cf2e7e700 1 -- 172.21.15.160:0/4187826538 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9cec0f51a0 msgr2=0x7f9cec1053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:30:50.684 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:50.683+0000 7f9cf2e7e700 1 --2- 172.21.15.160:0/4187826538 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9cec0f51a0 0x7f9cec1053b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f9cdc013510 tx=0x7f9cdc00b040).stop 2023-11-17T05:30:50.684 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:50.683+0000 7f9cf2e7e700 1 -- 172.21.15.160:0/4187826538 shutdown_connections 2023-11-17T05:30:50.684 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:50.683+0000 7f9cf2e7e700 1 --2- 172.21.15.160:0/4187826538 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f9cd802ff60 0x7f9cd8032420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:30:50.684 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:50.683+0000 7f9cf2e7e700 1 --2- 172.21.15.160:0/4187826538 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9cec0f51a0 0x7f9cec1053b0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:30:50.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:50.683+0000 7f9cf2e7e700 1 -- 172.21.15.160:0/4187826538 >> 172.21.15.160:0/4187826538 conn(0x7f9cec0f09a0 msgr2=0x7f9cec0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:30:50.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:50.683+0000 7f9cf2e7e700 1 -- 172.21.15.160:0/4187826538 shutdown_connections 2023-11-17T05:30:50.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:50.683+0000 7f9cf2e7e700 1 -- 172.21.15.160:0/4187826538 wait complete. 2023-11-17T05:30:50.685 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:30:51.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:50 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:30:50.682685+0000 mon.smithi003 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.160:0/4187826538' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:30:52.081 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:30:52.081 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:30:52.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:51 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:30:51.441293+0000 mgr.smithi003.eddcuc (mgr.14164) 167 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:53.238 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:53.237+0000 7fe49c130700 1 -- 172.21.15.160:0/1098463158 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4940f6510 msgr2=0x7fe4940f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:30:53.239 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:53.237+0000 7fe49c130700 1 --2- 172.21.15.160:0/1098463158 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4940f6510 0x7fe4940f6930 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fe484009fb0 tx=0x7fe484009b30).stop 2023-11-17T05:30:53.239 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:53.237+0000 7fe49c130700 1 -- 172.21.15.160:0/1098463158 shutdown_connections 2023-11-17T05:30:53.239 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:53.237+0000 7fe49c130700 1 --2- 172.21.15.160:0/1098463158 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4940f6510 0x7fe4940f6930 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:30:53.239 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:53.237+0000 7fe49c130700 1 -- 172.21.15.160:0/1098463158 >> 172.21.15.160:0/1098463158 conn(0x7fe4940f1c70 msgr2=0x7fe4940f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:30:53.239 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:53.237+0000 7fe49c130700 1 -- 172.21.15.160:0/1098463158 shutdown_connections 2023-11-17T05:30:53.239 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:53.237+0000 7fe49c130700 1 -- 172.21.15.160:0/1098463158 wait complete. 2023-11-17T05:30:53.239 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:53.238+0000 7fe49c130700 1 Processor -- start 2023-11-17T05:30:53.239 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:53.238+0000 7fe49c130700 1 -- start start 2023-11-17T05:30:53.241 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:53.238+0000 7fe49c130700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4940f6510 0x7fe494106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:30:53.241 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:53.238+0000 7fe49c130700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe494106c20 con 0x7fe4940f6510 2023-11-17T05:30:53.241 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:53.238+0000 7fe499ecc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4940f6510 0x7fe494106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:30:53.241 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:53.238+0000 7fe499ecc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4940f6510 0x7fe494106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:43672/0 (socket says 172.21.15.160:43672) 2023-11-17T05:30:53.241 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:53.238+0000 7fe499ecc700 1 -- 172.21.15.160:0/1951532308 learned_addr learned my addr 172.21.15.160:0/1951532308 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:30:53.241 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:53.239+0000 7fe499ecc700 1 -- 172.21.15.160:0/1951532308 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe484005040 con 0x7fe4940f6510 2023-11-17T05:30:53.241 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:53.239+0000 7fe499ecc700 1 --2- 172.21.15.160:0/1951532308 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4940f6510 0x7fe494106730 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7fe484014000 tx=0x7fe48400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:30:53.241 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:53.239+0000 7fe48a7fc700 1 -- 172.21.15.160:0/1951532308 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4840095f0 con 0x7fe4940f6510 2023-11-17T05:30:53.241 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:53.239+0000 7fe49c130700 1 -- 172.21.15.160:0/1951532308 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe494106e20 con 0x7fe4940f6510 2023-11-17T05:30:53.241 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:53.239+0000 7fe48a7fc700 1 -- 172.21.15.160:0/1951532308 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe484007e70 con 0x7fe4940f6510 2023-11-17T05:30:53.241 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:53.239+0000 7fe48a7fc700 1 -- 172.21.15.160:0/1951532308 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe484014ea0 con 0x7fe4940f6510 2023-11-17T05:30:53.241 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:53.239+0000 7fe49c130700 1 -- 172.21.15.160:0/1951532308 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4941072b0 con 0x7fe4940f6510 2023-11-17T05:30:53.242 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:53.240+0000 7fe49c130700 1 -- 172.21.15.160:0/1951532308 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe494045ad0 con 0x7fe4940f6510 2023-11-17T05:30:53.242 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:53.240+0000 7fe48a7fc700 1 -- 172.21.15.160:0/1951532308 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe484025070 con 0x7fe4940f6510 2023-11-17T05:30:53.242 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:53.240+0000 7fe48a7fc700 1 --2- 172.21.15.160:0/1951532308 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe48002ff60 0x7fe480032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:30:53.242 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:53.240+0000 7fe48a7fc700 1 -- 172.21.15.160:0/1951532308 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe484040a10 con 0x7fe4940f6510 2023-11-17T05:30:53.242 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:53.240+0000 7fe4996cb700 1 --2- 172.21.15.160:0/1951532308 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe48002ff60 0x7fe480032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:30:53.242 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:53.241+0000 7fe4996cb700 1 --2- 172.21.15.160:0/1951532308 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe48002ff60 0x7fe480032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fe4900096a0 tx=0x7fe490006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:30:53.245 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:53.245+0000 7fe48a7fc700 1 -- 172.21.15.160:0/1951532308 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe4840079c0 con 0x7fe4940f6510 2023-11-17T05:30:53.453 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:53.453+0000 7fe49c130700 1 -- 172.21.15.160:0/1951532308 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe4940fbf60 con 0x7fe4940f6510 2023-11-17T05:30:53.454 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:53.453+0000 7fe48a7fc700 1 -- 172.21.15.160:0/1951532308 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fe48400f300 con 0x7fe4940f6510 2023-11-17T05:30:53.454 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:30:53.455 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:30:53.456 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:53.455+0000 7fe49c130700 1 -- 172.21.15.160:0/1951532308 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe48002ff60 msgr2=0x7fe480032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:30:53.456 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:53.455+0000 7fe49c130700 1 --2- 172.21.15.160:0/1951532308 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe48002ff60 0x7fe480032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fe4900096a0 tx=0x7fe490006b40).stop 2023-11-17T05:30:53.456 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:53.455+0000 7fe49c130700 1 -- 172.21.15.160:0/1951532308 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4940f6510 msgr2=0x7fe494106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:30:53.456 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:53.455+0000 7fe49c130700 1 --2- 172.21.15.160:0/1951532308 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4940f6510 0x7fe494106730 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7fe484014000 tx=0x7fe48400b040).stop 2023-11-17T05:30:53.456 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:53.455+0000 7fe49c130700 1 -- 172.21.15.160:0/1951532308 shutdown_connections 2023-11-17T05:30:53.456 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:53.455+0000 7fe49c130700 1 --2- 172.21.15.160:0/1951532308 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe48002ff60 0x7fe480032420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:30:53.457 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:53.455+0000 7fe49c130700 1 --2- 172.21.15.160:0/1951532308 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe4940f6510 0x7fe494106730 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:30:53.457 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:53.455+0000 7fe49c130700 1 -- 172.21.15.160:0/1951532308 >> 172.21.15.160:0/1951532308 conn(0x7fe4940f1c70 msgr2=0x7fe4940f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:30:53.457 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:53.455+0000 7fe49c130700 1 -- 172.21.15.160:0/1951532308 shutdown_connections 2023-11-17T05:30:53.457 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:53.455+0000 7fe49c130700 1 -- 172.21.15.160:0/1951532308 wait complete. 2023-11-17T05:30:53.457 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:30:53.655 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:53 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:30:53.454595+0000 mon.smithi003 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.160:0/1951532308' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:30:54.753 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:54 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:30:53.441610+0000 mgr.smithi003.eddcuc (mgr.14164) 168 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:54.823 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:30:54.823 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:30:56.014 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:56.012+0000 7fa132a29700 1 -- 172.21.15.160:0/1748687584 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa12c0f5410 msgr2=0x7fa12c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:30:56.014 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:56.012+0000 7fa132a29700 1 --2- 172.21.15.160:0/1748687584 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa12c0f5410 0x7fa12c0f5830 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7fa11c009fb0 tx=0x7fa11c009b30).stop 2023-11-17T05:30:56.014 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:56.012+0000 7fa132a29700 1 -- 172.21.15.160:0/1748687584 shutdown_connections 2023-11-17T05:30:56.015 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:56.012+0000 7fa132a29700 1 --2- 172.21.15.160:0/1748687584 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa12c0f5410 0x7fa12c0f5830 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:30:56.015 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:56.012+0000 7fa132a29700 1 -- 172.21.15.160:0/1748687584 >> 172.21.15.160:0/1748687584 conn(0x7fa12c0f0b70 msgr2=0x7fa12c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:30:56.015 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:56.012+0000 7fa132a29700 1 -- 172.21.15.160:0/1748687584 shutdown_connections 2023-11-17T05:30:56.015 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:56.012+0000 7fa132a29700 1 -- 172.21.15.160:0/1748687584 wait complete. 2023-11-17T05:30:56.015 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:56.013+0000 7fa132a29700 1 Processor -- start 2023-11-17T05:30:56.015 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:56.013+0000 7fa132a29700 1 -- start start 2023-11-17T05:30:56.015 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:56.013+0000 7fa132a29700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa12c0f5410 0x7fa12c1066b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:30:56.015 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:56.013+0000 7fa132a29700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa12c106ba0 con 0x7fa12c0f5410 2023-11-17T05:30:56.015 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:56.013+0000 7fa12bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa12c0f5410 0x7fa12c1066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:30:56.015 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:56.013+0000 7fa12bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa12c0f5410 0x7fa12c1066b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:43700/0 (socket says 172.21.15.160:43700) 2023-11-17T05:30:56.015 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:56.013+0000 7fa12bfff700 1 -- 172.21.15.160:0/2120068890 learned_addr learned my addr 172.21.15.160:0/2120068890 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:30:56.015 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:56.013+0000 7fa12bfff700 1 -- 172.21.15.160:0/2120068890 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa11c005040 con 0x7fa12c0f5410 2023-11-17T05:30:56.015 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:56.014+0000 7fa12bfff700 1 --2- 172.21.15.160:0/2120068890 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa12c0f5410 0x7fa12c1066b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fa11c013510 tx=0x7fa11c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:30:56.016 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:56.014+0000 7fa128ff9700 1 -- 172.21.15.160:0/2120068890 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa11c009320 con 0x7fa12c0f5410 2023-11-17T05:30:56.017 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:56.014+0000 7fa132a29700 1 -- 172.21.15.160:0/2120068890 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa12c106da0 con 0x7fa12c0f5410 2023-11-17T05:30:56.017 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:56.014+0000 7fa128ff9700 1 -- 172.21.15.160:0/2120068890 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa11c013f00 con 0x7fa12c0f5410 2023-11-17T05:30:56.017 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:56.014+0000 7fa128ff9700 1 -- 172.21.15.160:0/2120068890 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa11c014d90 con 0x7fa12c0f5410 2023-11-17T05:30:56.017 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:56.014+0000 7fa132a29700 1 -- 172.21.15.160:0/2120068890 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa12c107230 con 0x7fa12c0f5410 2023-11-17T05:30:56.017 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:56.015+0000 7fa128ff9700 1 -- 172.21.15.160:0/2120068890 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa11c0263a0 con 0x7fa12c0f5410 2023-11-17T05:30:56.017 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:56.015+0000 7fa132a29700 1 -- 172.21.15.160:0/2120068890 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa12c045ad0 con 0x7fa12c0f5410 2023-11-17T05:30:56.017 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:56.015+0000 7fa128ff9700 1 --2- 172.21.15.160:0/2120068890 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa11402ff60 0x7fa114032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:30:56.017 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:56.015+0000 7fa128ff9700 1 -- 172.21.15.160:0/2120068890 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa11c0410a0 con 0x7fa12c0f5410 2023-11-17T05:30:56.017 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:56.015+0000 7fa12b7fe700 1 --2- 172.21.15.160:0/2120068890 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa11402ff60 0x7fa114032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:30:56.017 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:56.015+0000 7fa12b7fe700 1 --2- 172.21.15.160:0/2120068890 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa11402ff60 0x7fa114032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fa1200096a0 tx=0x7fa120006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:30:56.021 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:56.020+0000 7fa128ff9700 1 -- 172.21.15.160:0/2120068890 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa11c0126f0 con 0x7fa12c0f5410 2023-11-17T05:30:56.225 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:56.224+0000 7fa132a29700 1 -- 172.21.15.160:0/2120068890 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa12c1080e0 con 0x7fa12c0f5410 2023-11-17T05:30:56.226 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:56.225+0000 7fa128ff9700 1 -- 172.21.15.160:0/2120068890 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fa11c012330 con 0x7fa12c0f5410 2023-11-17T05:30:56.226 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:30:56.227 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:30:56.228 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:56.227+0000 7fa132a29700 1 -- 172.21.15.160:0/2120068890 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa11402ff60 msgr2=0x7fa114032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:30:56.228 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:56.227+0000 7fa132a29700 1 --2- 172.21.15.160:0/2120068890 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa11402ff60 0x7fa114032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fa1200096a0 tx=0x7fa120006b40).stop 2023-11-17T05:30:56.228 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:56.227+0000 7fa132a29700 1 -- 172.21.15.160:0/2120068890 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa12c0f5410 msgr2=0x7fa12c1066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:30:56.228 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:56.227+0000 7fa132a29700 1 --2- 172.21.15.160:0/2120068890 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa12c0f5410 0x7fa12c1066b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fa11c013510 tx=0x7fa11c00b040).stop 2023-11-17T05:30:56.228 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:56.227+0000 7fa132a29700 1 -- 172.21.15.160:0/2120068890 shutdown_connections 2023-11-17T05:30:56.229 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:56.227+0000 7fa132a29700 1 --2- 172.21.15.160:0/2120068890 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa11402ff60 0x7fa114032420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:30:56.229 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:56.227+0000 7fa132a29700 1 --2- 172.21.15.160:0/2120068890 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa12c0f5410 0x7fa12c1066b0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:30:56.229 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:56.227+0000 7fa132a29700 1 -- 172.21.15.160:0/2120068890 >> 172.21.15.160:0/2120068890 conn(0x7fa12c0f0b70 msgr2=0x7fa12c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:30:56.229 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:56.227+0000 7fa132a29700 1 -- 172.21.15.160:0/2120068890 shutdown_connections 2023-11-17T05:30:56.229 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:56.227+0000 7fa132a29700 1 -- 172.21.15.160:0/2120068890 wait complete. 2023-11-17T05:30:56.229 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:30:56.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:56 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:30:55.441808+0000 mgr.smithi003.eddcuc (mgr.14164) 169 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:56.771 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:56 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:30:55.442115+0000 mon.smithi003 (mon.0) 189 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' 2023-11-17T05:30:56.771 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:56 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cephadm 2023-11-17T05:30:55.443933+0000 mgr.smithi003.eddcuc (mgr.14164) 170 : cephadm [INF] Deploying daemon node-exporter.smithi003 on smithi003 2023-11-17T05:30:56.771 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:56 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:30:56.226344+0000 mon.smithi003 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.160:0/2120068890' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:30:57.597 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:30:57.598 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:30:58.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:58 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:30:57.442155+0000 mgr.smithi003.eddcuc (mgr.14164) 171 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:30:58.884 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:58.883+0000 7f1cbff2b700 1 -- 172.21.15.160:0/4094698291 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cb80f6510 msgr2=0x7f1cb80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:30:58.884 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:58.883+0000 7f1cbff2b700 1 --2- 172.21.15.160:0/4094698291 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cb80f6510 0x7f1cb80f6930 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f1ca8009fb0 tx=0x7f1ca8009b30).stop 2023-11-17T05:30:58.884 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:58.883+0000 7f1cbff2b700 1 -- 172.21.15.160:0/4094698291 shutdown_connections 2023-11-17T05:30:58.884 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:58.883+0000 7f1cbff2b700 1 --2- 172.21.15.160:0/4094698291 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cb80f6510 0x7f1cb80f6930 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:30:58.885 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:58.883+0000 7f1cbff2b700 1 -- 172.21.15.160:0/4094698291 >> 172.21.15.160:0/4094698291 conn(0x7f1cb80f1c70 msgr2=0x7f1cb80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:30:58.885 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:58.883+0000 7f1cbff2b700 1 -- 172.21.15.160:0/4094698291 shutdown_connections 2023-11-17T05:30:58.885 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:58.883+0000 7f1cbff2b700 1 -- 172.21.15.160:0/4094698291 wait complete. 2023-11-17T05:30:58.885 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:58.883+0000 7f1cbff2b700 1 Processor -- start 2023-11-17T05:30:58.885 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:58.883+0000 7f1cbff2b700 1 -- start start 2023-11-17T05:30:58.885 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:58.883+0000 7f1cbff2b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cb80f6510 0x7f1cb806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:30:58.885 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:58.884+0000 7f1cbff2b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb8069e20 con 0x7f1cb80f6510 2023-11-17T05:30:58.887 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:58.884+0000 7f1cbdcc7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cb80f6510 0x7f1cb806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:30:58.887 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:58.884+0000 7f1cbdcc7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cb80f6510 0x7f1cb806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:43716/0 (socket says 172.21.15.160:43716) 2023-11-17T05:30:58.887 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:58.884+0000 7f1cbdcc7700 1 -- 172.21.15.160:0/3092560363 learned_addr learned my addr 172.21.15.160:0/3092560363 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:30:58.887 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:58.884+0000 7f1cbdcc7700 1 -- 172.21.15.160:0/3092560363 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ca8005040 con 0x7f1cb80f6510 2023-11-17T05:30:58.887 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:58.884+0000 7f1cbdcc7700 1 --2- 172.21.15.160:0/3092560363 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cb80f6510 0x7f1cb806b9a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f1ca8014000 tx=0x7f1ca800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:30:58.887 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:58.884+0000 7f1cae7fc700 1 -- 172.21.15.160:0/3092560363 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ca80095f0 con 0x7f1cb80f6510 2023-11-17T05:30:58.887 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:58.885+0000 7f1cbff2b700 1 -- 172.21.15.160:0/3092560363 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cb8069fc0 con 0x7f1cb80f6510 2023-11-17T05:30:58.887 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:58.885+0000 7f1cae7fc700 1 -- 172.21.15.160:0/3092560363 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1ca8007e70 con 0x7f1cb80f6510 2023-11-17T05:30:58.887 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:58.885+0000 7f1cae7fc700 1 -- 172.21.15.160:0/3092560363 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ca8014ea0 con 0x7f1cb80f6510 2023-11-17T05:30:58.887 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:58.885+0000 7f1cbff2b700 1 -- 172.21.15.160:0/3092560363 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cb806a450 con 0x7f1cb80f6510 2023-11-17T05:30:58.888 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:58.885+0000 7f1cae7fc700 1 -- 172.21.15.160:0/3092560363 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1ca8025070 con 0x7f1cb80f6510 2023-11-17T05:30:58.888 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:58.885+0000 7f1cbff2b700 1 -- 172.21.15.160:0/3092560363 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cb8059100 con 0x7f1cb80f6510 2023-11-17T05:30:58.888 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:58.886+0000 7f1cae7fc700 1 --2- 172.21.15.160:0/3092560363 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f1ca402ff60 0x7f1ca4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:30:58.888 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:58.886+0000 7f1cae7fc700 1 -- 172.21.15.160:0/3092560363 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1ca8040a00 con 0x7f1cb80f6510 2023-11-17T05:30:58.888 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:58.886+0000 7f1cbd4c6700 1 --2- 172.21.15.160:0/3092560363 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f1ca402ff60 0x7f1ca4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:30:58.888 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:58.886+0000 7f1cbd4c6700 1 --2- 172.21.15.160:0/3092560363 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f1ca402ff60 0x7f1ca4032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f1cb40096a0 tx=0x7f1cb4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:30:58.891 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:58.890+0000 7f1cae7fc700 1 -- 172.21.15.160:0/3092560363 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1ca80079c0 con 0x7f1cb80f6510 2023-11-17T05:30:59.096 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:59.094+0000 7f1cbff2b700 1 -- 172.21.15.160:0/3092560363 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1cb8045ad0 con 0x7f1cb80f6510 2023-11-17T05:30:59.096 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:59.095+0000 7f1cae7fc700 1 -- 172.21.15.160:0/3092560363 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f1ca800f300 con 0x7f1cb80f6510 2023-11-17T05:30:59.096 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:30:59.096 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:30:59.099 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:59.097+0000 7f1cbff2b700 1 -- 172.21.15.160:0/3092560363 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f1ca402ff60 msgr2=0x7f1ca4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:30:59.099 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:59.097+0000 7f1cbff2b700 1 --2- 172.21.15.160:0/3092560363 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f1ca402ff60 0x7f1ca4032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f1cb40096a0 tx=0x7f1cb4006b40).stop 2023-11-17T05:30:59.099 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:59.097+0000 7f1cbff2b700 1 -- 172.21.15.160:0/3092560363 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cb80f6510 msgr2=0x7f1cb806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:30:59.100 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:59.097+0000 7f1cbff2b700 1 --2- 172.21.15.160:0/3092560363 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cb80f6510 0x7f1cb806b9a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f1ca8014000 tx=0x7f1ca800b040).stop 2023-11-17T05:30:59.100 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:59.097+0000 7f1cbff2b700 1 -- 172.21.15.160:0/3092560363 shutdown_connections 2023-11-17T05:30:59.100 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:59.097+0000 7f1cbff2b700 1 --2- 172.21.15.160:0/3092560363 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f1ca402ff60 0x7f1ca4032420 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:30:59.100 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:59.097+0000 7f1cbff2b700 1 --2- 172.21.15.160:0/3092560363 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1cb80f6510 0x7f1cb806b9a0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:30:59.100 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:59.097+0000 7f1cbff2b700 1 -- 172.21.15.160:0/3092560363 >> 172.21.15.160:0/3092560363 conn(0x7f1cb80f1c70 msgr2=0x7f1cb80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:30:59.100 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:59.097+0000 7f1cbff2b700 1 -- 172.21.15.160:0/3092560363 shutdown_connections 2023-11-17T05:30:59.100 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:30:59.097+0000 7f1cbff2b700 1 -- 172.21.15.160:0/3092560363 wait complete. 2023-11-17T05:30:59.100 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:31:00.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:59 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:30:59.096131+0000 mon.smithi003 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.160:0/3092560363' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:31:00.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:30:59 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:30:59.442556+0000 mgr.smithi003.eddcuc (mgr.14164) 172 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:31:00.484 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:31:00.485 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:31:01.700 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:01.699+0000 7ff562918700 1 -- 172.21.15.160:0/445701411 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff55c0f6510 msgr2=0x7ff55c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:01.701 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:01.699+0000 7ff562918700 1 --2- 172.21.15.160:0/445701411 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff55c0f6510 0x7ff55c0f6930 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7ff54c009fb0 tx=0x7ff54c009b30).stop 2023-11-17T05:31:01.701 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:01.699+0000 7ff562918700 1 -- 172.21.15.160:0/445701411 shutdown_connections 2023-11-17T05:31:01.701 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:01.699+0000 7ff562918700 1 --2- 172.21.15.160:0/445701411 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff55c0f6510 0x7ff55c0f6930 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:01.701 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:01.699+0000 7ff562918700 1 -- 172.21.15.160:0/445701411 >> 172.21.15.160:0/445701411 conn(0x7ff55c0f1c70 msgr2=0x7ff55c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:31:01.701 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:01.699+0000 7ff562918700 1 -- 172.21.15.160:0/445701411 shutdown_connections 2023-11-17T05:31:01.701 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:01.699+0000 7ff562918700 1 -- 172.21.15.160:0/445701411 wait complete. 2023-11-17T05:31:01.701 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:01.700+0000 7ff562918700 1 Processor -- start 2023-11-17T05:31:01.701 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:01.700+0000 7ff562918700 1 -- start start 2023-11-17T05:31:01.703 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:01.700+0000 7ff562918700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff55c0f6510 0x7ff55c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:31:01.703 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:01.700+0000 7ff562918700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff55c106c20 con 0x7ff55c0f6510 2023-11-17T05:31:01.703 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:01.700+0000 7ff55bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff55c0f6510 0x7ff55c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:31:01.703 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:01.700+0000 7ff55bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff55c0f6510 0x7ff55c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:41610/0 (socket says 172.21.15.160:41610) 2023-11-17T05:31:01.703 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:01.700+0000 7ff55bfff700 1 -- 172.21.15.160:0/3233746610 learned_addr learned my addr 172.21.15.160:0/3233746610 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:31:01.703 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:01.701+0000 7ff55bfff700 1 -- 172.21.15.160:0/3233746610 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff54c005040 con 0x7ff55c0f6510 2023-11-17T05:31:01.703 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:01.701+0000 7ff55bfff700 1 --2- 172.21.15.160:0/3233746610 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff55c0f6510 0x7ff55c106730 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7ff54c013510 tx=0x7ff54c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:31:01.703 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:01.701+0000 7ff558ff9700 1 -- 172.21.15.160:0/3233746610 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff54c009320 con 0x7ff55c0f6510 2023-11-17T05:31:01.703 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:01.701+0000 7ff562918700 1 -- 172.21.15.160:0/3233746610 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff55c106e20 con 0x7ff55c0f6510 2023-11-17T05:31:01.703 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:01.701+0000 7ff558ff9700 1 -- 172.21.15.160:0/3233746610 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff54c013f00 con 0x7ff55c0f6510 2023-11-17T05:31:01.704 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:01.701+0000 7ff558ff9700 1 -- 172.21.15.160:0/3233746610 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff54c014d90 con 0x7ff55c0f6510 2023-11-17T05:31:01.704 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:01.701+0000 7ff562918700 1 -- 172.21.15.160:0/3233746610 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff55c1072b0 con 0x7ff55c0f6510 2023-11-17T05:31:01.704 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:01.702+0000 7ff558ff9700 1 -- 172.21.15.160:0/3233746610 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff54c014420 con 0x7ff55c0f6510 2023-11-17T05:31:01.704 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:01.702+0000 7ff562918700 1 -- 172.21.15.160:0/3233746610 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff55c045ad0 con 0x7ff55c0f6510 2023-11-17T05:31:01.704 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:01.702+0000 7ff558ff9700 1 --2- 172.21.15.160:0/3233746610 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7ff54802ff60 0x7ff548032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:31:01.704 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:01.702+0000 7ff558ff9700 1 -- 172.21.15.160:0/3233746610 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff54c0410c0 con 0x7ff55c0f6510 2023-11-17T05:31:01.704 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:01.702+0000 7ff55b7fe700 1 --2- 172.21.15.160:0/3233746610 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7ff54802ff60 0x7ff548032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:31:01.704 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:01.703+0000 7ff55b7fe700 1 --2- 172.21.15.160:0/3233746610 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7ff54802ff60 0x7ff548032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7ff5440096a0 tx=0x7ff544006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:31:01.708 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:01.707+0000 7ff558ff9700 1 -- 172.21.15.160:0/3233746610 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff54c0095f0 con 0x7ff55c0f6510 2023-11-17T05:31:01.911 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:01.910+0000 7ff562918700 1 -- 172.21.15.160:0/3233746610 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff55c0fbf60 con 0x7ff55c0f6510 2023-11-17T05:31:01.912 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:01.911+0000 7ff558ff9700 1 -- 172.21.15.160:0/3233746610 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7ff54c01db70 con 0x7ff55c0f6510 2023-11-17T05:31:01.912 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:31:01.912 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:31:01.914 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:01.913+0000 7ff562918700 1 -- 172.21.15.160:0/3233746610 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7ff54802ff60 msgr2=0x7ff548032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:01.914 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:01.913+0000 7ff562918700 1 --2- 172.21.15.160:0/3233746610 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7ff54802ff60 0x7ff548032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7ff5440096a0 tx=0x7ff544006b40).stop 2023-11-17T05:31:01.914 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:01.913+0000 7ff562918700 1 -- 172.21.15.160:0/3233746610 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff55c0f6510 msgr2=0x7ff55c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:01.914 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:01.913+0000 7ff562918700 1 --2- 172.21.15.160:0/3233746610 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff55c0f6510 0x7ff55c106730 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7ff54c013510 tx=0x7ff54c00b040).stop 2023-11-17T05:31:01.914 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:01.913+0000 7ff562918700 1 -- 172.21.15.160:0/3233746610 shutdown_connections 2023-11-17T05:31:01.914 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:01.913+0000 7ff562918700 1 --2- 172.21.15.160:0/3233746610 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7ff54802ff60 0x7ff548032420 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:01.914 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:01.913+0000 7ff562918700 1 --2- 172.21.15.160:0/3233746610 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff55c0f6510 0x7ff55c106730 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:01.914 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:01.913+0000 7ff562918700 1 -- 172.21.15.160:0/3233746610 >> 172.21.15.160:0/3233746610 conn(0x7ff55c0f1c70 msgr2=0x7ff55c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:31:01.914 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:01.913+0000 7ff562918700 1 -- 172.21.15.160:0/3233746610 shutdown_connections 2023-11-17T05:31:01.914 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:01.913+0000 7ff562918700 1 -- 172.21.15.160:0/3233746610 wait complete. 2023-11-17T05:31:01.915 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:31:02.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:02 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:31:01.442894+0000 mgr.smithi003.eddcuc (mgr.14164) 173 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:31:02.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:02 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:31:01.912267+0000 mon.smithi003 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.160:0/3233746610' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:31:03.351 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:31:03.352 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:31:04.523 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:04.522+0000 7f94379bd700 1 -- 172.21.15.160:0/1186158194 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94300f5410 msgr2=0x7f94300f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:04.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:04.522+0000 7f94379bd700 1 --2- 172.21.15.160:0/1186158194 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94300f5410 0x7f94300f5830 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f9420009fb0 tx=0x7f9420009b30).stop 2023-11-17T05:31:04.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:04.522+0000 7f94379bd700 1 -- 172.21.15.160:0/1186158194 shutdown_connections 2023-11-17T05:31:04.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:04.522+0000 7f94379bd700 1 --2- 172.21.15.160:0/1186158194 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94300f5410 0x7f94300f5830 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:04.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:04.522+0000 7f94379bd700 1 -- 172.21.15.160:0/1186158194 >> 172.21.15.160:0/1186158194 conn(0x7f94300f0b70 msgr2=0x7f94300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:31:04.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:04.522+0000 7f94379bd700 1 -- 172.21.15.160:0/1186158194 shutdown_connections 2023-11-17T05:31:04.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:04.522+0000 7f94379bd700 1 -- 172.21.15.160:0/1186158194 wait complete. 2023-11-17T05:31:04.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:04.523+0000 7f94379bd700 1 Processor -- start 2023-11-17T05:31:04.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:04.523+0000 7f94379bd700 1 -- start start 2023-11-17T05:31:04.526 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:04.523+0000 7f94379bd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94300f5410 0x7f9430105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:31:04.526 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:04.523+0000 7f94379bd700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9430105b20 con 0x7f94300f5410 2023-11-17T05:31:04.526 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:04.523+0000 7f9435759700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94300f5410 0x7f9430105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:31:04.526 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:04.523+0000 7f9435759700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94300f5410 0x7f9430105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:41640/0 (socket says 172.21.15.160:41640) 2023-11-17T05:31:04.526 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:04.523+0000 7f9435759700 1 -- 172.21.15.160:0/1279993470 learned_addr learned my addr 172.21.15.160:0/1279993470 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:31:04.526 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:04.524+0000 7f9435759700 1 -- 172.21.15.160:0/1279993470 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9420005040 con 0x7f94300f5410 2023-11-17T05:31:04.526 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:04.524+0000 7f9435759700 1 --2- 172.21.15.160:0/1279993470 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94300f5410 0x7f9430105630 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f9420013510 tx=0x7f942000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:31:04.527 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:04.524+0000 7f9425ffb700 1 -- 172.21.15.160:0/1279993470 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9420009320 con 0x7f94300f5410 2023-11-17T05:31:04.527 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:04.524+0000 7f94379bd700 1 -- 172.21.15.160:0/1279993470 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9430105d20 con 0x7f94300f5410 2023-11-17T05:31:04.527 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:04.524+0000 7f9425ffb700 1 -- 172.21.15.160:0/1279993470 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9420013f00 con 0x7f94300f5410 2023-11-17T05:31:04.527 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:04.524+0000 7f9425ffb700 1 -- 172.21.15.160:0/1279993470 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9420014d90 con 0x7f94300f5410 2023-11-17T05:31:04.527 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:04.524+0000 7f94379bd700 1 -- 172.21.15.160:0/1279993470 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94301061b0 con 0x7f94300f5410 2023-11-17T05:31:04.527 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:04.525+0000 7f9425ffb700 1 -- 172.21.15.160:0/1279993470 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9420014420 con 0x7f94300f5410 2023-11-17T05:31:04.527 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:04.525+0000 7f94379bd700 1 -- 172.21.15.160:0/1279993470 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9430045ad0 con 0x7f94300f5410 2023-11-17T05:31:04.527 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:04.525+0000 7f9425ffb700 1 --2- 172.21.15.160:0/1279993470 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f941c032170 0x7f941c034630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:31:04.527 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:04.525+0000 7f9425ffb700 1 -- 172.21.15.160:0/1279993470 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9420041110 con 0x7f94300f5410 2023-11-17T05:31:04.527 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:04.525+0000 7f9434f58700 1 --2- 172.21.15.160:0/1279993470 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f941c032170 0x7f941c034630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:31:04.528 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:04.526+0000 7f9434f58700 1 --2- 172.21.15.160:0/1279993470 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f941c032170 0x7f941c034630 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f942c0096a0 tx=0x7f942c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:31:04.531 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:04.530+0000 7f9425ffb700 1 -- 172.21.15.160:0/1279993470 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f94200095f0 con 0x7f94300f5410 2023-11-17T05:31:04.738 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:04.737+0000 7f94379bd700 1 -- 172.21.15.160:0/1279993470 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f94300fae60 con 0x7f94300f5410 2023-11-17T05:31:04.738 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:04.738+0000 7f9425ffb700 1 -- 172.21.15.160:0/1279993470 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f942001db70 con 0x7f94300f5410 2023-11-17T05:31:04.739 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:31:04.739 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:31:04.740 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:04.739+0000 7f94379bd700 1 -- 172.21.15.160:0/1279993470 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f941c032170 msgr2=0x7f941c034630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:04.740 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:04.739+0000 7f94379bd700 1 --2- 172.21.15.160:0/1279993470 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f941c032170 0x7f941c034630 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f942c0096a0 tx=0x7f942c006b40).stop 2023-11-17T05:31:04.740 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:04.739+0000 7f94379bd700 1 -- 172.21.15.160:0/1279993470 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94300f5410 msgr2=0x7f9430105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:04.740 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:04.739+0000 7f94379bd700 1 --2- 172.21.15.160:0/1279993470 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94300f5410 0x7f9430105630 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f9420013510 tx=0x7f942000b040).stop 2023-11-17T05:31:04.741 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:04.739+0000 7f94379bd700 1 -- 172.21.15.160:0/1279993470 shutdown_connections 2023-11-17T05:31:04.741 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:04.739+0000 7f94379bd700 1 --2- 172.21.15.160:0/1279993470 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f941c032170 0x7f941c034630 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:04.741 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:04.739+0000 7f94379bd700 1 --2- 172.21.15.160:0/1279993470 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94300f5410 0x7f9430105630 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:04.741 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:04.739+0000 7f94379bd700 1 -- 172.21.15.160:0/1279993470 >> 172.21.15.160:0/1279993470 conn(0x7f94300f0b70 msgr2=0x7f94300f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:31:04.741 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:04.739+0000 7f94379bd700 1 -- 172.21.15.160:0/1279993470 shutdown_connections 2023-11-17T05:31:04.741 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:04.740+0000 7f94379bd700 1 -- 172.21.15.160:0/1279993470 wait complete. 2023-11-17T05:31:04.741 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:31:04.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:04 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:31:03.443235+0000 mgr.smithi003.eddcuc (mgr.14164) 174 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:31:05.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:05 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:31:04.738987+0000 mon.smithi003 (mon.0) 193 : audit [DBG] from='client.? 172.21.15.160:0/1279993470' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:31:06.127 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:31:06.127 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:31:06.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:06 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:31:05.443601+0000 mgr.smithi003.eddcuc (mgr.14164) 175 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:31:07.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:07.374+0000 7f958bfff700 1 -- 172.21.15.160:0/1922377129 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f958c0f51a0 msgr2=0x7f958c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:07.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:07.374+0000 7f958bfff700 1 --2- 172.21.15.160:0/1922377129 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f958c0f51a0 0x7f958c0f55c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f957c009fb0 tx=0x7f957c009b30).stop 2023-11-17T05:31:07.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:07.375+0000 7f958bfff700 1 -- 172.21.15.160:0/1922377129 shutdown_connections 2023-11-17T05:31:07.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:07.375+0000 7f958bfff700 1 --2- 172.21.15.160:0/1922377129 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f958c0f51a0 0x7f958c0f55c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:07.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:07.375+0000 7f958bfff700 1 -- 172.21.15.160:0/1922377129 >> 172.21.15.160:0/1922377129 conn(0x7f958c0f09a0 msgr2=0x7f958c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:31:07.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:07.375+0000 7f958bfff700 1 -- 172.21.15.160:0/1922377129 shutdown_connections 2023-11-17T05:31:07.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:07.375+0000 7f958bfff700 1 -- 172.21.15.160:0/1922377129 wait complete. 2023-11-17T05:31:07.378 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:07.375+0000 7f958bfff700 1 Processor -- start 2023-11-17T05:31:07.378 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:07.375+0000 7f958bfff700 1 -- start start 2023-11-17T05:31:07.378 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:07.375+0000 7f958bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f958c0f51a0 0x7f958c0f2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:31:07.378 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:07.375+0000 7f958bfff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f958c0f13d0 con 0x7f958c0f51a0 2023-11-17T05:31:07.378 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:07.376+0000 7f958affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f958c0f51a0 0x7f958c0f2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:31:07.378 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:07.376+0000 7f958affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f958c0f51a0 0x7f958c0f2f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:41664/0 (socket says 172.21.15.160:41664) 2023-11-17T05:31:07.378 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:07.376+0000 7f958affd700 1 -- 172.21.15.160:0/3064882893 learned_addr learned my addr 172.21.15.160:0/3064882893 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:31:07.378 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:07.376+0000 7f958affd700 1 -- 172.21.15.160:0/3064882893 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f957c005040 con 0x7f958c0f51a0 2023-11-17T05:31:07.379 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:07.376+0000 7f958affd700 1 --2- 172.21.15.160:0/3064882893 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f958c0f51a0 0x7f958c0f2f50 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f957c013510 tx=0x7f957c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:31:07.379 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:07.377+0000 7f95737fe700 1 -- 172.21.15.160:0/3064882893 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f957c009320 con 0x7f958c0f51a0 2023-11-17T05:31:07.379 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:07.377+0000 7f958bfff700 1 -- 172.21.15.160:0/3064882893 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f958c0f15d0 con 0x7f958c0f51a0 2023-11-17T05:31:07.379 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:07.377+0000 7f958bfff700 1 -- 172.21.15.160:0/3064882893 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f958c0f1a60 con 0x7f958c0f51a0 2023-11-17T05:31:07.379 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:07.377+0000 7f95737fe700 1 -- 172.21.15.160:0/3064882893 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f957c013f00 con 0x7f958c0f51a0 2023-11-17T05:31:07.379 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:07.377+0000 7f95737fe700 1 -- 172.21.15.160:0/3064882893 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f957c014cc0 con 0x7f958c0f51a0 2023-11-17T05:31:07.380 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:07.378+0000 7f95737fe700 1 -- 172.21.15.160:0/3064882893 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f957c014420 con 0x7f958c0f51a0 2023-11-17T05:31:07.380 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:07.378+0000 7f958bfff700 1 -- 172.21.15.160:0/3064882893 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f956c004fa0 con 0x7f958c0f51a0 2023-11-17T05:31:07.380 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:07.378+0000 7f95737fe700 1 --2- 172.21.15.160:0/3064882893 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f957402ff60 0x7f9574032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:31:07.380 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:07.378+0000 7f95737fe700 1 -- 172.21.15.160:0/3064882893 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f957c040fc0 con 0x7f958c0f51a0 2023-11-17T05:31:07.380 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:07.378+0000 7f958a7fc700 1 --2- 172.21.15.160:0/3064882893 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f957402ff60 0x7f9574032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:31:07.380 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:07.378+0000 7f958a7fc700 1 --2- 172.21.15.160:0/3064882893 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f957402ff60 0x7f9574032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f95800096a0 tx=0x7f9580006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:31:07.383 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:07.382+0000 7f95737fe700 1 -- 172.21.15.160:0/3064882893 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f957c0095f0 con 0x7f958c0f51a0 2023-11-17T05:31:07.588 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:07.587+0000 7f958bfff700 1 -- 172.21.15.160:0/3064882893 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f956c005670 con 0x7f958c0f51a0 2023-11-17T05:31:07.589 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:07.588+0000 7f95737fe700 1 -- 172.21.15.160:0/3064882893 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f957c010e60 con 0x7f958c0f51a0 2023-11-17T05:31:07.589 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:31:07.589 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:31:07.591 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:07.590+0000 7f958bfff700 1 -- 172.21.15.160:0/3064882893 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f957402ff60 msgr2=0x7f9574032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:07.591 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:07.590+0000 7f958bfff700 1 --2- 172.21.15.160:0/3064882893 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f957402ff60 0x7f9574032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f95800096a0 tx=0x7f9580006b40).stop 2023-11-17T05:31:07.591 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:07.590+0000 7f958bfff700 1 -- 172.21.15.160:0/3064882893 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f958c0f51a0 msgr2=0x7f958c0f2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:07.591 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:07.590+0000 7f958bfff700 1 --2- 172.21.15.160:0/3064882893 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f958c0f51a0 0x7f958c0f2f50 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f957c013510 tx=0x7f957c00b040).stop 2023-11-17T05:31:07.591 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:07.590+0000 7f958bfff700 1 -- 172.21.15.160:0/3064882893 shutdown_connections 2023-11-17T05:31:07.591 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:07.590+0000 7f958bfff700 1 --2- 172.21.15.160:0/3064882893 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f957402ff60 0x7f9574032420 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:07.591 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:07.590+0000 7f958bfff700 1 --2- 172.21.15.160:0/3064882893 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f958c0f51a0 0x7f958c0f2f50 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:07.591 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:07.590+0000 7f958bfff700 1 -- 172.21.15.160:0/3064882893 >> 172.21.15.160:0/3064882893 conn(0x7f958c0f09a0 msgr2=0x7f958c05fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:31:07.591 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:07.590+0000 7f958bfff700 1 -- 172.21.15.160:0/3064882893 shutdown_connections 2023-11-17T05:31:07.591 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:07.590+0000 7f958bfff700 1 -- 172.21.15.160:0/3064882893 wait complete. 2023-11-17T05:31:07.592 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:31:08.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:08 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:31:07.443913+0000 mgr.smithi003.eddcuc (mgr.14164) 176 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:31:08.771 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:08 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:31:07.589366+0000 mon.smithi003 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.160:0/3064882893' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:31:08.991 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:31:08.991 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:31:10.139 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:10.136+0000 7f4c1ce41700 1 -- 172.21.15.160:0/2257817131 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c180f6510 msgr2=0x7f4c180f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:10.139 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:10.136+0000 7f4c1ce41700 1 --2- 172.21.15.160:0/2257817131 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c180f6510 0x7f4c180f6930 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f4c08009fb0 tx=0x7f4c08009b30).stop 2023-11-17T05:31:10.139 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:10.136+0000 7f4c1ce41700 1 -- 172.21.15.160:0/2257817131 shutdown_connections 2023-11-17T05:31:10.139 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:10.136+0000 7f4c1ce41700 1 --2- 172.21.15.160:0/2257817131 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c180f6510 0x7f4c180f6930 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:10.139 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:10.136+0000 7f4c1ce41700 1 -- 172.21.15.160:0/2257817131 >> 172.21.15.160:0/2257817131 conn(0x7f4c180f1c70 msgr2=0x7f4c180f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:31:10.140 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:10.136+0000 7f4c1ce41700 1 -- 172.21.15.160:0/2257817131 shutdown_connections 2023-11-17T05:31:10.140 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:10.137+0000 7f4c1ce41700 1 -- 172.21.15.160:0/2257817131 wait complete. 2023-11-17T05:31:10.140 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:10.137+0000 7f4c1ce41700 1 Processor -- start 2023-11-17T05:31:10.140 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:10.137+0000 7f4c1ce41700 1 -- start start 2023-11-17T05:31:10.140 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:10.137+0000 7f4c1ce41700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c180f6510 0x7f4c18106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:31:10.140 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:10.137+0000 7f4c1ce41700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c18106c20 con 0x7f4c180f6510 2023-11-17T05:31:10.140 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:10.137+0000 7f4c1659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c180f6510 0x7f4c18106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:31:10.140 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:10.137+0000 7f4c1659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c180f6510 0x7f4c18106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:41692/0 (socket says 172.21.15.160:41692) 2023-11-17T05:31:10.140 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:10.137+0000 7f4c1659c700 1 -- 172.21.15.160:0/843189515 learned_addr learned my addr 172.21.15.160:0/843189515 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:31:10.140 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:10.138+0000 7f4c1659c700 1 -- 172.21.15.160:0/843189515 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c08005040 con 0x7f4c180f6510 2023-11-17T05:31:10.140 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:10.138+0000 7f4c1659c700 1 --2- 172.21.15.160:0/843189515 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c180f6510 0x7f4c18106730 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f4c08013510 tx=0x7f4c0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:31:10.140 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:10.138+0000 7f4c0effd700 1 -- 172.21.15.160:0/843189515 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c08009320 con 0x7f4c180f6510 2023-11-17T05:31:10.140 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:10.138+0000 7f4c1ce41700 1 -- 172.21.15.160:0/843189515 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c18106e20 con 0x7f4c180f6510 2023-11-17T05:31:10.141 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:10.138+0000 7f4c0effd700 1 -- 172.21.15.160:0/843189515 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4c08013f00 con 0x7f4c180f6510 2023-11-17T05:31:10.141 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:10.138+0000 7f4c1ce41700 1 -- 172.21.15.160:0/843189515 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c181072b0 con 0x7f4c180f6510 2023-11-17T05:31:10.141 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:10.138+0000 7f4c0effd700 1 -- 172.21.15.160:0/843189515 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c08014d90 con 0x7f4c180f6510 2023-11-17T05:31:10.141 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:10.139+0000 7f4c0effd700 1 -- 172.21.15.160:0/843189515 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4c08014420 con 0x7f4c180f6510 2023-11-17T05:31:10.142 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:10.139+0000 7f4c1ce41700 1 -- 172.21.15.160:0/843189515 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c18045ad0 con 0x7f4c180f6510 2023-11-17T05:31:10.142 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:10.139+0000 7f4c0effd700 1 --2- 172.21.15.160:0/843189515 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f4c0402ff60 0x7f4c04032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:31:10.142 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:10.139+0000 7f4c0effd700 1 -- 172.21.15.160:0/843189515 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4c08041040 con 0x7f4c180f6510 2023-11-17T05:31:10.142 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:10.139+0000 7f4c15d9b700 1 --2- 172.21.15.160:0/843189515 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f4c0402ff60 0x7f4c04032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:31:10.142 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:10.140+0000 7f4c15d9b700 1 --2- 172.21.15.160:0/843189515 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f4c0402ff60 0x7f4c04032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f4c000096a0 tx=0x7f4c00006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:31:10.144 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:10.144+0000 7f4c0effd700 1 -- 172.21.15.160:0/843189515 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4c080095f0 con 0x7f4c180f6510 2023-11-17T05:31:10.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:09 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:31:09.444150+0000 mgr.smithi003.eddcuc (mgr.14164) 177 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:31:10.351 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:10.351+0000 7f4c1ce41700 1 -- 172.21.15.160:0/843189515 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4c180fbf60 con 0x7f4c180f6510 2023-11-17T05:31:10.352 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:10.351+0000 7f4c0effd700 1 -- 172.21.15.160:0/843189515 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f4c0801db70 con 0x7f4c180f6510 2023-11-17T05:31:10.352 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:31:10.352 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:31:10.354 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:10.353+0000 7f4c1ce41700 1 -- 172.21.15.160:0/843189515 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f4c0402ff60 msgr2=0x7f4c04032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:10.354 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:10.353+0000 7f4c1ce41700 1 --2- 172.21.15.160:0/843189515 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f4c0402ff60 0x7f4c04032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f4c000096a0 tx=0x7f4c00006b40).stop 2023-11-17T05:31:10.354 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:10.353+0000 7f4c1ce41700 1 -- 172.21.15.160:0/843189515 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c180f6510 msgr2=0x7f4c18106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:10.355 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:10.353+0000 7f4c1ce41700 1 --2- 172.21.15.160:0/843189515 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c180f6510 0x7f4c18106730 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f4c08013510 tx=0x7f4c0800b040).stop 2023-11-17T05:31:10.355 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:10.353+0000 7f4c1ce41700 1 -- 172.21.15.160:0/843189515 shutdown_connections 2023-11-17T05:31:10.355 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:10.353+0000 7f4c1ce41700 1 --2- 172.21.15.160:0/843189515 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f4c0402ff60 0x7f4c04032420 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:10.355 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:10.353+0000 7f4c1ce41700 1 --2- 172.21.15.160:0/843189515 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c180f6510 0x7f4c18106730 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:10.355 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:10.353+0000 7f4c1ce41700 1 -- 172.21.15.160:0/843189515 >> 172.21.15.160:0/843189515 conn(0x7f4c180f1c70 msgr2=0x7f4c180f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:31:10.355 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:10.353+0000 7f4c1ce41700 1 -- 172.21.15.160:0/843189515 shutdown_connections 2023-11-17T05:31:10.355 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:10.353+0000 7f4c1ce41700 1 -- 172.21.15.160:0/843189515 wait complete. 2023-11-17T05:31:10.355 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:31:11.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:10 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:31:10.352637+0000 mon.smithi003 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.160:0/843189515' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:31:11.743 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:31:11.743 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:31:12.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:11 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:31:11.444429+0000 mgr.smithi003.eddcuc (mgr.14164) 178 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:31:12.954 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:12.952+0000 7f2365bcc700 1 -- 172.21.15.160:0/6006132 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2360060280 msgr2=0x7f23600f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:12.954 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:12.952+0000 7f2365bcc700 1 --2- 172.21.15.160:0/6006132 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2360060280 0x7f23600f90f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f2350009fb0 tx=0x7f2350009b30).stop 2023-11-17T05:31:12.954 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:12.952+0000 7f2365bcc700 1 -- 172.21.15.160:0/6006132 shutdown_connections 2023-11-17T05:31:12.955 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:12.952+0000 7f2365bcc700 1 --2- 172.21.15.160:0/6006132 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2360060280 0x7f23600f90f0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:12.956 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:12.952+0000 7f2365bcc700 1 -- 172.21.15.160:0/6006132 >> 172.21.15.160:0/6006132 conn(0x7f23600f0b70 msgr2=0x7f23600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:31:12.956 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:12.953+0000 7f2365bcc700 1 -- 172.21.15.160:0/6006132 shutdown_connections 2023-11-17T05:31:12.956 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:12.953+0000 7f2365bcc700 1 -- 172.21.15.160:0/6006132 wait complete. 2023-11-17T05:31:12.956 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:12.953+0000 7f2365bcc700 1 Processor -- start 2023-11-17T05:31:12.956 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:12.953+0000 7f2365bcc700 1 -- start start 2023-11-17T05:31:12.956 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:12.953+0000 7f2365bcc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2360060280 0x7f23600f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:31:12.956 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:12.953+0000 7f2365bcc700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23600f4ff0 con 0x7f2360060280 2023-11-17T05:31:12.958 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:12.954+0000 7f235f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2360060280 0x7f23600f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:31:12.958 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:12.954+0000 7f235f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2360060280 0x7f23600f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:59450/0 (socket says 172.21.15.160:59450) 2023-11-17T05:31:12.958 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:12.954+0000 7f235f7fe700 1 -- 172.21.15.160:0/1179828757 learned_addr learned my addr 172.21.15.160:0/1179828757 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:31:12.958 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:12.954+0000 7f235f7fe700 1 -- 172.21.15.160:0/1179828757 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2350005040 con 0x7f2360060280 2023-11-17T05:31:12.958 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:12.954+0000 7f235f7fe700 1 --2- 172.21.15.160:0/1179828757 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2360060280 0x7f23600f6b70 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f2350013510 tx=0x7f235000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:31:12.958 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:12.954+0000 7f2347fff700 1 -- 172.21.15.160:0/1179828757 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2350009320 con 0x7f2360060280 2023-11-17T05:31:12.958 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:12.954+0000 7f2365bcc700 1 -- 172.21.15.160:0/1179828757 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23600f5190 con 0x7f2360060280 2023-11-17T05:31:12.959 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:12.954+0000 7f2365bcc700 1 -- 172.21.15.160:0/1179828757 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23600f5620 con 0x7f2360060280 2023-11-17T05:31:12.959 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:12.955+0000 7f2347fff700 1 -- 172.21.15.160:0/1179828757 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2350013f00 con 0x7f2360060280 2023-11-17T05:31:12.959 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:12.955+0000 7f2347fff700 1 -- 172.21.15.160:0/1179828757 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2350014d30 con 0x7f2360060280 2023-11-17T05:31:12.959 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:12.955+0000 7f2365bcc700 1 -- 172.21.15.160:0/1179828757 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2340004fa0 con 0x7f2360060280 2023-11-17T05:31:12.959 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:12.955+0000 7f2347fff700 1 -- 172.21.15.160:0/1179828757 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2350014420 con 0x7f2360060280 2023-11-17T05:31:12.959 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:12.956+0000 7f2347fff700 1 --2- 172.21.15.160:0/1179828757 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f234802ff60 0x7f2348032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:31:12.959 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:12.956+0000 7f2347fff700 1 -- 172.21.15.160:0/1179828757 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2350041290 con 0x7f2360060280 2023-11-17T05:31:12.959 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:12.956+0000 7f235effd700 1 --2- 172.21.15.160:0/1179828757 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f234802ff60 0x7f2348032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:31:12.959 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:12.956+0000 7f235effd700 1 --2- 172.21.15.160:0/1179828757 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f234802ff60 0x7f2348032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f23540096a0 tx=0x7f2354006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:31:12.961 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:12.960+0000 7f2347fff700 1 -- 172.21.15.160:0/1179828757 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f235001c320 con 0x7f2360060280 2023-11-17T05:31:13.170 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:13.169+0000 7f2365bcc700 1 -- 172.21.15.160:0/1179828757 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2340005670 con 0x7f2360060280 2023-11-17T05:31:13.261 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:13.170+0000 7f2347fff700 1 -- 172.21.15.160:0/1179828757 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f235000fe60 con 0x7f2360060280 2023-11-17T05:31:13.262 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:31:13.262 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:31:13.262 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:13.171+0000 7f2365bcc700 1 -- 172.21.15.160:0/1179828757 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f234802ff60 msgr2=0x7f2348032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:13.262 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:13.171+0000 7f2365bcc700 1 --2- 172.21.15.160:0/1179828757 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f234802ff60 0x7f2348032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f23540096a0 tx=0x7f2354006b40).stop 2023-11-17T05:31:13.263 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:13.171+0000 7f2365bcc700 1 -- 172.21.15.160:0/1179828757 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2360060280 msgr2=0x7f23600f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:13.263 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:13.171+0000 7f2365bcc700 1 --2- 172.21.15.160:0/1179828757 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2360060280 0x7f23600f6b70 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f2350013510 tx=0x7f235000b040).stop 2023-11-17T05:31:13.263 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:13.171+0000 7f2365bcc700 1 -- 172.21.15.160:0/1179828757 shutdown_connections 2023-11-17T05:31:13.263 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:13.171+0000 7f2365bcc700 1 --2- 172.21.15.160:0/1179828757 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f234802ff60 0x7f2348032420 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:13.263 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:13.171+0000 7f2365bcc700 1 --2- 172.21.15.160:0/1179828757 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2360060280 0x7f23600f6b70 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:13.263 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:13.171+0000 7f2365bcc700 1 -- 172.21.15.160:0/1179828757 >> 172.21.15.160:0/1179828757 conn(0x7f23600f0b70 msgr2=0x7f23600f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:31:13.263 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:13.172+0000 7f2365bcc700 1 -- 172.21.15.160:0/1179828757 shutdown_connections 2023-11-17T05:31:13.263 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:13.172+0000 7f2365bcc700 1 -- 172.21.15.160:0/1179828757 wait complete. 2023-11-17T05:31:13.263 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:31:13.520 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:13 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:31:13.171105+0000 mon.smithi003 (mon.0) 196 : audit [DBG] from='client.? 172.21.15.160:0/1179828757' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:31:14.520 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:14 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:31:13.444713+0000 mgr.smithi003.eddcuc (mgr.14164) 179 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:31:14.598 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:31:14.598 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:31:15.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:15.752+0000 7fbf7f4f2700 1 -- 172.21.15.160:0/1823543774 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf780f51a0 msgr2=0x7fbf780f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:15.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:15.752+0000 7fbf7f4f2700 1 --2- 172.21.15.160:0/1823543774 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf780f51a0 0x7fbf780f55c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fbf74009fb0 tx=0x7fbf74009b30).stop 2023-11-17T05:31:15.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:15.752+0000 7fbf7f4f2700 1 -- 172.21.15.160:0/1823543774 shutdown_connections 2023-11-17T05:31:15.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:15.752+0000 7fbf7f4f2700 1 --2- 172.21.15.160:0/1823543774 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf780f51a0 0x7fbf780f55c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:15.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:15.752+0000 7fbf7f4f2700 1 -- 172.21.15.160:0/1823543774 >> 172.21.15.160:0/1823543774 conn(0x7fbf780f09a0 msgr2=0x7fbf780f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:31:15.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:15.752+0000 7fbf7f4f2700 1 -- 172.21.15.160:0/1823543774 shutdown_connections 2023-11-17T05:31:15.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:15.752+0000 7fbf7f4f2700 1 -- 172.21.15.160:0/1823543774 wait complete. 2023-11-17T05:31:15.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:15.752+0000 7fbf7f4f2700 1 Processor -- start 2023-11-17T05:31:15.755 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:15.753+0000 7fbf7f4f2700 1 -- start start 2023-11-17T05:31:15.755 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:15.753+0000 7fbf7f4f2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf780f51a0 0x7fbf781091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:31:15.755 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:15.753+0000 7fbf7f4f2700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf78060240 con 0x7fbf780f51a0 2023-11-17T05:31:15.755 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:15.753+0000 7fbf7e4f0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf780f51a0 0x7fbf781091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:31:15.755 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:15.753+0000 7fbf7e4f0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf780f51a0 0x7fbf781091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:59478/0 (socket says 172.21.15.160:59478) 2023-11-17T05:31:15.755 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:15.753+0000 7fbf7e4f0700 1 -- 172.21.15.160:0/1521721047 learned_addr learned my addr 172.21.15.160:0/1521721047 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:31:15.756 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:15.753+0000 7fbf7e4f0700 1 -- 172.21.15.160:0/1521721047 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf74005040 con 0x7fbf780f51a0 2023-11-17T05:31:15.756 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:15.753+0000 7fbf7e4f0700 1 --2- 172.21.15.160:0/1521721047 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf780f51a0 0x7fbf781091b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fbf74009830 tx=0x7fbf7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:31:15.756 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:15.754+0000 7fbf6affd700 1 -- 172.21.15.160:0/1521721047 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf74009320 con 0x7fbf780f51a0 2023-11-17T05:31:15.757 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:15.754+0000 7fbf7f4f2700 1 -- 172.21.15.160:0/1521721047 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf78060440 con 0x7fbf780f51a0 2023-11-17T05:31:15.757 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:15.754+0000 7fbf6affd700 1 -- 172.21.15.160:0/1521721047 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbf740136c0 con 0x7fbf780f51a0 2023-11-17T05:31:15.757 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:15.754+0000 7fbf7f4f2700 1 -- 172.21.15.160:0/1521721047 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf78109860 con 0x7fbf780f51a0 2023-11-17T05:31:15.757 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:15.754+0000 7fbf6affd700 1 -- 172.21.15.160:0/1521721047 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf7401ee90 con 0x7fbf780f51a0 2023-11-17T05:31:15.757 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:15.754+0000 7fbf6affd700 1 -- 172.21.15.160:0/1521721047 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbf74014480 con 0x7fbf780f51a0 2023-11-17T05:31:15.757 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:15.755+0000 7fbf6affd700 1 --2- 172.21.15.160:0/1521721047 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fbf6402ff10 0x7fbf640323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:31:15.757 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:15.755+0000 7fbf6affd700 1 -- 172.21.15.160:0/1521721047 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbf74040fd0 con 0x7fbf780f51a0 2023-11-17T05:31:15.757 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:15.755+0000 7fbf7f4f2700 1 -- 172.21.15.160:0/1521721047 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf78045ad0 con 0x7fbf780f51a0 2023-11-17T05:31:15.757 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:15.755+0000 7fbf7dcef700 1 --2- 172.21.15.160:0/1521721047 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fbf6402ff10 0x7fbf640323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:31:15.757 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:15.755+0000 7fbf7dcef700 1 --2- 172.21.15.160:0/1521721047 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fbf6402ff10 0x7fbf640323d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fbf6c0096a0 tx=0x7fbf6c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:31:15.760 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:15.760+0000 7fbf6affd700 1 -- 172.21.15.160:0/1521721047 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbf7401c910 con 0x7fbf780f51a0 2023-11-17T05:31:15.972 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:15.971+0000 7fbf7f4f2700 1 -- 172.21.15.160:0/1521721047 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbf78109f60 con 0x7fbf780f51a0 2023-11-17T05:31:15.972 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:15.971+0000 7fbf6affd700 1 -- 172.21.15.160:0/1521721047 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fbf7401c320 con 0x7fbf780f51a0 2023-11-17T05:31:15.972 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:31:15.973 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:31:15.974 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:15.973+0000 7fbf7f4f2700 1 -- 172.21.15.160:0/1521721047 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fbf6402ff10 msgr2=0x7fbf640323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:15.974 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:15.973+0000 7fbf7f4f2700 1 --2- 172.21.15.160:0/1521721047 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fbf6402ff10 0x7fbf640323d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fbf6c0096a0 tx=0x7fbf6c006b40).stop 2023-11-17T05:31:15.974 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:15.973+0000 7fbf7f4f2700 1 -- 172.21.15.160:0/1521721047 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf780f51a0 msgr2=0x7fbf781091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:15.975 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:15.973+0000 7fbf7f4f2700 1 --2- 172.21.15.160:0/1521721047 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf780f51a0 0x7fbf781091b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fbf74009830 tx=0x7fbf7400b040).stop 2023-11-17T05:31:15.975 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:15.973+0000 7fbf7f4f2700 1 -- 172.21.15.160:0/1521721047 shutdown_connections 2023-11-17T05:31:15.975 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:15.973+0000 7fbf7f4f2700 1 --2- 172.21.15.160:0/1521721047 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fbf6402ff10 0x7fbf640323d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:15.975 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:15.973+0000 7fbf7f4f2700 1 --2- 172.21.15.160:0/1521721047 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf780f51a0 0x7fbf781091b0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:15.975 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:15.973+0000 7fbf7f4f2700 1 -- 172.21.15.160:0/1521721047 >> 172.21.15.160:0/1521721047 conn(0x7fbf780f09a0 msgr2=0x7fbf780f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:31:15.975 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:15.973+0000 7fbf7f4f2700 1 -- 172.21.15.160:0/1521721047 shutdown_connections 2023-11-17T05:31:15.975 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:15.973+0000 7fbf7f4f2700 1 -- 172.21.15.160:0/1521721047 wait complete. 2023-11-17T05:31:15.975 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:31:16.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:16 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:31:15.445047+0000 mgr.smithi003.eddcuc (mgr.14164) 180 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:31:16.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:16 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:31:15.972608+0000 mon.smithi003 (mon.0) 197 : audit [DBG] from='client.? 172.21.15.160:0/1521721047' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:31:17.350 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:31:17.350 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:31:18.569 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:18.567+0000 7f8782691700 1 -- 172.21.15.160:0/1077213039 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f877c060220 msgr2=0x7f877c0f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:18.569 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:18.567+0000 7f8782691700 1 --2- 172.21.15.160:0/1077213039 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f877c060220 0x7f877c0f9150 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f876c009fb0 tx=0x7f876c009b30).stop 2023-11-17T05:31:18.569 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:18.568+0000 7f8782691700 1 -- 172.21.15.160:0/1077213039 shutdown_connections 2023-11-17T05:31:18.570 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:18.568+0000 7f8782691700 1 --2- 172.21.15.160:0/1077213039 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f877c060220 0x7f877c0f9150 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:18.570 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:18.568+0000 7f8782691700 1 -- 172.21.15.160:0/1077213039 >> 172.21.15.160:0/1077213039 conn(0x7f877c0f0b70 msgr2=0x7f877c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:31:18.570 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:18.568+0000 7f8782691700 1 -- 172.21.15.160:0/1077213039 shutdown_connections 2023-11-17T05:31:18.570 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:18.568+0000 7f8782691700 1 -- 172.21.15.160:0/1077213039 wait complete. 2023-11-17T05:31:18.570 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:18.568+0000 7f8782691700 1 Processor -- start 2023-11-17T05:31:18.570 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:18.568+0000 7f8782691700 1 -- start start 2023-11-17T05:31:18.570 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:18.568+0000 7f8782691700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f877c060220 0x7f877c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:31:18.570 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:18.568+0000 7f8782691700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f877c106bd0 con 0x7f877c060220 2023-11-17T05:31:18.571 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:18.569+0000 7f877bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f877c060220 0x7f877c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:31:18.572 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:18.569+0000 7f877bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f877c060220 0x7f877c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:59486/0 (socket says 172.21.15.160:59486) 2023-11-17T05:31:18.572 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:18.569+0000 7f877bfff700 1 -- 172.21.15.160:0/203210699 learned_addr learned my addr 172.21.15.160:0/203210699 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:31:18.572 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:18.569+0000 7f877bfff700 1 -- 172.21.15.160:0/203210699 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f876c005040 con 0x7f877c060220 2023-11-17T05:31:18.572 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:18.569+0000 7f877bfff700 1 --2- 172.21.15.160:0/203210699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f877c060220 0x7f877c1066e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f876c013510 tx=0x7f876c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:31:18.572 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:18.569+0000 7f8778ff9700 1 -- 172.21.15.160:0/203210699 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f876c009320 con 0x7f877c060220 2023-11-17T05:31:18.572 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:18.570+0000 7f8782691700 1 -- 172.21.15.160:0/203210699 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f877c106dd0 con 0x7f877c060220 2023-11-17T05:31:18.572 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:18.570+0000 7f8778ff9700 1 -- 172.21.15.160:0/203210699 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f876c013f00 con 0x7f877c060220 2023-11-17T05:31:18.572 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:18.570+0000 7f8778ff9700 1 -- 172.21.15.160:0/203210699 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f876c014dc0 con 0x7f877c060220 2023-11-17T05:31:18.572 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:18.570+0000 7f8782691700 1 -- 172.21.15.160:0/203210699 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f877c1071e0 con 0x7f877c060220 2023-11-17T05:31:18.572 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:18.570+0000 7f8778ff9700 1 -- 172.21.15.160:0/203210699 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f876c0095f0 con 0x7f877c060220 2023-11-17T05:31:18.572 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:18.570+0000 7f8782691700 1 -- 172.21.15.160:0/203210699 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f877c045ad0 con 0x7f877c060220 2023-11-17T05:31:18.572 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:18.571+0000 7f8778ff9700 1 --2- 172.21.15.160:0/203210699 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f876402ff60 0x7f8764032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:31:18.572 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:18.571+0000 7f8778ff9700 1 -- 172.21.15.160:0/203210699 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f876c0410c0 con 0x7f877c060220 2023-11-17T05:31:18.573 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:18.571+0000 7f877b7fe700 1 --2- 172.21.15.160:0/203210699 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f876402ff60 0x7f8764032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:31:18.574 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:18.571+0000 7f877b7fe700 1 --2- 172.21.15.160:0/203210699 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f876402ff60 0x7f8764032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f87700096a0 tx=0x7f8770006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:31:18.579 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:18.577+0000 7f8778ff9700 1 -- 172.21.15.160:0/203210699 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f876c014420 con 0x7f877c060220 2023-11-17T05:31:18.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:18 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:31:17.445424+0000 mgr.smithi003.eddcuc (mgr.14164) 181 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:31:18.781 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:18.780+0000 7f8782691700 1 -- 172.21.15.160:0/203210699 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f877c1080a0 con 0x7f877c060220 2023-11-17T05:31:18.782 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:18.781+0000 7f8778ff9700 1 -- 172.21.15.160:0/203210699 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f876c010230 con 0x7f877c060220 2023-11-17T05:31:18.782 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:31:18.782 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:31:18.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:18.783+0000 7f8782691700 1 -- 172.21.15.160:0/203210699 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f876402ff60 msgr2=0x7f8764032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:18.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:18.783+0000 7f8782691700 1 --2- 172.21.15.160:0/203210699 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f876402ff60 0x7f8764032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f87700096a0 tx=0x7f8770006b40).stop 2023-11-17T05:31:18.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:18.783+0000 7f8782691700 1 -- 172.21.15.160:0/203210699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f877c060220 msgr2=0x7f877c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:18.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:18.783+0000 7f8782691700 1 --2- 172.21.15.160:0/203210699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f877c060220 0x7f877c1066e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f876c013510 tx=0x7f876c00b040).stop 2023-11-17T05:31:18.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:18.783+0000 7f8782691700 1 -- 172.21.15.160:0/203210699 shutdown_connections 2023-11-17T05:31:18.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:18.783+0000 7f8782691700 1 --2- 172.21.15.160:0/203210699 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f876402ff60 0x7f8764032420 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:18.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:18.783+0000 7f8782691700 1 --2- 172.21.15.160:0/203210699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f877c060220 0x7f877c1066e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:18.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:18.783+0000 7f8782691700 1 -- 172.21.15.160:0/203210699 >> 172.21.15.160:0/203210699 conn(0x7f877c0f0b70 msgr2=0x7f877c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:31:18.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:18.783+0000 7f8782691700 1 -- 172.21.15.160:0/203210699 shutdown_connections 2023-11-17T05:31:18.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:18.783+0000 7f8782691700 1 -- 172.21.15.160:0/203210699 wait complete. 2023-11-17T05:31:18.785 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:31:19.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:19 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:31:18.782378+0000 mon.smithi003 (mon.0) 198 : audit [DBG] from='client.? 172.21.15.160:0/203210699' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:31:20.198 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:31:20.199 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:31:20.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:20 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:31:19.445838+0000 mgr.smithi003.eddcuc (mgr.14164) 182 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:31:21.369 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:21.368+0000 7f0086f47700 1 -- 172.21.15.160:0/136899891 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f00800f6a80 msgr2=0x7f00800f6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:21.369 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:21.368+0000 7f0086f47700 1 --2- 172.21.15.160:0/136899891 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f00800f6a80 0x7f00800f6ea0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f0074009fb0 tx=0x7f0074009b30).stop 2023-11-17T05:31:21.369 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:21.368+0000 7f0086f47700 1 -- 172.21.15.160:0/136899891 shutdown_connections 2023-11-17T05:31:21.369 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:21.368+0000 7f0086f47700 1 --2- 172.21.15.160:0/136899891 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f00800f6a80 0x7f00800f6ea0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:21.369 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:21.368+0000 7f0086f47700 1 -- 172.21.15.160:0/136899891 >> 172.21.15.160:0/136899891 conn(0x7f00800f02e0 msgr2=0x7f00800f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:31:21.370 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:21.368+0000 7f0086f47700 1 -- 172.21.15.160:0/136899891 shutdown_connections 2023-11-17T05:31:21.370 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:21.368+0000 7f0086f47700 1 -- 172.21.15.160:0/136899891 wait complete. 2023-11-17T05:31:21.370 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:21.369+0000 7f0086f47700 1 Processor -- start 2023-11-17T05:31:21.370 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:21.369+0000 7f0086f47700 1 -- start start 2023-11-17T05:31:21.370 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:21.369+0000 7f0086f47700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f00800f6a80 0x7f008010b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:31:21.370 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:21.369+0000 7f0086f47700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0080107220 con 0x7f00800f6a80 2023-11-17T05:31:21.372 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:21.370+0000 7f0084ce3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f00800f6a80 0x7f008010b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:31:21.372 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:21.370+0000 7f0084ce3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f00800f6a80 0x7f008010b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:40876/0 (socket says 172.21.15.160:40876) 2023-11-17T05:31:21.372 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:21.370+0000 7f0084ce3700 1 -- 172.21.15.160:0/1484913452 learned_addr learned my addr 172.21.15.160:0/1484913452 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:31:21.372 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:21.370+0000 7f0084ce3700 1 -- 172.21.15.160:0/1484913452 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0074005040 con 0x7f00800f6a80 2023-11-17T05:31:21.372 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:21.370+0000 7f0084ce3700 1 --2- 172.21.15.160:0/1484913452 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f00800f6a80 0x7f008010b830 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f0074013510 tx=0x7f007400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:31:21.372 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:21.370+0000 7f007d7fa700 1 -- 172.21.15.160:0/1484913452 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0074009320 con 0x7f00800f6a80 2023-11-17T05:31:21.372 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:21.370+0000 7f0086f47700 1 -- 172.21.15.160:0/1484913452 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00800f5310 con 0x7f00800f6a80 2023-11-17T05:31:21.372 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:21.370+0000 7f007d7fa700 1 -- 172.21.15.160:0/1484913452 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0074013f00 con 0x7f00800f6a80 2023-11-17T05:31:21.372 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:21.370+0000 7f0086f47700 1 -- 172.21.15.160:0/1484913452 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f008010bd20 con 0x7f00800f6a80 2023-11-17T05:31:21.372 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:21.370+0000 7f007d7fa700 1 -- 172.21.15.160:0/1484913452 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0074014de0 con 0x7f00800f6a80 2023-11-17T05:31:21.372 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:21.371+0000 7f007d7fa700 1 -- 172.21.15.160:0/1484913452 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f007401e420 con 0x7f00800f6a80 2023-11-17T05:31:21.373 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:21.371+0000 7f0086f47700 1 -- 172.21.15.160:0/1484913452 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0080045ad0 con 0x7f00800f6a80 2023-11-17T05:31:21.373 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:21.371+0000 7f007d7fa700 1 --2- 172.21.15.160:0/1484913452 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f007002ff60 0x7f0070032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:31:21.373 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:21.371+0000 7f007d7fa700 1 -- 172.21.15.160:0/1484913452 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0074040ff0 con 0x7f00800f6a80 2023-11-17T05:31:21.373 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:21.372+0000 7f007ffff700 1 --2- 172.21.15.160:0/1484913452 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f007002ff60 0x7f0070032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:31:21.373 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:21.372+0000 7f007ffff700 1 --2- 172.21.15.160:0/1484913452 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f007002ff60 0x7f0070032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f006c0096a0 tx=0x7f006c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:31:21.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:21.376+0000 7f007d7fa700 1 -- 172.21.15.160:0/1484913452 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f00740095f0 con 0x7f00800f6a80 2023-11-17T05:31:21.582 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:21.582+0000 7f0086f47700 1 -- 172.21.15.160:0/1484913452 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f008010c4e0 con 0x7f00800f6a80 2023-11-17T05:31:21.582 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:21.583+0000 7f007d7fa700 1 -- 172.21.15.160:0/1484913452 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f007401d380 con 0x7f00800f6a80 2023-11-17T05:31:21.583 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:31:21.583 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:31:21.584 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:21.584+0000 7f0086f47700 1 -- 172.21.15.160:0/1484913452 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f007002ff60 msgr2=0x7f0070032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:21.584 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:21.584+0000 7f0086f47700 1 --2- 172.21.15.160:0/1484913452 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f007002ff60 0x7f0070032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f006c0096a0 tx=0x7f006c006b40).stop 2023-11-17T05:31:21.584 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:21.584+0000 7f0086f47700 1 -- 172.21.15.160:0/1484913452 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f00800f6a80 msgr2=0x7f008010b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:21.584 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:21.584+0000 7f0086f47700 1 --2- 172.21.15.160:0/1484913452 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f00800f6a80 0x7f008010b830 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f0074013510 tx=0x7f007400b040).stop 2023-11-17T05:31:21.584 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:21.584+0000 7f0086f47700 1 -- 172.21.15.160:0/1484913452 shutdown_connections 2023-11-17T05:31:21.584 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:21.584+0000 7f0086f47700 1 --2- 172.21.15.160:0/1484913452 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f007002ff60 0x7f0070032420 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:21.584 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:21.584+0000 7f0086f47700 1 --2- 172.21.15.160:0/1484913452 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f00800f6a80 0x7f008010b830 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:21.584 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:21.584+0000 7f0086f47700 1 -- 172.21.15.160:0/1484913452 >> 172.21.15.160:0/1484913452 conn(0x7f00800f02e0 msgr2=0x7f00800f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:31:21.585 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:21.584+0000 7f0086f47700 1 -- 172.21.15.160:0/1484913452 shutdown_connections 2023-11-17T05:31:21.585 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:21.584+0000 7f0086f47700 1 -- 172.21.15.160:0/1484913452 wait complete. 2023-11-17T05:31:21.585 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:31:22.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:22 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:31:21.446197+0000 mgr.smithi003.eddcuc (mgr.14164) 183 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:31:22.771 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:22 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:31:21.582814+0000 mon.smithi003 (mon.0) 199 : audit [DBG] from='client.? 172.21.15.160:0/1484913452' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:31:22.988 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:31:22.988 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:31:24.152 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:24.149+0000 7fa225d73700 1 -- 172.21.15.160:0/577168575 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2200601e0 msgr2=0x7fa2200f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:24.152 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:24.149+0000 7fa225d73700 1 --2- 172.21.15.160:0/577168575 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2200601e0 0x7fa2200f8f20 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7fa214009fb0 tx=0x7fa214009b30).stop 2023-11-17T05:31:24.152 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:24.149+0000 7fa225d73700 1 -- 172.21.15.160:0/577168575 shutdown_connections 2023-11-17T05:31:24.152 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:24.149+0000 7fa225d73700 1 --2- 172.21.15.160:0/577168575 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2200601e0 0x7fa2200f8f20 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:24.152 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:24.149+0000 7fa225d73700 1 -- 172.21.15.160:0/577168575 >> 172.21.15.160:0/577168575 conn(0x7fa2200f09a0 msgr2=0x7fa2200f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:31:24.152 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:24.149+0000 7fa225d73700 1 -- 172.21.15.160:0/577168575 shutdown_connections 2023-11-17T05:31:24.152 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:24.149+0000 7fa225d73700 1 -- 172.21.15.160:0/577168575 wait complete. 2023-11-17T05:31:24.152 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:24.150+0000 7fa225d73700 1 Processor -- start 2023-11-17T05:31:24.152 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:24.150+0000 7fa225d73700 1 -- start start 2023-11-17T05:31:24.152 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:24.150+0000 7fa225d73700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2200601e0 0x7fa220106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:31:24.153 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:24.150+0000 7fa225d73700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2201069f0 con 0x7fa2200601e0 2023-11-17T05:31:24.153 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:24.150+0000 7fa224d71700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2200601e0 0x7fa220106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:31:24.153 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:24.150+0000 7fa224d71700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2200601e0 0x7fa220106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:40904/0 (socket says 172.21.15.160:40904) 2023-11-17T05:31:24.153 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:24.150+0000 7fa224d71700 1 -- 172.21.15.160:0/3307320822 learned_addr learned my addr 172.21.15.160:0/3307320822 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:31:24.153 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:24.150+0000 7fa224d71700 1 -- 172.21.15.160:0/3307320822 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa214005040 con 0x7fa2200601e0 2023-11-17T05:31:24.153 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:24.150+0000 7fa224d71700 1 --2- 172.21.15.160:0/3307320822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2200601e0 0x7fa220106500 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fa214013510 tx=0x7fa21400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:31:24.153 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:24.151+0000 7fa21d7fa700 1 -- 172.21.15.160:0/3307320822 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa214009320 con 0x7fa2200601e0 2023-11-17T05:31:24.153 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:24.151+0000 7fa225d73700 1 -- 172.21.15.160:0/3307320822 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa220106bf0 con 0x7fa2200601e0 2023-11-17T05:31:24.153 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:24.151+0000 7fa21d7fa700 1 -- 172.21.15.160:0/3307320822 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa214013f00 con 0x7fa2200601e0 2023-11-17T05:31:24.153 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:24.151+0000 7fa21d7fa700 1 -- 172.21.15.160:0/3307320822 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa214014d90 con 0x7fa2200601e0 2023-11-17T05:31:24.153 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:24.151+0000 7fa225d73700 1 -- 172.21.15.160:0/3307320822 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa220107080 con 0x7fa2200601e0 2023-11-17T05:31:24.154 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:24.151+0000 7fa21d7fa700 1 -- 172.21.15.160:0/3307320822 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa2140095f0 con 0x7fa2200601e0 2023-11-17T05:31:24.155 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:24.152+0000 7fa225d73700 1 -- 172.21.15.160:0/3307320822 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa220045ad0 con 0x7fa2200601e0 2023-11-17T05:31:24.155 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:24.152+0000 7fa21d7fa700 1 --2- 172.21.15.160:0/3307320822 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa21002ff60 0x7fa210032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:31:24.157 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:24.152+0000 7fa21d7fa700 1 -- 172.21.15.160:0/3307320822 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa2140410c0 con 0x7fa2200601e0 2023-11-17T05:31:24.158 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:24.152+0000 7fa21ffff700 1 --2- 172.21.15.160:0/3307320822 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa21002ff60 0x7fa210032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:31:24.158 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:24.152+0000 7fa21ffff700 1 --2- 172.21.15.160:0/3307320822 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa21002ff60 0x7fa210032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fa20c0096a0 tx=0x7fa20c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:31:24.158 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:24.157+0000 7fa21d7fa700 1 -- 172.21.15.160:0/3307320822 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa214014420 con 0x7fa2200601e0 2023-11-17T05:31:24.363 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:24.362+0000 7fa225d73700 1 -- 172.21.15.160:0/3307320822 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa220107f30 con 0x7fa2200601e0 2023-11-17T05:31:24.364 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:24.363+0000 7fa21d7fa700 1 -- 172.21.15.160:0/3307320822 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fa214010230 con 0x7fa2200601e0 2023-11-17T05:31:24.364 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:31:24.364 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:31:24.366 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:24.365+0000 7fa225d73700 1 -- 172.21.15.160:0/3307320822 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa21002ff60 msgr2=0x7fa210032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:24.366 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:24.365+0000 7fa225d73700 1 --2- 172.21.15.160:0/3307320822 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa21002ff60 0x7fa210032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fa20c0096a0 tx=0x7fa20c006b40).stop 2023-11-17T05:31:24.366 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:24.365+0000 7fa225d73700 1 -- 172.21.15.160:0/3307320822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2200601e0 msgr2=0x7fa220106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:24.366 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:24.365+0000 7fa225d73700 1 --2- 172.21.15.160:0/3307320822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2200601e0 0x7fa220106500 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fa214013510 tx=0x7fa21400b040).stop 2023-11-17T05:31:24.366 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:24.365+0000 7fa225d73700 1 -- 172.21.15.160:0/3307320822 shutdown_connections 2023-11-17T05:31:24.366 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:24.365+0000 7fa225d73700 1 --2- 172.21.15.160:0/3307320822 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa21002ff60 0x7fa210032420 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:24.366 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:24.365+0000 7fa225d73700 1 --2- 172.21.15.160:0/3307320822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2200601e0 0x7fa220106500 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:24.366 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:24.365+0000 7fa225d73700 1 -- 172.21.15.160:0/3307320822 >> 172.21.15.160:0/3307320822 conn(0x7fa2200f09a0 msgr2=0x7fa2200f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:31:24.366 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:24.365+0000 7fa225d73700 1 -- 172.21.15.160:0/3307320822 shutdown_connections 2023-11-17T05:31:24.366 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:24.365+0000 7fa225d73700 1 -- 172.21.15.160:0/3307320822 wait complete. 2023-11-17T05:31:24.367 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:31:24.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:31:23.446594+0000 mgr.smithi003.eddcuc (mgr.14164) 184 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:31:24.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:31:24.364381+0000 mon.smithi003 (mon.0) 200 : audit [DBG] from='client.? 172.21.15.160:0/3307320822' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:31:25.799 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:31:25.800 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:31:26.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:26 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:31:25.446808+0000 mgr.smithi003.eddcuc (mgr.14164) 185 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:31:26.917 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:26.916+0000 7f26faae2700 1 -- 172.21.15.160:0/839608026 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26f4060140 msgr2=0x7f26f40f8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:26.917 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:26.916+0000 7f26faae2700 1 --2- 172.21.15.160:0/839608026 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26f4060140 0x7f26f40f8fc0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f26e4009fb0 tx=0x7f26e4009b30).stop 2023-11-17T05:31:26.917 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:26.916+0000 7f26faae2700 1 -- 172.21.15.160:0/839608026 shutdown_connections 2023-11-17T05:31:26.917 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:26.916+0000 7f26faae2700 1 --2- 172.21.15.160:0/839608026 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26f4060140 0x7f26f40f8fc0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:26.918 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:26.916+0000 7f26faae2700 1 -- 172.21.15.160:0/839608026 >> 172.21.15.160:0/839608026 conn(0x7f26f40f09a0 msgr2=0x7f26f40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:31:26.918 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:26.916+0000 7f26faae2700 1 -- 172.21.15.160:0/839608026 shutdown_connections 2023-11-17T05:31:26.918 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:26.916+0000 7f26faae2700 1 -- 172.21.15.160:0/839608026 wait complete. 2023-11-17T05:31:26.918 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:26.916+0000 7f26faae2700 1 Processor -- start 2023-11-17T05:31:26.918 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:26.916+0000 7f26faae2700 1 -- start start 2023-11-17T05:31:26.918 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:26.916+0000 7f26faae2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26f4060140 0x7f26f4106530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:31:26.918 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:26.916+0000 7f26faae2700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f4106a20 con 0x7f26f4060140 2023-11-17T05:31:26.920 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:26.917+0000 7f26f9ae0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26f4060140 0x7f26f4106530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:31:26.920 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:26.917+0000 7f26f9ae0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26f4060140 0x7f26f4106530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:40918/0 (socket says 172.21.15.160:40918) 2023-11-17T05:31:26.920 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:26.917+0000 7f26f9ae0700 1 -- 172.21.15.160:0/2679330558 learned_addr learned my addr 172.21.15.160:0/2679330558 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:31:26.920 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:26.917+0000 7f26f9ae0700 1 -- 172.21.15.160:0/2679330558 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e4005040 con 0x7f26f4060140 2023-11-17T05:31:26.920 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:26.917+0000 7f26f9ae0700 1 --2- 172.21.15.160:0/2679330558 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26f4060140 0x7f26f4106530 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f26e4013510 tx=0x7f26e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:31:26.921 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:26.917+0000 7f26ea7fc700 1 -- 172.21.15.160:0/2679330558 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26e4009320 con 0x7f26f4060140 2023-11-17T05:31:26.921 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:26.917+0000 7f26faae2700 1 -- 172.21.15.160:0/2679330558 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26f4106c20 con 0x7f26f4060140 2023-11-17T05:31:26.921 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:26.917+0000 7f26ea7fc700 1 -- 172.21.15.160:0/2679330558 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f26e4013f00 con 0x7f26f4060140 2023-11-17T05:31:26.921 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:26.917+0000 7f26ea7fc700 1 -- 172.21.15.160:0/2679330558 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26e4014d90 con 0x7f26f4060140 2023-11-17T05:31:26.921 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:26.917+0000 7f26faae2700 1 -- 172.21.15.160:0/2679330558 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26f41070b0 con 0x7f26f4060140 2023-11-17T05:31:26.921 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:26.918+0000 7f26ea7fc700 1 -- 172.21.15.160:0/2679330558 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f26e40095f0 con 0x7f26f4060140 2023-11-17T05:31:26.921 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:26.918+0000 7f26faae2700 1 -- 172.21.15.160:0/2679330558 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26f4045ad0 con 0x7f26f4060140 2023-11-17T05:31:26.921 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:26.918+0000 7f26ea7fc700 1 --2- 172.21.15.160:0/2679330558 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f26e002ffb0 0x7f26e0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:31:26.921 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:26.918+0000 7f26ea7fc700 1 -- 172.21.15.160:0/2679330558 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f26e4040ff0 con 0x7f26f4060140 2023-11-17T05:31:26.921 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:26.919+0000 7f26f92df700 1 --2- 172.21.15.160:0/2679330558 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f26e002ffb0 0x7f26e0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:31:26.921 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:26.919+0000 7f26f92df700 1 --2- 172.21.15.160:0/2679330558 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f26e002ffb0 0x7f26e0032470 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f26f00096a0 tx=0x7f26f0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:31:26.924 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:26.923+0000 7f26ea7fc700 1 -- 172.21.15.160:0/2679330558 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f26e406d2b0 con 0x7f26f4060140 2023-11-17T05:31:27.132 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:27.131+0000 7f26faae2700 1 -- 172.21.15.160:0/2679330558 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f26f4107f60 con 0x7f26f4060140 2023-11-17T05:31:27.133 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:27.132+0000 7f26ea7fc700 1 -- 172.21.15.160:0/2679330558 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f26e4014420 con 0x7f26f4060140 2023-11-17T05:31:27.133 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:31:27.133 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:31:27.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:27.134+0000 7f26faae2700 1 -- 172.21.15.160:0/2679330558 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f26e002ffb0 msgr2=0x7f26e0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:27.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:27.134+0000 7f26faae2700 1 --2- 172.21.15.160:0/2679330558 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f26e002ffb0 0x7f26e0032470 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f26f00096a0 tx=0x7f26f0006b40).stop 2023-11-17T05:31:27.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:27.134+0000 7f26faae2700 1 -- 172.21.15.160:0/2679330558 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26f4060140 msgr2=0x7f26f4106530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:27.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:27.134+0000 7f26faae2700 1 --2- 172.21.15.160:0/2679330558 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26f4060140 0x7f26f4106530 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f26e4013510 tx=0x7f26e400b040).stop 2023-11-17T05:31:27.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:27.134+0000 7f26faae2700 1 -- 172.21.15.160:0/2679330558 shutdown_connections 2023-11-17T05:31:27.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:27.134+0000 7f26faae2700 1 --2- 172.21.15.160:0/2679330558 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f26e002ffb0 0x7f26e0032470 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:27.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:27.134+0000 7f26faae2700 1 --2- 172.21.15.160:0/2679330558 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26f4060140 0x7f26f4106530 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:27.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:27.134+0000 7f26faae2700 1 -- 172.21.15.160:0/2679330558 >> 172.21.15.160:0/2679330558 conn(0x7f26f40f09a0 msgr2=0x7f26f40f14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:31:27.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:27.134+0000 7f26faae2700 1 -- 172.21.15.160:0/2679330558 shutdown_connections 2023-11-17T05:31:27.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:27.134+0000 7f26faae2700 1 -- 172.21.15.160:0/2679330558 wait complete. 2023-11-17T05:31:27.136 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:31:27.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:27 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:31:27.133139+0000 mon.smithi003 (mon.0) 201 : audit [DBG] from='client.? 172.21.15.160:0/2679330558' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:31:28.506 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:31:28.506 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:31:28.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:31:27.447056+0000 mgr.smithi003.eddcuc (mgr.14164) 186 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:31:28.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:31:27.551841+0000 mon.smithi003 (mon.0) 202 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.eddcuc/mirror_snapshot_schedule"}]: dispatch 2023-11-17T05:31:28.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:31:27.555497+0000 mon.smithi003 (mon.0) 203 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.eddcuc/trash_purge_schedule"}]: dispatch 2023-11-17T05:31:29.700 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:29.699+0000 7fb40cfb4700 1 -- 172.21.15.160:0/2541413218 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4080f5410 msgr2=0x7fb4080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:29.701 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:29.699+0000 7fb40cfb4700 1 --2- 172.21.15.160:0/2541413218 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4080f5410 0x7fb4080f5830 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7fb3fc009fb0 tx=0x7fb3fc009b30).stop 2023-11-17T05:31:29.701 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:29.699+0000 7fb40cfb4700 1 -- 172.21.15.160:0/2541413218 shutdown_connections 2023-11-17T05:31:29.701 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:29.699+0000 7fb40cfb4700 1 --2- 172.21.15.160:0/2541413218 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4080f5410 0x7fb4080f5830 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:29.701 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:29.699+0000 7fb40cfb4700 1 -- 172.21.15.160:0/2541413218 >> 172.21.15.160:0/2541413218 conn(0x7fb4080f0b70 msgr2=0x7fb4080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:31:29.701 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:29.699+0000 7fb40cfb4700 1 -- 172.21.15.160:0/2541413218 shutdown_connections 2023-11-17T05:31:29.701 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:29.699+0000 7fb40cfb4700 1 -- 172.21.15.160:0/2541413218 wait complete. 2023-11-17T05:31:29.702 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:29.699+0000 7fb40cfb4700 1 Processor -- start 2023-11-17T05:31:29.702 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:29.699+0000 7fb40cfb4700 1 -- start start 2023-11-17T05:31:29.702 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:29.699+0000 7fb40cfb4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4080f5410 0x7fb408106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:31:29.702 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:29.699+0000 7fb40cfb4700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb408106c00 con 0x7fb4080f5410 2023-11-17T05:31:29.702 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:29.700+0000 7fb40659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4080f5410 0x7fb408106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:31:29.702 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:29.700+0000 7fb40659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4080f5410 0x7fb408106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:40934/0 (socket says 172.21.15.160:40934) 2023-11-17T05:31:29.702 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:29.700+0000 7fb40659c700 1 -- 172.21.15.160:0/1898320331 learned_addr learned my addr 172.21.15.160:0/1898320331 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:31:29.702 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:29.700+0000 7fb40659c700 1 -- 172.21.15.160:0/1898320331 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3fc005040 con 0x7fb4080f5410 2023-11-17T05:31:29.703 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:29.700+0000 7fb40659c700 1 --2- 172.21.15.160:0/1898320331 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4080f5410 0x7fb408106710 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fb3fc013510 tx=0x7fb3fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:31:29.703 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:29.700+0000 7fb3f6ffd700 1 -- 172.21.15.160:0/1898320331 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3fc009320 con 0x7fb4080f5410 2023-11-17T05:31:29.703 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:29.700+0000 7fb40cfb4700 1 -- 172.21.15.160:0/1898320331 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb408106e00 con 0x7fb4080f5410 2023-11-17T05:31:29.704 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:29.700+0000 7fb3f6ffd700 1 -- 172.21.15.160:0/1898320331 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb3fc013f00 con 0x7fb4080f5410 2023-11-17T05:31:29.704 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:29.700+0000 7fb3f6ffd700 1 -- 172.21.15.160:0/1898320331 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3fc014d90 con 0x7fb4080f5410 2023-11-17T05:31:29.704 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:29.700+0000 7fb40cfb4700 1 -- 172.21.15.160:0/1898320331 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb408107290 con 0x7fb4080f5410 2023-11-17T05:31:29.704 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:29.701+0000 7fb3f6ffd700 1 -- 172.21.15.160:0/1898320331 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb3fc0095f0 con 0x7fb4080f5410 2023-11-17T05:31:29.704 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:29.701+0000 7fb3f6ffd700 1 --2- 172.21.15.160:0/1898320331 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fb3f002ff60 0x7fb3f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:31:29.704 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:29.701+0000 7fb40cfb4700 1 -- 172.21.15.160:0/1898320331 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb408045ad0 con 0x7fb4080f5410 2023-11-17T05:31:29.704 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:29.701+0000 7fb3f6ffd700 1 -- 172.21.15.160:0/1898320331 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb3fc0410a0 con 0x7fb4080f5410 2023-11-17T05:31:29.704 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:29.701+0000 7fb405d9b700 1 --2- 172.21.15.160:0/1898320331 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fb3f002ff60 0x7fb3f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:31:29.704 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:29.702+0000 7fb405d9b700 1 --2- 172.21.15.160:0/1898320331 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fb3f002ff60 0x7fb3f0032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fb3f80096a0 tx=0x7fb3f8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:31:29.707 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:29.706+0000 7fb3f6ffd700 1 -- 172.21.15.160:0/1898320331 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb3fc06d2b0 con 0x7fb4080f5410 2023-11-17T05:31:29.913 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:29.911+0000 7fb40cfb4700 1 -- 172.21.15.160:0/1898320331 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb4080faf50 con 0x7fb4080f5410 2023-11-17T05:31:29.913 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:29.912+0000 7fb3f6ffd700 1 -- 172.21.15.160:0/1898320331 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fb3fc010230 con 0x7fb4080f5410 2023-11-17T05:31:29.914 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:31:29.914 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:31:29.915 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:29.914+0000 7fb40cfb4700 1 -- 172.21.15.160:0/1898320331 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fb3f002ff60 msgr2=0x7fb3f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:29.915 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:29.914+0000 7fb40cfb4700 1 --2- 172.21.15.160:0/1898320331 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fb3f002ff60 0x7fb3f0032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fb3f80096a0 tx=0x7fb3f8006b40).stop 2023-11-17T05:31:29.915 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:29.914+0000 7fb40cfb4700 1 -- 172.21.15.160:0/1898320331 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4080f5410 msgr2=0x7fb408106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:29.915 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:29.914+0000 7fb40cfb4700 1 --2- 172.21.15.160:0/1898320331 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4080f5410 0x7fb408106710 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fb3fc013510 tx=0x7fb3fc00b040).stop 2023-11-17T05:31:29.915 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:29.914+0000 7fb40cfb4700 1 -- 172.21.15.160:0/1898320331 shutdown_connections 2023-11-17T05:31:29.915 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:29.914+0000 7fb40cfb4700 1 --2- 172.21.15.160:0/1898320331 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fb3f002ff60 0x7fb3f0032420 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:29.915 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:29.914+0000 7fb40cfb4700 1 --2- 172.21.15.160:0/1898320331 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4080f5410 0x7fb408106710 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:29.915 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:29.914+0000 7fb40cfb4700 1 -- 172.21.15.160:0/1898320331 >> 172.21.15.160:0/1898320331 conn(0x7fb4080f0b70 msgr2=0x7fb40805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:31:29.915 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:29.914+0000 7fb40cfb4700 1 -- 172.21.15.160:0/1898320331 shutdown_connections 2023-11-17T05:31:29.915 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:29.914+0000 7fb40cfb4700 1 -- 172.21.15.160:0/1898320331 wait complete. 2023-11-17T05:31:29.916 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:31:30.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:29 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:31:29.447276+0000 mgr.smithi003.eddcuc (mgr.14164) 187 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:31:30.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:29 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:31:29.913314+0000 mon.smithi003 (mon.0) 204 : audit [DBG] from='client.? 172.21.15.160:0/1898320331' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:31:31.344 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:31:31.345 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:31:32.549 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:32.548+0000 7f2cd83d3700 1 -- 172.21.15.160:0/2840925244 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2cd0060280 msgr2=0x7f2cd00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:32.549 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:32.548+0000 7f2cd83d3700 1 --2- 172.21.15.160:0/2840925244 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2cd0060280 0x7f2cd00f90f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f2ccc009fb0 tx=0x7f2ccc009b30).stop 2023-11-17T05:31:32.550 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:32.548+0000 7f2cd83d3700 1 -- 172.21.15.160:0/2840925244 shutdown_connections 2023-11-17T05:31:32.550 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:32.548+0000 7f2cd83d3700 1 --2- 172.21.15.160:0/2840925244 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2cd0060280 0x7f2cd00f90f0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:32.550 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:32.548+0000 7f2cd83d3700 1 -- 172.21.15.160:0/2840925244 >> 172.21.15.160:0/2840925244 conn(0x7f2cd00f0b70 msgr2=0x7f2cd00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:31:32.550 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:32.548+0000 7f2cd83d3700 1 -- 172.21.15.160:0/2840925244 shutdown_connections 2023-11-17T05:31:32.550 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:32.548+0000 7f2cd83d3700 1 -- 172.21.15.160:0/2840925244 wait complete. 2023-11-17T05:31:32.550 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:32.548+0000 7f2cd83d3700 1 Processor -- start 2023-11-17T05:31:32.550 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:32.548+0000 7f2cd83d3700 1 -- start start 2023-11-17T05:31:32.550 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:32.549+0000 7f2cd83d3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2cd0060280 0x7f2cd01066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:31:32.550 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:32.549+0000 7f2cd83d3700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd0106bd0 con 0x7f2cd0060280 2023-11-17T05:31:32.552 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:32.549+0000 7f2cd616f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2cd0060280 0x7f2cd01066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:31:32.552 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:32.549+0000 7f2cd616f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2cd0060280 0x7f2cd01066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:58884/0 (socket says 172.21.15.160:58884) 2023-11-17T05:31:32.552 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:32.549+0000 7f2cd616f700 1 -- 172.21.15.160:0/2796457721 learned_addr learned my addr 172.21.15.160:0/2796457721 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:31:32.552 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:32.549+0000 7f2cd616f700 1 -- 172.21.15.160:0/2796457721 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ccc005040 con 0x7f2cd0060280 2023-11-17T05:31:32.552 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:32.549+0000 7f2cd616f700 1 --2- 172.21.15.160:0/2796457721 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2cd0060280 0x7f2cd01066e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f2ccc013510 tx=0x7f2ccc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:31:32.553 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:32.549+0000 7f2cc6ffd700 1 -- 172.21.15.160:0/2796457721 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ccc009320 con 0x7f2cd0060280 2023-11-17T05:31:32.553 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:32.549+0000 7f2cd83d3700 1 -- 172.21.15.160:0/2796457721 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cd0106dd0 con 0x7f2cd0060280 2023-11-17T05:31:32.553 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:32.549+0000 7f2cc6ffd700 1 -- 172.21.15.160:0/2796457721 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2ccc013f00 con 0x7f2cd0060280 2023-11-17T05:31:32.553 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:32.549+0000 7f2cc6ffd700 1 -- 172.21.15.160:0/2796457721 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ccc014dc0 con 0x7f2cd0060280 2023-11-17T05:31:32.553 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:32.550+0000 7f2cd83d3700 1 -- 172.21.15.160:0/2796457721 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cd0107260 con 0x7f2cd0060280 2023-11-17T05:31:32.553 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:32.550+0000 7f2cc6ffd700 1 -- 172.21.15.160:0/2796457721 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2ccc0095f0 con 0x7f2cd0060280 2023-11-17T05:31:32.553 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:32.551+0000 7f2cd83d3700 1 -- 172.21.15.160:0/2796457721 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cd0045ad0 con 0x7f2cd0060280 2023-11-17T05:31:32.553 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:32.551+0000 7f2cc6ffd700 1 --2- 172.21.15.160:0/2796457721 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f2cbc02ff60 0x7f2cbc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:31:32.553 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:32.551+0000 7f2cc6ffd700 1 -- 172.21.15.160:0/2796457721 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2ccc041000 con 0x7f2cd0060280 2023-11-17T05:31:32.553 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:32.551+0000 7f2cd596e700 1 --2- 172.21.15.160:0/2796457721 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f2cbc02ff60 0x7f2cbc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:31:32.553 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:32.551+0000 7f2cd596e700 1 --2- 172.21.15.160:0/2796457721 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f2cbc02ff60 0x7f2cbc032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f2cc00096a0 tx=0x7f2cc0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:31:32.558 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:32.555+0000 7f2cc6ffd700 1 -- 172.21.15.160:0/2796457721 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2ccc014420 con 0x7f2cd0060280 2023-11-17T05:31:32.763 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:32.762+0000 7f2cd83d3700 1 -- 172.21.15.160:0/2796457721 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2cd0108110 con 0x7f2cd0060280 2023-11-17T05:31:32.763 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:32.762+0000 7f2cc6ffd700 1 -- 172.21.15.160:0/2796457721 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f2ccc010230 con 0x7f2cd0060280 2023-11-17T05:31:32.763 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:31:32.763 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:31:32.765 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:32.764+0000 7f2cd83d3700 1 -- 172.21.15.160:0/2796457721 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f2cbc02ff60 msgr2=0x7f2cbc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:32.765 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:32.764+0000 7f2cd83d3700 1 --2- 172.21.15.160:0/2796457721 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f2cbc02ff60 0x7f2cbc032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f2cc00096a0 tx=0x7f2cc0006b40).stop 2023-11-17T05:31:32.765 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:32.764+0000 7f2cd83d3700 1 -- 172.21.15.160:0/2796457721 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2cd0060280 msgr2=0x7f2cd01066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:32.765 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:32.764+0000 7f2cd83d3700 1 --2- 172.21.15.160:0/2796457721 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2cd0060280 0x7f2cd01066e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f2ccc013510 tx=0x7f2ccc00b040).stop 2023-11-17T05:31:32.765 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:32.764+0000 7f2cd83d3700 1 -- 172.21.15.160:0/2796457721 shutdown_connections 2023-11-17T05:31:32.765 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:32.764+0000 7f2cd83d3700 1 --2- 172.21.15.160:0/2796457721 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f2cbc02ff60 0x7f2cbc032420 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:32.765 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:32.764+0000 7f2cd83d3700 1 --2- 172.21.15.160:0/2796457721 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2cd0060280 0x7f2cd01066e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:32.765 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:32.764+0000 7f2cd83d3700 1 -- 172.21.15.160:0/2796457721 >> 172.21.15.160:0/2796457721 conn(0x7f2cd00f0b70 msgr2=0x7f2cd00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:31:32.765 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:32.764+0000 7f2cd83d3700 1 -- 172.21.15.160:0/2796457721 shutdown_connections 2023-11-17T05:31:32.765 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:32.764+0000 7f2cd83d3700 1 -- 172.21.15.160:0/2796457721 wait complete. 2023-11-17T05:31:32.766 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:31:32.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:32 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:31:31.447642+0000 mgr.smithi003.eddcuc (mgr.14164) 188 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:31:33.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:31:32.763470+0000 mon.smithi003 (mon.0) 205 : audit [DBG] from='client.? 172.21.15.160:0/2796457721' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:31:34.151 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:31:34.151 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:31:34.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:34 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:31:33.447921+0000 mgr.smithi003.eddcuc (mgr.14164) 189 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:31:35.393 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:35.392+0000 7fc74f0e1700 1 -- 172.21.15.160:0/259740095 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7480f5410 msgr2=0x7fc7480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:35.393 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:35.392+0000 7fc74f0e1700 1 --2- 172.21.15.160:0/259740095 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7480f5410 0x7fc7480f5830 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fc73c009fb0 tx=0x7fc73c009b30).stop 2023-11-17T05:31:35.393 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:35.392+0000 7fc74f0e1700 1 -- 172.21.15.160:0/259740095 shutdown_connections 2023-11-17T05:31:35.393 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:35.392+0000 7fc74f0e1700 1 --2- 172.21.15.160:0/259740095 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7480f5410 0x7fc7480f5830 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:35.394 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:35.392+0000 7fc74f0e1700 1 -- 172.21.15.160:0/259740095 >> 172.21.15.160:0/259740095 conn(0x7fc7480f0b70 msgr2=0x7fc7480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:31:35.394 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:35.392+0000 7fc74f0e1700 1 -- 172.21.15.160:0/259740095 shutdown_connections 2023-11-17T05:31:35.394 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:35.392+0000 7fc74f0e1700 1 -- 172.21.15.160:0/259740095 wait complete. 2023-11-17T05:31:35.394 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:35.392+0000 7fc74f0e1700 1 Processor -- start 2023-11-17T05:31:35.394 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:35.392+0000 7fc74f0e1700 1 -- start start 2023-11-17T05:31:35.395 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:35.393+0000 7fc74f0e1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7480f5410 0x7fc7481093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:31:35.396 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:35.393+0000 7fc74f0e1700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc748060220 con 0x7fc7480f5410 2023-11-17T05:31:35.396 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:35.393+0000 7fc74ce7d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7480f5410 0x7fc7481093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:31:35.396 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:35.393+0000 7fc74ce7d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7480f5410 0x7fc7481093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:58908/0 (socket says 172.21.15.160:58908) 2023-11-17T05:31:35.396 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:35.393+0000 7fc74ce7d700 1 -- 172.21.15.160:0/744262756 learned_addr learned my addr 172.21.15.160:0/744262756 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:31:35.396 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:35.393+0000 7fc74ce7d700 1 -- 172.21.15.160:0/744262756 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc73c005040 con 0x7fc7480f5410 2023-11-17T05:31:35.396 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:35.393+0000 7fc74ce7d700 1 --2- 172.21.15.160:0/744262756 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7480f5410 0x7fc7481093f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fc73c013510 tx=0x7fc73c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:31:35.396 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:35.394+0000 7fc7457fa700 1 -- 172.21.15.160:0/744262756 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc73c009320 con 0x7fc7480f5410 2023-11-17T05:31:35.396 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:35.394+0000 7fc74f0e1700 1 -- 172.21.15.160:0/744262756 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc748060420 con 0x7fc7480f5410 2023-11-17T05:31:35.396 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:35.394+0000 7fc7457fa700 1 -- 172.21.15.160:0/744262756 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc73c013f00 con 0x7fc7480f5410 2023-11-17T05:31:35.396 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:35.394+0000 7fc7457fa700 1 -- 172.21.15.160:0/744262756 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc73c014d90 con 0x7fc7480f5410 2023-11-17T05:31:35.396 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:35.394+0000 7fc74f0e1700 1 -- 172.21.15.160:0/744262756 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc748109aa0 con 0x7fc7480f5410 2023-11-17T05:31:35.396 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:35.394+0000 7fc7457fa700 1 -- 172.21.15.160:0/744262756 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc73c0095f0 con 0x7fc7480f5410 2023-11-17T05:31:35.396 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:35.394+0000 7fc74f0e1700 1 -- 172.21.15.160:0/744262756 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc748059100 con 0x7fc7480f5410 2023-11-17T05:31:35.397 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:35.395+0000 7fc7457fa700 1 --2- 172.21.15.160:0/744262756 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc73802ff60 0x7fc738032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:31:35.397 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:35.395+0000 7fc7457fa700 1 -- 172.21.15.160:0/744262756 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc73c041080 con 0x7fc7480f5410 2023-11-17T05:31:35.397 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:35.395+0000 7fc747fff700 1 --2- 172.21.15.160:0/744262756 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc73802ff60 0x7fc738032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:31:35.397 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:35.395+0000 7fc747fff700 1 --2- 172.21.15.160:0/744262756 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc73802ff60 0x7fc738032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fc7340096a0 tx=0x7fc734006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:31:35.400 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:35.399+0000 7fc7457fa700 1 -- 172.21.15.160:0/744262756 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc73c014420 con 0x7fc7480f5410 2023-11-17T05:31:35.605 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:35.603+0000 7fc74f0e1700 1 -- 172.21.15.160:0/744262756 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc748045ad0 con 0x7fc7480f5410 2023-11-17T05:31:35.605 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:35.604+0000 7fc7457fa700 1 -- 172.21.15.160:0/744262756 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fc73c01d7f0 con 0x7fc7480f5410 2023-11-17T05:31:35.605 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:31:35.605 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:31:35.606 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:35.605+0000 7fc74f0e1700 1 -- 172.21.15.160:0/744262756 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc73802ff60 msgr2=0x7fc738032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:35.606 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:35.605+0000 7fc74f0e1700 1 --2- 172.21.15.160:0/744262756 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc73802ff60 0x7fc738032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fc7340096a0 tx=0x7fc734006b40).stop 2023-11-17T05:31:35.607 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:35.605+0000 7fc74f0e1700 1 -- 172.21.15.160:0/744262756 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7480f5410 msgr2=0x7fc7481093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:35.607 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:35.605+0000 7fc74f0e1700 1 --2- 172.21.15.160:0/744262756 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7480f5410 0x7fc7481093f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fc73c013510 tx=0x7fc73c00b040).stop 2023-11-17T05:31:35.607 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:35.605+0000 7fc74f0e1700 1 -- 172.21.15.160:0/744262756 shutdown_connections 2023-11-17T05:31:35.607 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:35.605+0000 7fc74f0e1700 1 --2- 172.21.15.160:0/744262756 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc73802ff60 0x7fc738032420 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:35.607 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:35.605+0000 7fc74f0e1700 1 --2- 172.21.15.160:0/744262756 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7480f5410 0x7fc7481093f0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:35.607 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:35.605+0000 7fc74f0e1700 1 -- 172.21.15.160:0/744262756 >> 172.21.15.160:0/744262756 conn(0x7fc7480f0b70 msgr2=0x7fc7480f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:31:35.607 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:35.605+0000 7fc74f0e1700 1 -- 172.21.15.160:0/744262756 shutdown_connections 2023-11-17T05:31:35.607 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:35.605+0000 7fc74f0e1700 1 -- 172.21.15.160:0/744262756 wait complete. 2023-11-17T05:31:35.607 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:31:36.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:36 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:31:35.448177+0000 mgr.smithi003.eddcuc (mgr.14164) 190 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:31:36.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:36 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:31:35.604788+0000 mon.smithi003 (mon.0) 206 : audit [DBG] from='client.? 172.21.15.160:0/744262756' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:31:37.022 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:31:37.023 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:31:38.194 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:38.192+0000 7f40f45cf700 1 -- 172.21.15.160:0/2139833558 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40ec0f5410 msgr2=0x7f40ec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:38.195 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:38.192+0000 7f40f45cf700 1 --2- 172.21.15.160:0/2139833558 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40ec0f5410 0x7f40ec0f5830 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f40dc009fb0 tx=0x7f40dc009b30).stop 2023-11-17T05:31:38.195 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:38.192+0000 7f40f45cf700 1 -- 172.21.15.160:0/2139833558 shutdown_connections 2023-11-17T05:31:38.195 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:38.192+0000 7f40f45cf700 1 --2- 172.21.15.160:0/2139833558 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40ec0f5410 0x7f40ec0f5830 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:38.195 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:38.192+0000 7f40f45cf700 1 -- 172.21.15.160:0/2139833558 >> 172.21.15.160:0/2139833558 conn(0x7f40ec0f0b70 msgr2=0x7f40ec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:31:38.195 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:38.192+0000 7f40f45cf700 1 -- 172.21.15.160:0/2139833558 shutdown_connections 2023-11-17T05:31:38.195 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:38.192+0000 7f40f45cf700 1 -- 172.21.15.160:0/2139833558 wait complete. 2023-11-17T05:31:38.195 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:38.193+0000 7f40f45cf700 1 Processor -- start 2023-11-17T05:31:38.195 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:38.193+0000 7f40f45cf700 1 -- start start 2023-11-17T05:31:38.195 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:38.193+0000 7f40f45cf700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40ec0f5410 0x7f40ec0fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:31:38.195 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:38.193+0000 7f40f45cf700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40ec0fb820 con 0x7f40ec0f5410 2023-11-17T05:31:38.195 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:38.193+0000 7f40f236b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40ec0f5410 0x7f40ec0fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:31:38.195 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:38.193+0000 7f40f236b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40ec0f5410 0x7f40ec0fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:58938/0 (socket says 172.21.15.160:58938) 2023-11-17T05:31:38.196 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:38.193+0000 7f40f236b700 1 -- 172.21.15.160:0/3275178164 learned_addr learned my addr 172.21.15.160:0/3275178164 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:31:38.196 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:38.194+0000 7f40f236b700 1 -- 172.21.15.160:0/3275178164 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40dc005040 con 0x7f40ec0f5410 2023-11-17T05:31:38.196 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:38.194+0000 7f40f236b700 1 --2- 172.21.15.160:0/3275178164 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40ec0f5410 0x7f40ec0fd3a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f40dc013510 tx=0x7f40dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:31:38.196 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:38.194+0000 7f40e2ffd700 1 -- 172.21.15.160:0/3275178164 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40dc009320 con 0x7f40ec0f5410 2023-11-17T05:31:38.197 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:38.194+0000 7f40e2ffd700 1 -- 172.21.15.160:0/3275178164 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f40dc013f00 con 0x7f40ec0f5410 2023-11-17T05:31:38.197 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:38.194+0000 7f40f45cf700 1 -- 172.21.15.160:0/3275178164 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40ec0fb9c0 con 0x7f40ec0f5410 2023-11-17T05:31:38.197 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:38.194+0000 7f40e2ffd700 1 -- 172.21.15.160:0/3275178164 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40dc014d90 con 0x7f40ec0f5410 2023-11-17T05:31:38.197 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:38.194+0000 7f40f45cf700 1 -- 172.21.15.160:0/3275178164 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40ec0fbe50 con 0x7f40ec0f5410 2023-11-17T05:31:38.197 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:38.195+0000 7f40e2ffd700 1 -- 172.21.15.160:0/3275178164 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f40dc0095f0 con 0x7f40ec0f5410 2023-11-17T05:31:38.197 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:38.195+0000 7f40f45cf700 1 -- 172.21.15.160:0/3275178164 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40ec059100 con 0x7f40ec0f5410 2023-11-17T05:31:38.197 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:38.195+0000 7f40e2ffd700 1 --2- 172.21.15.160:0/3275178164 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f40d802ff60 0x7f40d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:31:38.197 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:38.195+0000 7f40e2ffd700 1 -- 172.21.15.160:0/3275178164 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f40dc041080 con 0x7f40ec0f5410 2023-11-17T05:31:38.197 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:38.195+0000 7f40f1b6a700 1 --2- 172.21.15.160:0/3275178164 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f40d802ff60 0x7f40d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:31:38.197 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:38.196+0000 7f40f1b6a700 1 --2- 172.21.15.160:0/3275178164 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f40d802ff60 0x7f40d8032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f40e80096a0 tx=0x7f40e8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:31:38.201 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:38.200+0000 7f40e2ffd700 1 -- 172.21.15.160:0/3275178164 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f40dc014420 con 0x7f40ec0f5410 2023-11-17T05:31:38.408 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:38.406+0000 7f40f45cf700 1 -- 172.21.15.160:0/3275178164 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f40ec0fae60 con 0x7f40ec0f5410 2023-11-17T05:31:38.408 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:38.407+0000 7f40e2ffd700 1 -- 172.21.15.160:0/3275178164 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f40dc010230 con 0x7f40ec0f5410 2023-11-17T05:31:38.409 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:31:38.409 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:31:38.410 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:38.409+0000 7f40f45cf700 1 -- 172.21.15.160:0/3275178164 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f40d802ff60 msgr2=0x7f40d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:38.410 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:38.409+0000 7f40f45cf700 1 --2- 172.21.15.160:0/3275178164 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f40d802ff60 0x7f40d8032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f40e80096a0 tx=0x7f40e8006b40).stop 2023-11-17T05:31:38.410 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:38.409+0000 7f40f45cf700 1 -- 172.21.15.160:0/3275178164 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40ec0f5410 msgr2=0x7f40ec0fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:38.410 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:38.409+0000 7f40f45cf700 1 --2- 172.21.15.160:0/3275178164 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40ec0f5410 0x7f40ec0fd3a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f40dc013510 tx=0x7f40dc00b040).stop 2023-11-17T05:31:38.410 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:38.409+0000 7f40f45cf700 1 -- 172.21.15.160:0/3275178164 shutdown_connections 2023-11-17T05:31:38.410 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:38.409+0000 7f40f45cf700 1 --2- 172.21.15.160:0/3275178164 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f40d802ff60 0x7f40d8032420 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:38.410 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:38.409+0000 7f40f45cf700 1 --2- 172.21.15.160:0/3275178164 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f40ec0f5410 0x7f40ec0fd3a0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:38.410 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:38.409+0000 7f40f45cf700 1 -- 172.21.15.160:0/3275178164 >> 172.21.15.160:0/3275178164 conn(0x7f40ec0f0b70 msgr2=0x7f40ec05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:31:38.410 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:38.409+0000 7f40f45cf700 1 -- 172.21.15.160:0/3275178164 shutdown_connections 2023-11-17T05:31:38.410 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:38.409+0000 7f40f45cf700 1 -- 172.21.15.160:0/3275178164 wait complete. 2023-11-17T05:31:38.411 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:31:38.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:31:37.448557+0000 mgr.smithi003.eddcuc (mgr.14164) 191 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:31:38.852 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:31:38.408360+0000 mon.smithi003 (mon.0) 207 : audit [DBG] from='client.? 172.21.15.160:0/3275178164' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:31:39.854 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:31:39.855 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:31:40.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:39 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:31:39.448879+0000 mgr.smithi003.eddcuc (mgr.14164) 192 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:31:41.108 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:41.106+0000 7f59d0f93700 1 -- 172.21.15.160:0/592431940 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f59cc0f5410 msgr2=0x7f59cc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:41.108 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:41.106+0000 7f59d0f93700 1 --2- 172.21.15.160:0/592431940 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f59cc0f5410 0x7f59cc0f5830 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f59b4009fb0 tx=0x7f59b4009b30).stop 2023-11-17T05:31:41.108 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:41.106+0000 7f59d0f93700 1 -- 172.21.15.160:0/592431940 shutdown_connections 2023-11-17T05:31:41.108 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:41.106+0000 7f59d0f93700 1 --2- 172.21.15.160:0/592431940 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f59cc0f5410 0x7f59cc0f5830 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:41.108 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:41.106+0000 7f59d0f93700 1 -- 172.21.15.160:0/592431940 >> 172.21.15.160:0/592431940 conn(0x7f59cc0f0b70 msgr2=0x7f59cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:31:41.108 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:41.107+0000 7f59d0f93700 1 -- 172.21.15.160:0/592431940 shutdown_connections 2023-11-17T05:31:41.109 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:41.107+0000 7f59d0f93700 1 -- 172.21.15.160:0/592431940 wait complete. 2023-11-17T05:31:41.109 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:41.107+0000 7f59d0f93700 1 Processor -- start 2023-11-17T05:31:41.109 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:41.107+0000 7f59d0f93700 1 -- start start 2023-11-17T05:31:41.109 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:41.107+0000 7f59d0f93700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f59cc0f5410 0x7f59cc1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:31:41.109 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:41.107+0000 7f59d0f93700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59cc106bd0 con 0x7f59cc0f5410 2023-11-17T05:31:41.110 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:41.108+0000 7f59ca59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f59cc0f5410 0x7f59cc1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:31:41.110 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:41.108+0000 7f59ca59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f59cc0f5410 0x7f59cc1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:52098/0 (socket says 172.21.15.160:52098) 2023-11-17T05:31:41.111 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:41.108+0000 7f59ca59c700 1 -- 172.21.15.160:0/2414420128 learned_addr learned my addr 172.21.15.160:0/2414420128 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:31:41.111 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:41.108+0000 7f59ca59c700 1 -- 172.21.15.160:0/2414420128 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b4005040 con 0x7f59cc0f5410 2023-11-17T05:31:41.111 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:41.108+0000 7f59ca59c700 1 --2- 172.21.15.160:0/2414420128 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f59cc0f5410 0x7f59cc1066e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f59b4014000 tx=0x7f59b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:31:41.111 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:41.108+0000 7f59c2ffd700 1 -- 172.21.15.160:0/2414420128 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59b40095f0 con 0x7f59cc0f5410 2023-11-17T05:31:41.111 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:41.108+0000 7f59d0f93700 1 -- 172.21.15.160:0/2414420128 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59cc106dd0 con 0x7f59cc0f5410 2023-11-17T05:31:41.111 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:41.108+0000 7f59c2ffd700 1 -- 172.21.15.160:0/2414420128 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f59b4007e70 con 0x7f59cc0f5410 2023-11-17T05:31:41.111 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:41.108+0000 7f59c2ffd700 1 -- 172.21.15.160:0/2414420128 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59b4014ea0 con 0x7f59cc0f5410 2023-11-17T05:31:41.111 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:41.108+0000 7f59d0f93700 1 -- 172.21.15.160:0/2414420128 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59cc107260 con 0x7f59cc0f5410 2023-11-17T05:31:41.111 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:41.109+0000 7f59c2ffd700 1 -- 172.21.15.160:0/2414420128 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f59b4025070 con 0x7f59cc0f5410 2023-11-17T05:31:41.111 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:41.109+0000 7f59d0f93700 1 -- 172.21.15.160:0/2414420128 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59cc045ad0 con 0x7f59cc0f5410 2023-11-17T05:31:41.111 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:41.109+0000 7f59c2ffd700 1 --2- 172.21.15.160:0/2414420128 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f59b802ff60 0x7f59b8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:31:41.111 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:41.109+0000 7f59c2ffd700 1 -- 172.21.15.160:0/2414420128 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f59b4040d30 con 0x7f59cc0f5410 2023-11-17T05:31:41.112 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:41.110+0000 7f59c9d9b700 1 --2- 172.21.15.160:0/2414420128 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f59b802ff60 0x7f59b8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:31:41.112 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:41.110+0000 7f59c9d9b700 1 --2- 172.21.15.160:0/2414420128 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f59b802ff60 0x7f59b8032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f59bc0096a0 tx=0x7f59bc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:31:41.115 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:41.115+0000 7f59c2ffd700 1 -- 172.21.15.160:0/2414420128 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f59b400f710 con 0x7f59cc0f5410 2023-11-17T05:31:41.320 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:41.319+0000 7f59d0f93700 1 -- 172.21.15.160:0/2414420128 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f59cc1080b0 con 0x7f59cc0f5410 2023-11-17T05:31:41.321 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:41.320+0000 7f59c2ffd700 1 -- 172.21.15.160:0/2414420128 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f59b4025300 con 0x7f59cc0f5410 2023-11-17T05:31:41.349 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:31:41.349 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:31:41.350 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:41.321+0000 7f59d0f93700 1 -- 172.21.15.160:0/2414420128 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f59b802ff60 msgr2=0x7f59b8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:41.350 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:41.321+0000 7f59d0f93700 1 --2- 172.21.15.160:0/2414420128 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f59b802ff60 0x7f59b8032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f59bc0096a0 tx=0x7f59bc006b40).stop 2023-11-17T05:31:41.350 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:41.321+0000 7f59d0f93700 1 -- 172.21.15.160:0/2414420128 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f59cc0f5410 msgr2=0x7f59cc1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:41.350 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:41.321+0000 7f59d0f93700 1 --2- 172.21.15.160:0/2414420128 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f59cc0f5410 0x7f59cc1066e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f59b4014000 tx=0x7f59b400b040).stop 2023-11-17T05:31:41.350 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:41.322+0000 7f59d0f93700 1 -- 172.21.15.160:0/2414420128 shutdown_connections 2023-11-17T05:31:41.350 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:41.322+0000 7f59d0f93700 1 --2- 172.21.15.160:0/2414420128 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f59b802ff60 0x7f59b8032420 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:41.350 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:41.322+0000 7f59d0f93700 1 --2- 172.21.15.160:0/2414420128 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f59cc0f5410 0x7f59cc1066e0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:41.350 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:41.322+0000 7f59d0f93700 1 -- 172.21.15.160:0/2414420128 >> 172.21.15.160:0/2414420128 conn(0x7f59cc0f0b70 msgr2=0x7f59cc05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:31:41.350 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:41.322+0000 7f59d0f93700 1 -- 172.21.15.160:0/2414420128 shutdown_connections 2023-11-17T05:31:41.350 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:41.322+0000 7f59d0f93700 1 -- 172.21.15.160:0/2414420128 wait complete. 2023-11-17T05:31:41.351 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:31:41.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:41 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:31:41.320967+0000 mon.smithi003 (mon.0) 208 : audit [DBG] from='client.? 172.21.15.160:0/2414420128' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:31:42.705 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:31:42.706 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:31:42.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:42 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:31:41.449235+0000 mgr.smithi003.eddcuc (mgr.14164) 193 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:31:43.916 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:43.914+0000 7f601dc34700 1 -- 172.21.15.160:0/222403713 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60180f6510 msgr2=0x7f60180f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:43.916 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:43.914+0000 7f601dc34700 1 --2- 172.21.15.160:0/222403713 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60180f6510 0x7f60180f6930 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f6008009fb0 tx=0x7f6008009b30).stop 2023-11-17T05:31:43.916 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:43.914+0000 7f601dc34700 1 -- 172.21.15.160:0/222403713 shutdown_connections 2023-11-17T05:31:43.916 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:43.914+0000 7f601dc34700 1 --2- 172.21.15.160:0/222403713 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60180f6510 0x7f60180f6930 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:43.916 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:43.914+0000 7f601dc34700 1 -- 172.21.15.160:0/222403713 >> 172.21.15.160:0/222403713 conn(0x7f60180f1c70 msgr2=0x7f60180f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:31:43.916 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:43.914+0000 7f601dc34700 1 -- 172.21.15.160:0/222403713 shutdown_connections 2023-11-17T05:31:43.916 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:43.914+0000 7f601dc34700 1 -- 172.21.15.160:0/222403713 wait complete. 2023-11-17T05:31:43.916 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:43.914+0000 7f601dc34700 1 Processor -- start 2023-11-17T05:31:43.916 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:43.915+0000 7f601dc34700 1 -- start start 2023-11-17T05:31:43.919 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:43.915+0000 7f601dc34700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60180f6510 0x7f6018106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:31:43.919 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:43.915+0000 7f601dc34700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6018106c20 con 0x7f60180f6510 2023-11-17T05:31:43.919 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:43.915+0000 7f60177fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60180f6510 0x7f6018106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:31:43.920 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:43.915+0000 7f60177fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60180f6510 0x7f6018106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:52118/0 (socket says 172.21.15.160:52118) 2023-11-17T05:31:43.920 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:43.915+0000 7f60177fe700 1 -- 172.21.15.160:0/3618734934 learned_addr learned my addr 172.21.15.160:0/3618734934 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:31:43.920 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:43.915+0000 7f60177fe700 1 -- 172.21.15.160:0/3618734934 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6008005040 con 0x7f60180f6510 2023-11-17T05:31:43.920 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:43.915+0000 7f60177fe700 1 --2- 172.21.15.160:0/3618734934 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60180f6510 0x7f6018106730 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f6008013510 tx=0x7f600800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:31:43.920 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:43.916+0000 7f5ffffff700 1 -- 172.21.15.160:0/3618734934 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6008009320 con 0x7f60180f6510 2023-11-17T05:31:43.920 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:43.916+0000 7f601dc34700 1 -- 172.21.15.160:0/3618734934 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6018106e20 con 0x7f60180f6510 2023-11-17T05:31:43.920 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:43.916+0000 7f5ffffff700 1 -- 172.21.15.160:0/3618734934 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6008013f00 con 0x7f60180f6510 2023-11-17T05:31:43.920 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:43.916+0000 7f5ffffff700 1 -- 172.21.15.160:0/3618734934 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6008014d90 con 0x7f60180f6510 2023-11-17T05:31:43.920 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:43.916+0000 7f601dc34700 1 -- 172.21.15.160:0/3618734934 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60181072b0 con 0x7f60180f6510 2023-11-17T05:31:43.920 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:43.916+0000 7f5ffffff700 1 -- 172.21.15.160:0/3618734934 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f60080263a0 con 0x7f60180f6510 2023-11-17T05:31:43.920 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:43.917+0000 7f601dc34700 1 -- 172.21.15.160:0/3618734934 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6018045ad0 con 0x7f60180f6510 2023-11-17T05:31:43.920 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:43.917+0000 7f5ffffff700 1 --2- 172.21.15.160:0/3618734934 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f600002ff60 0x7f6000032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:31:43.920 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:43.917+0000 7f5ffffff700 1 -- 172.21.15.160:0/3618734934 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f60080410c0 con 0x7f60180f6510 2023-11-17T05:31:43.921 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:43.917+0000 7f6016ffd700 1 --2- 172.21.15.160:0/3618734934 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f600002ff60 0x7f6000032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:31:43.921 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:43.917+0000 7f6016ffd700 1 --2- 172.21.15.160:0/3618734934 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f600002ff60 0x7f6000032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f600c0096a0 tx=0x7f600c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:31:43.922 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:43.922+0000 7f5ffffff700 1 -- 172.21.15.160:0/3618734934 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f60080126f0 con 0x7f60180f6510 2023-11-17T05:31:44.128 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:44.127+0000 7f601dc34700 1 -- 172.21.15.160:0/3618734934 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f60180fbf60 con 0x7f60180f6510 2023-11-17T05:31:44.130 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:44.128+0000 7f5ffffff700 1 -- 172.21.15.160:0/3618734934 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f6008012330 con 0x7f60180f6510 2023-11-17T05:31:44.130 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:31:44.130 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:31:44.131 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:44.130+0000 7f601dc34700 1 -- 172.21.15.160:0/3618734934 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f600002ff60 msgr2=0x7f6000032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:44.131 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:44.130+0000 7f601dc34700 1 --2- 172.21.15.160:0/3618734934 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f600002ff60 0x7f6000032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f600c0096a0 tx=0x7f600c006b40).stop 2023-11-17T05:31:44.131 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:44.130+0000 7f601dc34700 1 -- 172.21.15.160:0/3618734934 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60180f6510 msgr2=0x7f6018106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:44.131 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:44.130+0000 7f601dc34700 1 --2- 172.21.15.160:0/3618734934 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60180f6510 0x7f6018106730 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f6008013510 tx=0x7f600800b040).stop 2023-11-17T05:31:44.131 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:44.130+0000 7f601dc34700 1 -- 172.21.15.160:0/3618734934 shutdown_connections 2023-11-17T05:31:44.131 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:44.130+0000 7f601dc34700 1 --2- 172.21.15.160:0/3618734934 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f600002ff60 0x7f6000032420 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:44.133 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:44.130+0000 7f601dc34700 1 --2- 172.21.15.160:0/3618734934 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60180f6510 0x7f6018106730 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:44.133 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:44.130+0000 7f601dc34700 1 -- 172.21.15.160:0/3618734934 >> 172.21.15.160:0/3618734934 conn(0x7f60180f1c70 msgr2=0x7f60180f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:31:44.133 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:44.130+0000 7f601dc34700 1 -- 172.21.15.160:0/3618734934 shutdown_connections 2023-11-17T05:31:44.133 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:44.130+0000 7f601dc34700 1 -- 172.21.15.160:0/3618734934 wait complete. 2023-11-17T05:31:44.134 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:31:44.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:44 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:31:43.449586+0000 mgr.smithi003.eddcuc (mgr.14164) 194 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:31:44.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:44 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:31:44.129217+0000 mon.smithi003 (mon.0) 209 : audit [DBG] from='client.? 172.21.15.160:0/3618734934' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:31:45.554 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:31:45.555 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:31:46.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:46.751+0000 7f9a3a3a0700 1 -- 172.21.15.160:0/1655795984 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9a340f5410 msgr2=0x7f9a340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:46.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:46.751+0000 7f9a3a3a0700 1 --2- 172.21.15.160:0/1655795984 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9a340f5410 0x7f9a340f5830 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f9a24009fb0 tx=0x7f9a24009b30).stop 2023-11-17T05:31:46.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:46.751+0000 7f9a3a3a0700 1 -- 172.21.15.160:0/1655795984 shutdown_connections 2023-11-17T05:31:46.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:46.751+0000 7f9a3a3a0700 1 --2- 172.21.15.160:0/1655795984 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9a340f5410 0x7f9a340f5830 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:46.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:46.751+0000 7f9a3a3a0700 1 -- 172.21.15.160:0/1655795984 >> 172.21.15.160:0/1655795984 conn(0x7f9a340f0b70 msgr2=0x7f9a340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:31:46.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:46.751+0000 7f9a3a3a0700 1 -- 172.21.15.160:0/1655795984 shutdown_connections 2023-11-17T05:31:46.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:46.751+0000 7f9a3a3a0700 1 -- 172.21.15.160:0/1655795984 wait complete. 2023-11-17T05:31:46.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:46.752+0000 7f9a3a3a0700 1 Processor -- start 2023-11-17T05:31:46.755 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:46.752+0000 7f9a3a3a0700 1 -- start start 2023-11-17T05:31:46.755 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:46.752+0000 7f9a3a3a0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9a340f5410 0x7f9a34109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:31:46.755 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:46.752+0000 7f9a3a3a0700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a34060240 con 0x7f9a340f5410 2023-11-17T05:31:46.755 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:46.752+0000 7f9a33fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9a340f5410 0x7f9a34109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:31:46.755 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:46.752+0000 7f9a33fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9a340f5410 0x7f9a34109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:52138/0 (socket says 172.21.15.160:52138) 2023-11-17T05:31:46.755 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:46.752+0000 7f9a33fff700 1 -- 172.21.15.160:0/861329011 learned_addr learned my addr 172.21.15.160:0/861329011 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:31:46.755 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:46.753+0000 7f9a33fff700 1 -- 172.21.15.160:0/861329011 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a24005040 con 0x7f9a340f5410 2023-11-17T05:31:46.755 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:46.753+0000 7f9a33fff700 1 --2- 172.21.15.160:0/861329011 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9a340f5410 0x7f9a34109430 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f9a24014000 tx=0x7f9a2400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:31:46.755 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:46.753+0000 7f9a30ff9700 1 -- 172.21.15.160:0/861329011 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a240095f0 con 0x7f9a340f5410 2023-11-17T05:31:46.755 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:46.753+0000 7f9a3a3a0700 1 -- 172.21.15.160:0/861329011 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a34060440 con 0x7f9a340f5410 2023-11-17T05:31:46.755 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:46.753+0000 7f9a30ff9700 1 -- 172.21.15.160:0/861329011 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9a24007e70 con 0x7f9a340f5410 2023-11-17T05:31:46.755 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:46.753+0000 7f9a30ff9700 1 -- 172.21.15.160:0/861329011 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a24014ea0 con 0x7f9a340f5410 2023-11-17T05:31:46.755 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:46.753+0000 7f9a3a3a0700 1 -- 172.21.15.160:0/861329011 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a34109ae0 con 0x7f9a340f5410 2023-11-17T05:31:46.756 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:46.754+0000 7f9a30ff9700 1 -- 172.21.15.160:0/861329011 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9a24014690 con 0x7f9a340f5410 2023-11-17T05:31:46.756 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:46.754+0000 7f9a3a3a0700 1 -- 172.21.15.160:0/861329011 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a34045ad0 con 0x7f9a340f5410 2023-11-17T05:31:46.757 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:46.754+0000 7f9a30ff9700 1 --2- 172.21.15.160:0/861329011 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f9a1c02ff60 0x7f9a1c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:31:46.757 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:46.754+0000 7f9a30ff9700 1 -- 172.21.15.160:0/861329011 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9a24040930 con 0x7f9a340f5410 2023-11-17T05:31:46.757 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:46.754+0000 7f9a337fe700 1 --2- 172.21.15.160:0/861329011 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f9a1c02ff60 0x7f9a1c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:31:46.757 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:46.754+0000 7f9a337fe700 1 --2- 172.21.15.160:0/861329011 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f9a1c02ff60 0x7f9a1c032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f9a280096a0 tx=0x7f9a28006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:31:46.760 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:46.759+0000 7f9a30ff9700 1 -- 172.21.15.160:0/861329011 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9a2400f300 con 0x7f9a340f5410 2023-11-17T05:31:46.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:46 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:31:45.449960+0000 mgr.smithi003.eddcuc (mgr.14164) 195 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:31:46.964 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:46.964+0000 7f9a3a3a0700 1 -- 172.21.15.160:0/861329011 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9a340fae60 con 0x7f9a340f5410 2023-11-17T05:31:47.693 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:46.964+0000 7f9a30ff9700 1 -- 172.21.15.160:0/861329011 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f9a2400f300 con 0x7f9a340f5410 2023-11-17T05:31:47.693 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:31:47.694 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:31:47.695 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:46.966+0000 7f9a3a3a0700 1 -- 172.21.15.160:0/861329011 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f9a1c02ff60 msgr2=0x7f9a1c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:47.695 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:46.966+0000 7f9a3a3a0700 1 --2- 172.21.15.160:0/861329011 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f9a1c02ff60 0x7f9a1c032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f9a280096a0 tx=0x7f9a28006b40).stop 2023-11-17T05:31:47.695 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:46.966+0000 7f9a3a3a0700 1 -- 172.21.15.160:0/861329011 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9a340f5410 msgr2=0x7f9a34109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:47.695 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:46.966+0000 7f9a3a3a0700 1 --2- 172.21.15.160:0/861329011 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9a340f5410 0x7f9a34109430 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f9a24014000 tx=0x7f9a2400b040).stop 2023-11-17T05:31:47.695 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:46.966+0000 7f9a3a3a0700 1 -- 172.21.15.160:0/861329011 shutdown_connections 2023-11-17T05:31:47.695 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:46.966+0000 7f9a3a3a0700 1 --2- 172.21.15.160:0/861329011 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f9a1c02ff60 0x7f9a1c032420 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:47.695 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:46.966+0000 7f9a3a3a0700 1 --2- 172.21.15.160:0/861329011 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9a340f5410 0x7f9a34109430 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:47.696 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:46.966+0000 7f9a3a3a0700 1 -- 172.21.15.160:0/861329011 >> 172.21.15.160:0/861329011 conn(0x7f9a340f0b70 msgr2=0x7f9a340f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:31:47.696 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:46.967+0000 7f9a3a3a0700 1 -- 172.21.15.160:0/861329011 shutdown_connections 2023-11-17T05:31:47.696 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:46.967+0000 7f9a3a3a0700 1 -- 172.21.15.160:0/861329011 wait complete. 2023-11-17T05:31:47.696 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:31:47.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:47 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:31:46.965654+0000 mon.smithi003 (mon.0) 210 : audit [DBG] from='client.? 172.21.15.160:0/861329011' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:31:48.697 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:31:48.697 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:31:48.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:48 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:31:47.450320+0000 mgr.smithi003.eddcuc (mgr.14164) 196 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:31:49.949 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:49.947+0000 7f68c961b700 1 -- 172.21.15.160:0/511734654 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68c4060280 msgr2=0x7f68c40f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:49.949 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:49.947+0000 7f68c961b700 1 --2- 172.21.15.160:0/511734654 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68c4060280 0x7f68c40f90f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f68b8009fb0 tx=0x7f68b8009b30).stop 2023-11-17T05:31:49.949 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:49.947+0000 7f68c961b700 1 -- 172.21.15.160:0/511734654 shutdown_connections 2023-11-17T05:31:49.949 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:49.947+0000 7f68c961b700 1 --2- 172.21.15.160:0/511734654 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68c4060280 0x7f68c40f90f0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:49.952 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:49.947+0000 7f68c961b700 1 -- 172.21.15.160:0/511734654 >> 172.21.15.160:0/511734654 conn(0x7f68c40f0b70 msgr2=0x7f68c40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:31:49.952 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:49.947+0000 7f68c961b700 1 -- 172.21.15.160:0/511734654 shutdown_connections 2023-11-17T05:31:49.952 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:49.947+0000 7f68c961b700 1 -- 172.21.15.160:0/511734654 wait complete. 2023-11-17T05:31:49.952 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:49.948+0000 7f68c961b700 1 Processor -- start 2023-11-17T05:31:49.952 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:49.948+0000 7f68c961b700 1 -- start start 2023-11-17T05:31:49.952 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:49.948+0000 7f68c961b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68c4060280 0x7f68c41066c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:31:49.952 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:49.948+0000 7f68c961b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68c4106bb0 con 0x7f68c4060280 2023-11-17T05:31:49.953 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:49.948+0000 7f68c2ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68c4060280 0x7f68c41066c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:31:49.953 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:49.948+0000 7f68c2ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68c4060280 0x7f68c41066c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:52144/0 (socket says 172.21.15.160:52144) 2023-11-17T05:31:49.953 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:49.948+0000 7f68c2ffd700 1 -- 172.21.15.160:0/3918720848 learned_addr learned my addr 172.21.15.160:0/3918720848 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:31:49.953 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:49.948+0000 7f68c2ffd700 1 -- 172.21.15.160:0/3918720848 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b8005040 con 0x7f68c4060280 2023-11-17T05:31:49.953 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:49.948+0000 7f68c2ffd700 1 --2- 172.21.15.160:0/3918720848 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68c4060280 0x7f68c41066c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f68b8013510 tx=0x7f68b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:31:49.954 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:49.949+0000 7f68ab7fe700 1 -- 172.21.15.160:0/3918720848 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68b8009320 con 0x7f68c4060280 2023-11-17T05:31:49.954 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:49.949+0000 7f68c961b700 1 -- 172.21.15.160:0/3918720848 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68c4106db0 con 0x7f68c4060280 2023-11-17T05:31:49.954 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:49.949+0000 7f68ab7fe700 1 -- 172.21.15.160:0/3918720848 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f68b8013f00 con 0x7f68c4060280 2023-11-17T05:31:49.954 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:49.949+0000 7f68ab7fe700 1 -- 172.21.15.160:0/3918720848 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68b8014dc0 con 0x7f68c4060280 2023-11-17T05:31:49.954 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:49.949+0000 7f68c961b700 1 -- 172.21.15.160:0/3918720848 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68c4107240 con 0x7f68c4060280 2023-11-17T05:31:49.954 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:49.949+0000 7f68ab7fe700 1 -- 172.21.15.160:0/3918720848 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f68b80095f0 con 0x7f68c4060280 2023-11-17T05:31:49.954 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:49.949+0000 7f68c961b700 1 -- 172.21.15.160:0/3918720848 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68c4045ad0 con 0x7f68c4060280 2023-11-17T05:31:49.954 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:49.950+0000 7f68ab7fe700 1 --2- 172.21.15.160:0/3918720848 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f68ac02ff60 0x7f68ac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:31:49.955 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:49.950+0000 7f68ab7fe700 1 -- 172.21.15.160:0/3918720848 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f68b8041100 con 0x7f68c4060280 2023-11-17T05:31:49.955 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:49.950+0000 7f68c27fc700 1 --2- 172.21.15.160:0/3918720848 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f68ac02ff60 0x7f68ac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:31:49.955 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:49.950+0000 7f68c27fc700 1 --2- 172.21.15.160:0/3918720848 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f68ac02ff60 0x7f68ac032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f68b40096a0 tx=0x7f68b4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:31:49.955 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:49.954+0000 7f68ab7fe700 1 -- 172.21.15.160:0/3918720848 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f68b8014420 con 0x7f68c4060280 2023-11-17T05:31:50.162 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:50.161+0000 7f68c961b700 1 -- 172.21.15.160:0/3918720848 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f68c41080f0 con 0x7f68c4060280 2023-11-17T05:31:59.573 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:50.162+0000 7f68ab7fe700 1 -- 172.21.15.160:0/3918720848 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f68b8010230 con 0x7f68c4060280 2023-11-17T05:31:59.574 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:31:59.574 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:31:59.575 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:49 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:31:49.450699+0000 mgr.smithi003.eddcuc (mgr.14164) 197 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:31:59.575 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:50 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:31:50.163067+0000 mon.smithi003 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.160:0/3918720848' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:31:59.575 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:51 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:31:51.451026+0000 mgr.smithi003.eddcuc (mgr.14164) 198 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:31:59.575 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:54 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:31:53.451444+0000 mgr.smithi003.eddcuc (mgr.14164) 199 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:31:59.575 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:56 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:31:55.451783+0000 mgr.smithi003.eddcuc (mgr.14164) 200 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:31:59.575 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:58 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:31:57.452124+0000 mgr.smithi003.eddcuc (mgr.14164) 201 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:31:59.576 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:50.164+0000 7f68c961b700 1 -- 172.21.15.160:0/3918720848 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f68ac02ff60 msgr2=0x7f68ac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:59.576 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:50.164+0000 7f68c961b700 1 --2- 172.21.15.160:0/3918720848 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f68ac02ff60 0x7f68ac032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f68b40096a0 tx=0x7f68b4006b40).stop 2023-11-17T05:31:59.576 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:50.164+0000 7f68c961b700 1 -- 172.21.15.160:0/3918720848 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68c4060280 msgr2=0x7f68c41066c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:31:59.576 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:50.164+0000 7f68c961b700 1 --2- 172.21.15.160:0/3918720848 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68c4060280 0x7f68c41066c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f68b8013510 tx=0x7f68b800b040).stop 2023-11-17T05:31:59.576 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:50.164+0000 7f68c961b700 1 -- 172.21.15.160:0/3918720848 shutdown_connections 2023-11-17T05:31:59.576 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:50.164+0000 7f68c961b700 1 --2- 172.21.15.160:0/3918720848 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f68ac02ff60 0x7f68ac032420 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:59.577 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:50.164+0000 7f68c961b700 1 --2- 172.21.15.160:0/3918720848 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68c4060280 0x7f68c41066c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:31:59.577 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:50.164+0000 7f68c961b700 1 -- 172.21.15.160:0/3918720848 >> 172.21.15.160:0/3918720848 conn(0x7f68c40f0b70 msgr2=0x7f68c40f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:31:59.577 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:50.164+0000 7f68c961b700 1 -- 172.21.15.160:0/3918720848 shutdown_connections 2023-11-17T05:31:59.577 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:31:50.164+0000 7f68c961b700 1 -- 172.21.15.160:0/3918720848 wait complete. 2023-11-17T05:31:59.577 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:32:00.275 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:31:59 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:31:59.452436+0000 mgr.smithi003.eddcuc (mgr.14164) 202 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:32:00.578 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:32:00.578 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:32:01.853 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:01.851+0000 7f8f45824700 1 -- 172.21.15.160:0/1809955187 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8f400f59d0 msgr2=0x7f8f400f5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:32:01.853 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:01.851+0000 7f8f45824700 1 --2- 172.21.15.160:0/1809955187 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8f400f59d0 0x7f8f400f5df0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f8f34009fb0 tx=0x7f8f34009b30).stop 2023-11-17T05:32:01.853 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:01.851+0000 7f8f45824700 1 -- 172.21.15.160:0/1809955187 shutdown_connections 2023-11-17T05:32:01.853 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:01.851+0000 7f8f45824700 1 --2- 172.21.15.160:0/1809955187 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8f400f59d0 0x7f8f400f5df0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:32:01.853 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:01.851+0000 7f8f45824700 1 -- 172.21.15.160:0/1809955187 >> 172.21.15.160:0/1809955187 conn(0x7f8f400f02e0 msgr2=0x7f8f400f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:32:01.854 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:01.851+0000 7f8f45824700 1 -- 172.21.15.160:0/1809955187 shutdown_connections 2023-11-17T05:32:01.854 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:01.851+0000 7f8f45824700 1 -- 172.21.15.160:0/1809955187 wait complete. 2023-11-17T05:32:01.854 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:01.852+0000 7f8f45824700 1 Processor -- start 2023-11-17T05:32:01.854 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:01.852+0000 7f8f45824700 1 -- start start 2023-11-17T05:32:01.854 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:01.852+0000 7f8f45824700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8f400f59d0 0x7f8f401093c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:32:01.854 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:01.852+0000 7f8f45824700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f400f53b0 con 0x7f8f400f59d0 2023-11-17T05:32:01.854 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:01.852+0000 7f8f3effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8f400f59d0 0x7f8f401093c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:32:01.854 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:01.852+0000 7f8f3effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8f400f59d0 0x7f8f401093c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:42722/0 (socket says 172.21.15.160:42722) 2023-11-17T05:32:01.854 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:01.852+0000 7f8f3effd700 1 -- 172.21.15.160:0/3694736966 learned_addr learned my addr 172.21.15.160:0/3694736966 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:32:01.855 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:01.853+0000 7f8f3effd700 1 -- 172.21.15.160:0/3694736966 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f34005040 con 0x7f8f400f59d0 2023-11-17T05:32:01.856 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:01.853+0000 7f8f3effd700 1 --2- 172.21.15.160:0/3694736966 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8f400f59d0 0x7f8f401093c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f8f34013510 tx=0x7f8f3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:32:01.856 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:01.853+0000 7f8f277fe700 1 -- 172.21.15.160:0/3694736966 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f34009320 con 0x7f8f400f59d0 2023-11-17T05:32:01.856 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:01.853+0000 7f8f45824700 1 -- 172.21.15.160:0/3694736966 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f400f55b0 con 0x7f8f400f59d0 2023-11-17T05:32:01.856 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:01.853+0000 7f8f277fe700 1 -- 172.21.15.160:0/3694736966 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8f34013f00 con 0x7f8f400f59d0 2023-11-17T05:32:01.856 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:01.853+0000 7f8f277fe700 1 -- 172.21.15.160:0/3694736966 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f34014d90 con 0x7f8f400f59d0 2023-11-17T05:32:01.856 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:01.853+0000 7f8f45824700 1 -- 172.21.15.160:0/3694736966 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f40109a70 con 0x7f8f400f59d0 2023-11-17T05:32:01.856 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:01.854+0000 7f8f277fe700 1 -- 172.21.15.160:0/3694736966 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8f340263a0 con 0x7f8f400f59d0 2023-11-17T05:32:01.856 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:01.854+0000 7f8f45824700 1 -- 172.21.15.160:0/3694736966 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f40045ad0 con 0x7f8f400f59d0 2023-11-17T05:32:01.856 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:01.854+0000 7f8f277fe700 1 --2- 172.21.15.160:0/3694736966 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f8f2802ff60 0x7f8f28032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:32:01.856 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:01.854+0000 7f8f277fe700 1 -- 172.21.15.160:0/3694736966 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8f34041310 con 0x7f8f400f59d0 2023-11-17T05:32:01.856 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:01.854+0000 7f8f3e7fc700 1 --2- 172.21.15.160:0/3694736966 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f8f2802ff60 0x7f8f28032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:32:01.857 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:01.855+0000 7f8f3e7fc700 1 --2- 172.21.15.160:0/3694736966 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f8f2802ff60 0x7f8f28032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f8f300096a0 tx=0x7f8f30006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:32:01.861 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:01.859+0000 7f8f277fe700 1 -- 172.21.15.160:0/3694736966 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8f34012510 con 0x7f8f400f59d0 2023-11-17T05:32:02.064 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:02.063+0000 7f8f45824700 1 -- 172.21.15.160:0/3694736966 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8f4010a290 con 0x7f8f400f59d0 2023-11-17T05:32:02.065 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:02.064+0000 7f8f277fe700 1 -- 172.21.15.160:0/3694736966 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f8f34012330 con 0x7f8f400f59d0 2023-11-17T05:32:02.065 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:32:02.065 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:32:02.066 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:02.066+0000 7f8f45824700 1 -- 172.21.15.160:0/3694736966 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f8f2802ff60 msgr2=0x7f8f28032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:32:02.066 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:02.066+0000 7f8f45824700 1 --2- 172.21.15.160:0/3694736966 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f8f2802ff60 0x7f8f28032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f8f300096a0 tx=0x7f8f30006b40).stop 2023-11-17T05:32:02.067 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:02.066+0000 7f8f45824700 1 -- 172.21.15.160:0/3694736966 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8f400f59d0 msgr2=0x7f8f401093c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:32:02.067 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:02.066+0000 7f8f45824700 1 --2- 172.21.15.160:0/3694736966 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8f400f59d0 0x7f8f401093c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f8f34013510 tx=0x7f8f3400b040).stop 2023-11-17T05:32:02.067 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:02.066+0000 7f8f45824700 1 -- 172.21.15.160:0/3694736966 shutdown_connections 2023-11-17T05:32:02.067 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:02.066+0000 7f8f45824700 1 --2- 172.21.15.160:0/3694736966 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f8f2802ff60 0x7f8f28032420 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:32:02.067 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:02.066+0000 7f8f45824700 1 --2- 172.21.15.160:0/3694736966 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8f400f59d0 0x7f8f401093c0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:32:02.067 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:02.066+0000 7f8f45824700 1 -- 172.21.15.160:0/3694736966 >> 172.21.15.160:0/3694736966 conn(0x7f8f400f02e0 msgr2=0x7f8f400f0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:32:02.067 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:02.066+0000 7f8f45824700 1 -- 172.21.15.160:0/3694736966 shutdown_connections 2023-11-17T05:32:02.067 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:02.066+0000 7f8f45824700 1 -- 172.21.15.160:0/3694736966 wait complete. 2023-11-17T05:32:02.067 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:32:02.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:02 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:32:01.452715+0000 mgr.smithi003.eddcuc (mgr.14164) 203 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:32:02.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:02 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:32:02.065079+0000 mon.smithi003 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.160:0/3694736966' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:32:03.456 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:32:03.456 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:32:04.645 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:04.643+0000 7f814b397700 1 -- 172.21.15.160:0/3694010303 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81440f62a0 msgr2=0x7f81440f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:32:04.645 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:04.643+0000 7f814b397700 1 --2- 172.21.15.160:0/3694010303 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81440f62a0 0x7f81440f66c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f8134009fb0 tx=0x7f8134009b30).stop 2023-11-17T05:32:04.645 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:04.643+0000 7f814b397700 1 -- 172.21.15.160:0/3694010303 shutdown_connections 2023-11-17T05:32:04.645 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:04.643+0000 7f814b397700 1 --2- 172.21.15.160:0/3694010303 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81440f62a0 0x7f81440f66c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:32:04.645 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:04.643+0000 7f814b397700 1 -- 172.21.15.160:0/3694010303 >> 172.21.15.160:0/3694010303 conn(0x7f81440f1aa0 msgr2=0x7f81440f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:32:04.645 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:04.643+0000 7f814b397700 1 -- 172.21.15.160:0/3694010303 shutdown_connections 2023-11-17T05:32:04.645 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:04.643+0000 7f814b397700 1 -- 172.21.15.160:0/3694010303 wait complete. 2023-11-17T05:32:04.645 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:04.643+0000 7f814b397700 1 Processor -- start 2023-11-17T05:32:04.645 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:04.643+0000 7f814b397700 1 -- start start 2023-11-17T05:32:04.645 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:04.643+0000 7f814b397700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81440f62a0 0x7f81441064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:32:04.646 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:04.643+0000 7f814b397700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81441069a0 con 0x7f81440f62a0 2023-11-17T05:32:04.646 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:04.644+0000 7f814a395700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81440f62a0 0x7f81441064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:32:04.646 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:04.644+0000 7f814a395700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81440f62a0 0x7f81441064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:42752/0 (socket says 172.21.15.160:42752) 2023-11-17T05:32:04.646 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:04.644+0000 7f814a395700 1 -- 172.21.15.160:0/1934560781 learned_addr learned my addr 172.21.15.160:0/1934560781 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:32:04.646 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:04.644+0000 7f814a395700 1 -- 172.21.15.160:0/1934560781 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8134005040 con 0x7f81440f62a0 2023-11-17T05:32:04.646 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:04.644+0000 7f814a395700 1 --2- 172.21.15.160:0/1934560781 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81440f62a0 0x7f81441064b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f8134013510 tx=0x7f813400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:32:04.647 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:04.644+0000 7f813affd700 1 -- 172.21.15.160:0/1934560781 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8134009320 con 0x7f81440f62a0 2023-11-17T05:32:04.647 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:04.644+0000 7f814b397700 1 -- 172.21.15.160:0/1934560781 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8144106ba0 con 0x7f81440f62a0 2023-11-17T05:32:04.647 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:04.644+0000 7f814b397700 1 -- 172.21.15.160:0/1934560781 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8144107030 con 0x7f81440f62a0 2023-11-17T05:32:04.647 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:04.644+0000 7f813affd700 1 -- 172.21.15.160:0/1934560781 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8134013f00 con 0x7f81440f62a0 2023-11-17T05:32:04.647 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:04.644+0000 7f813affd700 1 -- 172.21.15.160:0/1934560781 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8134014d90 con 0x7f81440f62a0 2023-11-17T05:32:04.647 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:04.645+0000 7f813affd700 1 -- 172.21.15.160:0/1934560781 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8134014420 con 0x7f81440f62a0 2023-11-17T05:32:04.648 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:04.645+0000 7f814b397700 1 -- 172.21.15.160:0/1934560781 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8144045ad0 con 0x7f81440f62a0 2023-11-17T05:32:04.648 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:04.645+0000 7f813affd700 1 --2- 172.21.15.160:0/1934560781 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f813002ff60 0x7f8130032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:32:04.648 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:04.645+0000 7f813affd700 1 -- 172.21.15.160:0/1934560781 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8134041310 con 0x7f81440f62a0 2023-11-17T05:32:04.648 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:04.646+0000 7f8149b94700 1 --2- 172.21.15.160:0/1934560781 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f813002ff60 0x7f8130032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:32:04.648 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:04.646+0000 7f8149b94700 1 --2- 172.21.15.160:0/1934560781 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f813002ff60 0x7f8130032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f81400096a0 tx=0x7f8140006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:32:04.651 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:04.650+0000 7f813affd700 1 -- 172.21.15.160:0/1934560781 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8134012d00 con 0x7f81440f62a0 2023-11-17T05:32:04.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:04 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:32:03.452972+0000 mgr.smithi003.eddcuc (mgr.14164) 204 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:32:04.856 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:04.855+0000 7f814b397700 1 -- 172.21.15.160:0/1934560781 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8144107ee0 con 0x7f81440f62a0 2023-11-17T05:32:04.857 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:04.856+0000 7f813affd700 1 -- 172.21.15.160:0/1934560781 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f813401c320 con 0x7f81440f62a0 2023-11-17T05:32:04.857 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:32:04.857 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:32:04.858 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:04.857+0000 7f814b397700 1 -- 172.21.15.160:0/1934560781 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f813002ff60 msgr2=0x7f8130032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:32:04.858 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:04.857+0000 7f814b397700 1 --2- 172.21.15.160:0/1934560781 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f813002ff60 0x7f8130032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f81400096a0 tx=0x7f8140006b40).stop 2023-11-17T05:32:04.858 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:04.857+0000 7f814b397700 1 -- 172.21.15.160:0/1934560781 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81440f62a0 msgr2=0x7f81441064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:32:04.858 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:04.857+0000 7f814b397700 1 --2- 172.21.15.160:0/1934560781 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81440f62a0 0x7f81441064b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f8134013510 tx=0x7f813400b040).stop 2023-11-17T05:32:04.859 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:04.857+0000 7f814b397700 1 -- 172.21.15.160:0/1934560781 shutdown_connections 2023-11-17T05:32:04.859 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:04.857+0000 7f814b397700 1 --2- 172.21.15.160:0/1934560781 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f813002ff60 0x7f8130032420 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:32:04.859 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:04.857+0000 7f814b397700 1 --2- 172.21.15.160:0/1934560781 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81440f62a0 0x7f81441064b0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:32:04.859 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:04.857+0000 7f814b397700 1 -- 172.21.15.160:0/1934560781 >> 172.21.15.160:0/1934560781 conn(0x7f81440f1aa0 msgr2=0x7f81440f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:32:04.859 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:04.857+0000 7f814b397700 1 -- 172.21.15.160:0/1934560781 shutdown_connections 2023-11-17T05:32:04.859 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:04.857+0000 7f814b397700 1 -- 172.21.15.160:0/1934560781 wait complete. 2023-11-17T05:32:04.859 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:32:05.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:05 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:32:04.856907+0000 mon.smithi003 (mon.0) 213 : audit [DBG] from='client.? 172.21.15.160:0/1934560781' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:32:06.257 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:32:06.257 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:32:06.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:06 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:32:05.453319+0000 mgr.smithi003.eddcuc (mgr.14164) 205 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:32:07.453 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:07.452+0000 7ff875a01700 1 -- 172.21.15.160:0/670707227 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8700f51a0 msgr2=0x7ff8700f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:32:07.454 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:07.452+0000 7ff875a01700 1 --2- 172.21.15.160:0/670707227 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8700f51a0 0x7ff8700f55c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7ff864009fb0 tx=0x7ff864009b30).stop 2023-11-17T05:32:07.454 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:07.452+0000 7ff875a01700 1 -- 172.21.15.160:0/670707227 shutdown_connections 2023-11-17T05:32:07.454 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:07.452+0000 7ff875a01700 1 --2- 172.21.15.160:0/670707227 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8700f51a0 0x7ff8700f55c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:32:07.454 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:07.452+0000 7ff875a01700 1 -- 172.21.15.160:0/670707227 >> 172.21.15.160:0/670707227 conn(0x7ff8700f09a0 msgr2=0x7ff8700f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:32:07.454 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:07.452+0000 7ff875a01700 1 -- 172.21.15.160:0/670707227 shutdown_connections 2023-11-17T05:32:07.454 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:07.452+0000 7ff875a01700 1 -- 172.21.15.160:0/670707227 wait complete. 2023-11-17T05:32:07.454 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:07.453+0000 7ff875a01700 1 Processor -- start 2023-11-17T05:32:07.454 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:07.453+0000 7ff875a01700 1 -- start start 2023-11-17T05:32:07.455 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:07.453+0000 7ff875a01700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8700f51a0 0x7ff8701064c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:32:07.455 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:07.453+0000 7ff875a01700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8701069b0 con 0x7ff8700f51a0 2023-11-17T05:32:07.455 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:07.453+0000 7ff8749ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8700f51a0 0x7ff8701064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:32:07.455 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:07.453+0000 7ff8749ff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8700f51a0 0x7ff8701064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:42780/0 (socket says 172.21.15.160:42780) 2023-11-17T05:32:07.455 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:07.453+0000 7ff8749ff700 1 -- 172.21.15.160:0/3913231381 learned_addr learned my addr 172.21.15.160:0/3913231381 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:32:07.456 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:07.454+0000 7ff8749ff700 1 -- 172.21.15.160:0/3913231381 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff864005040 con 0x7ff8700f51a0 2023-11-17T05:32:07.456 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:07.454+0000 7ff8749ff700 1 --2- 172.21.15.160:0/3913231381 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8700f51a0 0x7ff8701064c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7ff864013510 tx=0x7ff86400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:32:07.456 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:07.454+0000 7ff86d7fa700 1 -- 172.21.15.160:0/3913231381 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff864009320 con 0x7ff8700f51a0 2023-11-17T05:32:07.456 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:07.454+0000 7ff875a01700 1 -- 172.21.15.160:0/3913231381 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff870106bb0 con 0x7ff8700f51a0 2023-11-17T05:32:07.456 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:07.454+0000 7ff875a01700 1 -- 172.21.15.160:0/3913231381 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff870107040 con 0x7ff8700f51a0 2023-11-17T05:32:07.456 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:07.454+0000 7ff86d7fa700 1 -- 172.21.15.160:0/3913231381 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff864013f00 con 0x7ff8700f51a0 2023-11-17T05:32:07.456 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:07.454+0000 7ff86d7fa700 1 -- 172.21.15.160:0/3913231381 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff864014d90 con 0x7ff8700f51a0 2023-11-17T05:32:07.457 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:07.455+0000 7ff86d7fa700 1 -- 172.21.15.160:0/3913231381 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff8640263a0 con 0x7ff8700f51a0 2023-11-17T05:32:07.457 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:07.455+0000 7ff875a01700 1 -- 172.21.15.160:0/3913231381 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff870045ad0 con 0x7ff8700f51a0 2023-11-17T05:32:07.457 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:07.455+0000 7ff86d7fa700 1 --2- 172.21.15.160:0/3913231381 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7ff86002ff60 0x7ff860032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:32:07.457 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:07.455+0000 7ff86d7fa700 1 -- 172.21.15.160:0/3913231381 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff8640412c0 con 0x7ff8700f51a0 2023-11-17T05:32:07.457 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:07.455+0000 7ff86ffff700 1 --2- 172.21.15.160:0/3913231381 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7ff86002ff60 0x7ff860032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:32:07.457 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:07.456+0000 7ff86ffff700 1 --2- 172.21.15.160:0/3913231381 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7ff86002ff60 0x7ff860032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7ff85c0096a0 tx=0x7ff85c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:32:07.461 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:07.460+0000 7ff86d7fa700 1 -- 172.21.15.160:0/3913231381 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff86401c320 con 0x7ff8700f51a0 2023-11-17T05:32:07.667 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:07.667+0000 7ff875a01700 1 -- 172.21.15.160:0/3913231381 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff870107ef0 con 0x7ff8700f51a0 2023-11-17T05:32:07.668 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:07.667+0000 7ff86d7fa700 1 -- 172.21.15.160:0/3913231381 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7ff86400fe60 con 0x7ff8700f51a0 2023-11-17T05:32:07.668 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:32:07.668 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:32:07.674 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:07.669+0000 7ff875a01700 1 -- 172.21.15.160:0/3913231381 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7ff86002ff60 msgr2=0x7ff860032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:32:07.675 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:07.669+0000 7ff875a01700 1 --2- 172.21.15.160:0/3913231381 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7ff86002ff60 0x7ff860032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7ff85c0096a0 tx=0x7ff85c006b40).stop 2023-11-17T05:32:07.676 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:07.669+0000 7ff875a01700 1 -- 172.21.15.160:0/3913231381 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8700f51a0 msgr2=0x7ff8701064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:32:07.678 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:07.669+0000 7ff875a01700 1 --2- 172.21.15.160:0/3913231381 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8700f51a0 0x7ff8701064c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7ff864013510 tx=0x7ff86400b040).stop 2023-11-17T05:32:07.678 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:07.669+0000 7ff875a01700 1 -- 172.21.15.160:0/3913231381 shutdown_connections 2023-11-17T05:32:07.678 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:07.669+0000 7ff875a01700 1 --2- 172.21.15.160:0/3913231381 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7ff86002ff60 0x7ff860032420 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:32:07.678 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:07.669+0000 7ff875a01700 1 --2- 172.21.15.160:0/3913231381 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8700f51a0 0x7ff8701064c0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:32:07.678 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:07.669+0000 7ff875a01700 1 -- 172.21.15.160:0/3913231381 >> 172.21.15.160:0/3913231381 conn(0x7ff8700f09a0 msgr2=0x7ff8700f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:32:07.678 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:07.669+0000 7ff875a01700 1 -- 172.21.15.160:0/3913231381 shutdown_connections 2023-11-17T05:32:07.678 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:07.669+0000 7ff875a01700 1 -- 172.21.15.160:0/3913231381 wait complete. 2023-11-17T05:32:07.679 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:32:08.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:08 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:32:07.453586+0000 mgr.smithi003.eddcuc (mgr.14164) 206 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:32:08.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:08 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:32:07.668389+0000 mon.smithi003 (mon.0) 214 : audit [DBG] from='client.? 172.21.15.160:0/3913231381' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:32:09.058 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:32:09.058 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:32:10.240 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:10.238+0000 7f3f7d39d700 1 -- 172.21.15.160:0/614504366 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f780f62a0 msgr2=0x7f3f780f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:32:10.240 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:10.238+0000 7f3f7d39d700 1 --2- 172.21.15.160:0/614504366 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f780f62a0 0x7f3f780f66c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f3f68009fb0 tx=0x7f3f68009b30).stop 2023-11-17T05:32:10.240 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:10.238+0000 7f3f7d39d700 1 -- 172.21.15.160:0/614504366 shutdown_connections 2023-11-17T05:32:10.240 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:10.238+0000 7f3f7d39d700 1 --2- 172.21.15.160:0/614504366 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f780f62a0 0x7f3f780f66c0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:32:10.241 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:10.238+0000 7f3f7d39d700 1 -- 172.21.15.160:0/614504366 >> 172.21.15.160:0/614504366 conn(0x7f3f780f1aa0 msgr2=0x7f3f780f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:32:10.241 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:10.238+0000 7f3f7d39d700 1 -- 172.21.15.160:0/614504366 shutdown_connections 2023-11-17T05:32:10.241 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:10.238+0000 7f3f7d39d700 1 -- 172.21.15.160:0/614504366 wait complete. 2023-11-17T05:32:10.241 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:10.239+0000 7f3f7d39d700 1 Processor -- start 2023-11-17T05:32:10.241 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:10.239+0000 7f3f7d39d700 1 -- start start 2023-11-17T05:32:10.241 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:10.239+0000 7f3f7d39d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f780f62a0 0x7f3f781064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:32:10.241 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:10.239+0000 7f3f7d39d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f781069a0 con 0x7f3f780f62a0 2023-11-17T05:32:10.241 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:10.239+0000 7f3f77fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f780f62a0 0x7f3f781064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:32:10.241 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:10.240+0000 7f3f77fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f780f62a0 0x7f3f781064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:42792/0 (socket says 172.21.15.160:42792) 2023-11-17T05:32:10.241 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:10.240+0000 7f3f77fff700 1 -- 172.21.15.160:0/3519329354 learned_addr learned my addr 172.21.15.160:0/3519329354 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:32:10.243 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:10.240+0000 7f3f77fff700 1 -- 172.21.15.160:0/3519329354 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f68005040 con 0x7f3f780f62a0 2023-11-17T05:32:10.243 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:10.240+0000 7f3f77fff700 1 --2- 172.21.15.160:0/3519329354 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f780f62a0 0x7f3f781064b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f3f68013510 tx=0x7f3f6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:32:10.243 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:10.240+0000 7f3f74ff9700 1 -- 172.21.15.160:0/3519329354 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f68009320 con 0x7f3f780f62a0 2023-11-17T05:32:10.243 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:10.240+0000 7f3f7d39d700 1 -- 172.21.15.160:0/3519329354 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f78106ba0 con 0x7f3f780f62a0 2023-11-17T05:32:10.243 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:10.240+0000 7f3f74ff9700 1 -- 172.21.15.160:0/3519329354 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3f68013f00 con 0x7f3f780f62a0 2023-11-17T05:32:10.243 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:10.240+0000 7f3f74ff9700 1 -- 172.21.15.160:0/3519329354 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f68014dc0 con 0x7f3f780f62a0 2023-11-17T05:32:10.243 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:10.240+0000 7f3f7d39d700 1 -- 172.21.15.160:0/3519329354 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f78107030 con 0x7f3f780f62a0 2023-11-17T05:32:10.243 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:10.241+0000 7f3f74ff9700 1 -- 172.21.15.160:0/3519329354 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3f680095f0 con 0x7f3f780f62a0 2023-11-17T05:32:10.243 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:10.241+0000 7f3f7d39d700 1 -- 172.21.15.160:0/3519329354 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f78045ad0 con 0x7f3f780f62a0 2023-11-17T05:32:10.243 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:10.241+0000 7f3f74ff9700 1 --2- 172.21.15.160:0/3519329354 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f3f6002ff60 0x7f3f60032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:32:10.244 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:10.241+0000 7f3f74ff9700 1 -- 172.21.15.160:0/3519329354 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3f680410c0 con 0x7f3f780f62a0 2023-11-17T05:32:10.244 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:10.241+0000 7f3f777fe700 1 --2- 172.21.15.160:0/3519329354 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f3f6002ff60 0x7f3f60032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:32:10.244 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:10.242+0000 7f3f777fe700 1 --2- 172.21.15.160:0/3519329354 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f3f6002ff60 0x7f3f60032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f3f6c0096a0 tx=0x7f3f6c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:32:10.246 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:10.246+0000 7f3f74ff9700 1 -- 172.21.15.160:0/3519329354 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3f68014420 con 0x7f3f780f62a0 2023-11-17T05:32:10.271 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:09 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:32:09.453859+0000 mgr.smithi003.eddcuc (mgr.14164) 207 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:32:10.454 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:10.451+0000 7f3f7d39d700 1 -- 172.21.15.160:0/3519329354 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3f78107ee0 con 0x7f3f780f62a0 2023-11-17T05:32:10.454 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:10.452+0000 7f3f74ff9700 1 -- 172.21.15.160:0/3519329354 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f3f68010230 con 0x7f3f780f62a0 2023-11-17T05:32:10.455 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:32:10.456 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:32:10.457 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:10.453+0000 7f3f7d39d700 1 -- 172.21.15.160:0/3519329354 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f3f6002ff60 msgr2=0x7f3f60032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:32:10.457 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:10.453+0000 7f3f7d39d700 1 --2- 172.21.15.160:0/3519329354 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f3f6002ff60 0x7f3f60032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f3f6c0096a0 tx=0x7f3f6c006b40).stop 2023-11-17T05:32:10.457 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:10.453+0000 7f3f7d39d700 1 -- 172.21.15.160:0/3519329354 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f780f62a0 msgr2=0x7f3f781064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:32:10.458 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:10.453+0000 7f3f7d39d700 1 --2- 172.21.15.160:0/3519329354 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f780f62a0 0x7f3f781064b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f3f68013510 tx=0x7f3f6800b040).stop 2023-11-17T05:32:10.458 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:10.453+0000 7f3f7d39d700 1 -- 172.21.15.160:0/3519329354 shutdown_connections 2023-11-17T05:32:10.458 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:10.453+0000 7f3f7d39d700 1 --2- 172.21.15.160:0/3519329354 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f3f6002ff60 0x7f3f60032420 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:32:10.459 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:10.453+0000 7f3f7d39d700 1 --2- 172.21.15.160:0/3519329354 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f780f62a0 0x7f3f781064b0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:32:10.459 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:10.453+0000 7f3f7d39d700 1 -- 172.21.15.160:0/3519329354 >> 172.21.15.160:0/3519329354 conn(0x7f3f780f1aa0 msgr2=0x7f3f780f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:32:10.459 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:10.453+0000 7f3f7d39d700 1 -- 172.21.15.160:0/3519329354 shutdown_connections 2023-11-17T05:32:10.459 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:10.453+0000 7f3f7d39d700 1 -- 172.21.15.160:0/3519329354 wait complete. 2023-11-17T05:32:10.460 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:32:11.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:10 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:32:10.452855+0000 mon.smithi003 (mon.0) 215 : audit [DBG] from='client.? 172.21.15.160:0/3519329354' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:32:11.836 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:32:11.836 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:32:12.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:11 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:32:11.454206+0000 mgr.smithi003.eddcuc (mgr.14164) 208 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:32:13.088 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:13.087+0000 7f046ee92700 1 -- 172.21.15.160:0/535640909 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04680f62a0 msgr2=0x7f04680f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:32:13.088 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:13.087+0000 7f046ee92700 1 --2- 172.21.15.160:0/535640909 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04680f62a0 0x7f04680f66c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f0458009fb0 tx=0x7f0458009b30).stop 2023-11-17T05:32:13.088 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:13.087+0000 7f046ee92700 1 -- 172.21.15.160:0/535640909 shutdown_connections 2023-11-17T05:32:13.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:13.087+0000 7f046ee92700 1 --2- 172.21.15.160:0/535640909 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04680f62a0 0x7f04680f66c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:32:13.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:13.087+0000 7f046ee92700 1 -- 172.21.15.160:0/535640909 >> 172.21.15.160:0/535640909 conn(0x7f04680f1aa0 msgr2=0x7f04680f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:32:13.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:13.087+0000 7f046ee92700 1 -- 172.21.15.160:0/535640909 shutdown_connections 2023-11-17T05:32:13.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:13.087+0000 7f046ee92700 1 -- 172.21.15.160:0/535640909 wait complete. 2023-11-17T05:32:13.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:13.087+0000 7f046ee92700 1 Processor -- start 2023-11-17T05:32:13.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:13.087+0000 7f046ee92700 1 -- start start 2023-11-17T05:32:13.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:13.087+0000 7f046ee92700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04680f62a0 0x7f04681064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:32:13.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:13.087+0000 7f046ee92700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04681069a0 con 0x7f04680f62a0 2023-11-17T05:32:13.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:13.088+0000 7f046de90700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04680f62a0 0x7f04681064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:32:13.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:13.088+0000 7f046de90700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04680f62a0 0x7f04681064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:58962/0 (socket says 172.21.15.160:58962) 2023-11-17T05:32:13.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:13.088+0000 7f046de90700 1 -- 172.21.15.160:0/197898222 learned_addr learned my addr 172.21.15.160:0/197898222 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:32:13.091 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:13.088+0000 7f046de90700 1 -- 172.21.15.160:0/197898222 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0458005040 con 0x7f04680f62a0 2023-11-17T05:32:13.091 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:13.088+0000 7f046de90700 1 --2- 172.21.15.160:0/197898222 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04680f62a0 0x7f04681064b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f0458013510 tx=0x7f045800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:32:13.091 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:13.088+0000 7f045e7fc700 1 -- 172.21.15.160:0/197898222 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0458009320 con 0x7f04680f62a0 2023-11-17T05:32:13.091 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:13.088+0000 7f046ee92700 1 -- 172.21.15.160:0/197898222 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0468106ba0 con 0x7f04680f62a0 2023-11-17T05:32:13.091 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:13.088+0000 7f045e7fc700 1 -- 172.21.15.160:0/197898222 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0458013f00 con 0x7f04680f62a0 2023-11-17T05:32:13.091 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:13.088+0000 7f045e7fc700 1 -- 172.21.15.160:0/197898222 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0458014dc0 con 0x7f04680f62a0 2023-11-17T05:32:13.091 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:13.088+0000 7f046ee92700 1 -- 172.21.15.160:0/197898222 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0468107030 con 0x7f04680f62a0 2023-11-17T05:32:13.091 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:13.089+0000 7f045e7fc700 1 -- 172.21.15.160:0/197898222 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0458014420 con 0x7f04680f62a0 2023-11-17T05:32:13.091 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:13.089+0000 7f046ee92700 1 -- 172.21.15.160:0/197898222 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0468045ad0 con 0x7f04680f62a0 2023-11-17T05:32:13.091 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:13.089+0000 7f045e7fc700 1 --2- 172.21.15.160:0/197898222 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f045402ff60 0x7f0454032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:32:13.091 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:13.089+0000 7f045e7fc700 1 -- 172.21.15.160:0/197898222 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f04580410f0 con 0x7f04680f62a0 2023-11-17T05:32:13.091 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:13.089+0000 7f046d68f700 1 --2- 172.21.15.160:0/197898222 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f045402ff60 0x7f0454032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:32:13.092 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:13.090+0000 7f046d68f700 1 --2- 172.21.15.160:0/197898222 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f045402ff60 0x7f0454032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f04640096a0 tx=0x7f0464006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:32:13.094 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:13.094+0000 7f045e7fc700 1 -- 172.21.15.160:0/197898222 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f04580095f0 con 0x7f04680f62a0 2023-11-17T05:32:13.299 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:13.298+0000 7f046ee92700 1 -- 172.21.15.160:0/197898222 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0468107ee0 con 0x7f04680f62a0 2023-11-17T05:32:13.300 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:13.299+0000 7f045e7fc700 1 -- 172.21.15.160:0/197898222 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f045801cb10 con 0x7f04680f62a0 2023-11-17T05:32:13.301 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:32:13.312 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:32:13.313 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:13.301+0000 7f046ee92700 1 -- 172.21.15.160:0/197898222 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f045402ff60 msgr2=0x7f0454032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:32:13.313 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:13.301+0000 7f046ee92700 1 --2- 172.21.15.160:0/197898222 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f045402ff60 0x7f0454032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f04640096a0 tx=0x7f0464006b40).stop 2023-11-17T05:32:13.313 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:13.301+0000 7f046ee92700 1 -- 172.21.15.160:0/197898222 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04680f62a0 msgr2=0x7f04681064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:32:13.313 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:13.301+0000 7f046ee92700 1 --2- 172.21.15.160:0/197898222 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04680f62a0 0x7f04681064b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f0458013510 tx=0x7f045800b040).stop 2023-11-17T05:32:13.313 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:13.301+0000 7f046ee92700 1 -- 172.21.15.160:0/197898222 shutdown_connections 2023-11-17T05:32:13.314 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:13.301+0000 7f046ee92700 1 --2- 172.21.15.160:0/197898222 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f045402ff60 0x7f0454032420 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:32:13.314 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:13.301+0000 7f046ee92700 1 --2- 172.21.15.160:0/197898222 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04680f62a0 0x7f04681064b0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:32:13.314 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:13.301+0000 7f046ee92700 1 -- 172.21.15.160:0/197898222 >> 172.21.15.160:0/197898222 conn(0x7f04680f1aa0 msgr2=0x7f04680f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:32:13.314 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:13.301+0000 7f046ee92700 1 -- 172.21.15.160:0/197898222 shutdown_connections 2023-11-17T05:32:13.314 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:13.301+0000 7f046ee92700 1 -- 172.21.15.160:0/197898222 wait complete. 2023-11-17T05:32:13.314 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:32:13.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:13 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:32:13.300236+0000 mon.smithi003 (mon.0) 216 : audit [DBG] from='client.? 172.21.15.160:0/197898222' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:32:14.709 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:32:14.710 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:32:14.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:14 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:32:13.454530+0000 mgr.smithi003.eddcuc (mgr.14164) 209 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:32:15.846 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:15.844+0000 7f25b2d0d700 1 -- 172.21.15.160:0/3188879963 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25ac0f5410 msgr2=0x7f25ac0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:32:15.846 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:15.844+0000 7f25b2d0d700 1 --2- 172.21.15.160:0/3188879963 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25ac0f5410 0x7f25ac0f5830 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f25a0009fb0 tx=0x7f25a0009b30).stop 2023-11-17T05:32:15.846 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:15.844+0000 7f25b2d0d700 1 -- 172.21.15.160:0/3188879963 shutdown_connections 2023-11-17T05:32:15.846 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:15.844+0000 7f25b2d0d700 1 --2- 172.21.15.160:0/3188879963 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25ac0f5410 0x7f25ac0f5830 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:32:15.846 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:15.844+0000 7f25b2d0d700 1 -- 172.21.15.160:0/3188879963 >> 172.21.15.160:0/3188879963 conn(0x7f25ac0f0b70 msgr2=0x7f25ac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:32:15.846 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:15.844+0000 7f25b2d0d700 1 -- 172.21.15.160:0/3188879963 shutdown_connections 2023-11-17T05:32:15.846 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:15.844+0000 7f25b2d0d700 1 -- 172.21.15.160:0/3188879963 wait complete. 2023-11-17T05:32:15.846 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:15.845+0000 7f25b2d0d700 1 Processor -- start 2023-11-17T05:32:15.846 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:15.845+0000 7f25b2d0d700 1 -- start start 2023-11-17T05:32:15.846 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:15.845+0000 7f25b2d0d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25ac0f5410 0x7f25ac105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:32:15.846 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:15.845+0000 7f25b2d0d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac105b20 con 0x7f25ac0f5410 2023-11-17T05:32:15.848 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:15.845+0000 7f25b0aa9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25ac0f5410 0x7f25ac105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:32:15.848 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:15.845+0000 7f25b0aa9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25ac0f5410 0x7f25ac105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:58976/0 (socket says 172.21.15.160:58976) 2023-11-17T05:32:15.848 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:15.845+0000 7f25b0aa9700 1 -- 172.21.15.160:0/3853048417 learned_addr learned my addr 172.21.15.160:0/3853048417 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:32:15.848 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:15.845+0000 7f25b0aa9700 1 -- 172.21.15.160:0/3853048417 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a0005040 con 0x7f25ac0f5410 2023-11-17T05:32:15.848 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:15.846+0000 7f25b0aa9700 1 --2- 172.21.15.160:0/3853048417 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25ac0f5410 0x7f25ac105630 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f25a0013510 tx=0x7f25a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:32:15.848 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:15.846+0000 7f25a97fa700 1 -- 172.21.15.160:0/3853048417 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25a0009320 con 0x7f25ac0f5410 2023-11-17T05:32:15.848 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:15.846+0000 7f25b2d0d700 1 -- 172.21.15.160:0/3853048417 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25ac105d20 con 0x7f25ac0f5410 2023-11-17T05:32:15.848 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:15.846+0000 7f25a97fa700 1 -- 172.21.15.160:0/3853048417 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f25a0013f00 con 0x7f25ac0f5410 2023-11-17T05:32:15.849 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:15.846+0000 7f25a97fa700 1 -- 172.21.15.160:0/3853048417 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25a0014d90 con 0x7f25ac0f5410 2023-11-17T05:32:15.849 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:15.846+0000 7f25b2d0d700 1 -- 172.21.15.160:0/3853048417 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25ac1061b0 con 0x7f25ac0f5410 2023-11-17T05:32:15.849 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:15.847+0000 7f25a97fa700 1 -- 172.21.15.160:0/3853048417 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f25a0014420 con 0x7f25ac0f5410 2023-11-17T05:32:15.849 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:15.847+0000 7f25b2d0d700 1 -- 172.21.15.160:0/3853048417 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25ac045ad0 con 0x7f25ac0f5410 2023-11-17T05:32:15.849 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:15.847+0000 7f25a97fa700 1 --2- 172.21.15.160:0/3853048417 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f259c02ff60 0x7f259c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:32:15.849 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:15.847+0000 7f25a97fa700 1 -- 172.21.15.160:0/3853048417 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f25a0041050 con 0x7f25ac0f5410 2023-11-17T05:32:15.849 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:15.847+0000 7f25abfff700 1 --2- 172.21.15.160:0/3853048417 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f259c02ff60 0x7f259c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:32:15.849 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:15.848+0000 7f25abfff700 1 --2- 172.21.15.160:0/3853048417 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f259c02ff60 0x7f259c032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f25980096a0 tx=0x7f2598006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:32:15.853 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:15.852+0000 7f25a97fa700 1 -- 172.21.15.160:0/3853048417 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f25a00095f0 con 0x7f25ac0f5410 2023-11-17T05:32:16.059 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:16.058+0000 7f25b2d0d700 1 -- 172.21.15.160:0/3853048417 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f25ac0fae60 con 0x7f25ac0f5410 2023-11-17T05:32:16.060 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:16.059+0000 7f25a97fa700 1 -- 172.21.15.160:0/3853048417 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f25a001db70 con 0x7f25ac0f5410 2023-11-17T05:32:16.060 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:32:16.060 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:32:16.061 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:16.060+0000 7f25b2d0d700 1 -- 172.21.15.160:0/3853048417 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f259c02ff60 msgr2=0x7f259c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:32:16.061 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:16.060+0000 7f25b2d0d700 1 --2- 172.21.15.160:0/3853048417 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f259c02ff60 0x7f259c032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f25980096a0 tx=0x7f2598006b40).stop 2023-11-17T05:32:16.062 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:16.060+0000 7f25b2d0d700 1 -- 172.21.15.160:0/3853048417 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25ac0f5410 msgr2=0x7f25ac105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:32:16.062 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:16.060+0000 7f25b2d0d700 1 --2- 172.21.15.160:0/3853048417 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25ac0f5410 0x7f25ac105630 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f25a0013510 tx=0x7f25a000b040).stop 2023-11-17T05:32:16.062 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:16.061+0000 7f25b2d0d700 1 -- 172.21.15.160:0/3853048417 shutdown_connections 2023-11-17T05:32:16.062 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:16.061+0000 7f25b2d0d700 1 --2- 172.21.15.160:0/3853048417 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f259c02ff60 0x7f259c032420 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:32:16.062 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:16.061+0000 7f25b2d0d700 1 --2- 172.21.15.160:0/3853048417 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25ac0f5410 0x7f25ac105630 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:32:16.062 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:16.061+0000 7f25b2d0d700 1 -- 172.21.15.160:0/3853048417 >> 172.21.15.160:0/3853048417 conn(0x7f25ac0f0b70 msgr2=0x7f25ac0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:32:16.062 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:16.061+0000 7f25b2d0d700 1 -- 172.21.15.160:0/3853048417 shutdown_connections 2023-11-17T05:32:16.062 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:16.061+0000 7f25b2d0d700 1 -- 172.21.15.160:0/3853048417 wait complete. 2023-11-17T05:32:16.062 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:32:16.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:16 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:32:15.454787+0000 mgr.smithi003.eddcuc (mgr.14164) 210 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:32:16.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:16 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:32:16.060190+0000 mon.smithi003 (mon.0) 217 : audit [DBG] from='client.? 172.21.15.160:0/3853048417' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:32:17.455 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:32:17.455 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:32:18.623 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:18.621+0000 7f1703ce6700 1 -- 172.21.15.160:0/3538294225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16fc0f5410 msgr2=0x7f16fc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:32:18.623 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:18.621+0000 7f1703ce6700 1 --2- 172.21.15.160:0/3538294225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16fc0f5410 0x7f16fc0f5830 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f16ec009fb0 tx=0x7f16ec009b30).stop 2023-11-17T05:32:18.623 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:18.621+0000 7f1703ce6700 1 -- 172.21.15.160:0/3538294225 shutdown_connections 2023-11-17T05:32:18.623 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:18.621+0000 7f1703ce6700 1 --2- 172.21.15.160:0/3538294225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16fc0f5410 0x7f16fc0f5830 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:32:18.623 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:18.621+0000 7f1703ce6700 1 -- 172.21.15.160:0/3538294225 >> 172.21.15.160:0/3538294225 conn(0x7f16fc0f0b70 msgr2=0x7f16fc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:32:18.623 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:18.621+0000 7f1703ce6700 1 -- 172.21.15.160:0/3538294225 shutdown_connections 2023-11-17T05:32:18.623 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:18.621+0000 7f1703ce6700 1 -- 172.21.15.160:0/3538294225 wait complete. 2023-11-17T05:32:18.623 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:18.622+0000 7f1703ce6700 1 Processor -- start 2023-11-17T05:32:18.624 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:18.622+0000 7f1703ce6700 1 -- start start 2023-11-17T05:32:18.624 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:18.622+0000 7f1703ce6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16fc0f5410 0x7f16fc1088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:32:18.624 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:18.622+0000 7f1703ce6700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16fc108dc0 con 0x7f16fc0f5410 2023-11-17T05:32:18.624 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:18.622+0000 7f1701a82700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16fc0f5410 0x7f16fc1088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:32:18.624 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:18.622+0000 7f1701a82700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16fc0f5410 0x7f16fc1088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:59008/0 (socket says 172.21.15.160:59008) 2023-11-17T05:32:18.624 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:18.622+0000 7f1701a82700 1 -- 172.21.15.160:0/1840625016 learned_addr learned my addr 172.21.15.160:0/1840625016 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:32:18.624 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:18.622+0000 7f1701a82700 1 -- 172.21.15.160:0/1840625016 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16ec005040 con 0x7f16fc0f5410 2023-11-17T05:32:18.625 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:18.622+0000 7f1701a82700 1 --2- 172.21.15.160:0/1840625016 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16fc0f5410 0x7f16fc1088d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f16ec013510 tx=0x7f16ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:32:18.625 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:18.623+0000 7f16f27fc700 1 -- 172.21.15.160:0/1840625016 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16ec009320 con 0x7f16fc0f5410 2023-11-17T05:32:18.625 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:18.623+0000 7f1703ce6700 1 -- 172.21.15.160:0/1840625016 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16fc108fc0 con 0x7f16fc0f5410 2023-11-17T05:32:18.625 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:18.623+0000 7f16f27fc700 1 -- 172.21.15.160:0/1840625016 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f16ec013f00 con 0x7f16fc0f5410 2023-11-17T05:32:18.625 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:18.623+0000 7f16f27fc700 1 -- 172.21.15.160:0/1840625016 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16ec014d90 con 0x7f16fc0f5410 2023-11-17T05:32:18.626 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:18.623+0000 7f1703ce6700 1 -- 172.21.15.160:0/1840625016 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16fc109450 con 0x7f16fc0f5410 2023-11-17T05:32:18.626 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:18.623+0000 7f1703ce6700 1 -- 172.21.15.160:0/1840625016 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16fc045ad0 con 0x7f16fc0f5410 2023-11-17T05:32:18.626 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:18.623+0000 7f16f27fc700 1 -- 172.21.15.160:0/1840625016 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f16ec0095f0 con 0x7f16fc0f5410 2023-11-17T05:32:18.626 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:18.624+0000 7f16f27fc700 1 --2- 172.21.15.160:0/1840625016 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f16e802ff60 0x7f16e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:32:18.626 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:18.624+0000 7f16f27fc700 1 -- 172.21.15.160:0/1840625016 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f16ec041090 con 0x7f16fc0f5410 2023-11-17T05:32:18.626 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:18.624+0000 7f1701281700 1 --2- 172.21.15.160:0/1840625016 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f16e802ff60 0x7f16e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:32:18.626 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:18.624+0000 7f1701281700 1 --2- 172.21.15.160:0/1840625016 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f16e802ff60 0x7f16e8032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f16f80096a0 tx=0x7f16f8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:32:18.629 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:18.628+0000 7f16f27fc700 1 -- 172.21.15.160:0/1840625016 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f16ec014420 con 0x7f16fc0f5410 2023-11-17T05:32:18.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:18 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:32:17.455034+0000 mgr.smithi003.eddcuc (mgr.14164) 211 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:32:18.834 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:18.833+0000 7f1703ce6700 1 -- 172.21.15.160:0/1840625016 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f16fc10a300 con 0x7f16fc0f5410 2023-11-17T05:32:18.835 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:18.834+0000 7f16f27fc700 1 -- 172.21.15.160:0/1840625016 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f16ec01d7f0 con 0x7f16fc0f5410 2023-11-17T05:32:18.835 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:32:18.835 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:32:18.836 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:18.835+0000 7f1703ce6700 1 -- 172.21.15.160:0/1840625016 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f16e802ff60 msgr2=0x7f16e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:32:18.837 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:18.835+0000 7f1703ce6700 1 --2- 172.21.15.160:0/1840625016 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f16e802ff60 0x7f16e8032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f16f80096a0 tx=0x7f16f8006b40).stop 2023-11-17T05:32:18.837 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:18.835+0000 7f1703ce6700 1 -- 172.21.15.160:0/1840625016 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16fc0f5410 msgr2=0x7f16fc1088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:32:18.837 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:18.835+0000 7f1703ce6700 1 --2- 172.21.15.160:0/1840625016 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16fc0f5410 0x7f16fc1088d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f16ec013510 tx=0x7f16ec00b040).stop 2023-11-17T05:32:18.837 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:18.836+0000 7f1703ce6700 1 -- 172.21.15.160:0/1840625016 shutdown_connections 2023-11-17T05:32:18.837 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:18.836+0000 7f1703ce6700 1 --2- 172.21.15.160:0/1840625016 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f16e802ff60 0x7f16e8032420 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:32:18.837 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:18.836+0000 7f1703ce6700 1 --2- 172.21.15.160:0/1840625016 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16fc0f5410 0x7f16fc1088d0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:32:18.837 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:18.836+0000 7f1703ce6700 1 -- 172.21.15.160:0/1840625016 >> 172.21.15.160:0/1840625016 conn(0x7f16fc0f0b70 msgr2=0x7f16fc05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:32:18.837 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:18.836+0000 7f1703ce6700 1 -- 172.21.15.160:0/1840625016 shutdown_connections 2023-11-17T05:32:18.837 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:18.836+0000 7f1703ce6700 1 -- 172.21.15.160:0/1840625016 wait complete. 2023-11-17T05:32:18.837 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:32:19.771 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:19 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:32:18.835124+0000 mon.smithi003 (mon.0) 218 : audit [DBG] from='client.? 172.21.15.160:0/1840625016' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:32:20.249 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:32:20.250 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:32:20.771 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:20 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:32:19.455381+0000 mgr.smithi003.eddcuc (mgr.14164) 212 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:32:21.447 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:21.445+0000 7f0b39aa0700 1 -- 172.21.15.160:0/3126393239 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b340f5410 msgr2=0x7f0b340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:32:21.447 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:21.445+0000 7f0b39aa0700 1 --2- 172.21.15.160:0/3126393239 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b340f5410 0x7f0b340f5830 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f0b28009fb0 tx=0x7f0b28009b30).stop 2023-11-17T05:32:21.447 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:21.445+0000 7f0b39aa0700 1 -- 172.21.15.160:0/3126393239 shutdown_connections 2023-11-17T05:32:21.447 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:21.445+0000 7f0b39aa0700 1 --2- 172.21.15.160:0/3126393239 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b340f5410 0x7f0b340f5830 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:32:21.447 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:21.445+0000 7f0b39aa0700 1 -- 172.21.15.160:0/3126393239 >> 172.21.15.160:0/3126393239 conn(0x7f0b340f0b70 msgr2=0x7f0b340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:32:21.447 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:21.445+0000 7f0b39aa0700 1 -- 172.21.15.160:0/3126393239 shutdown_connections 2023-11-17T05:32:21.447 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:21.445+0000 7f0b39aa0700 1 -- 172.21.15.160:0/3126393239 wait complete. 2023-11-17T05:32:21.447 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:21.446+0000 7f0b39aa0700 1 Processor -- start 2023-11-17T05:32:21.449 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:21.446+0000 7f0b39aa0700 1 -- start start 2023-11-17T05:32:21.449 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:21.446+0000 7f0b39aa0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b340f5410 0x7f0b34105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:32:21.449 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:21.446+0000 7f0b39aa0700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b34105b20 con 0x7f0b340f5410 2023-11-17T05:32:21.449 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:21.447+0000 7f0b337fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b340f5410 0x7f0b34105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:32:21.449 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:21.447+0000 7f0b337fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b340f5410 0x7f0b34105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:39756/0 (socket says 172.21.15.160:39756) 2023-11-17T05:32:21.449 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:21.447+0000 7f0b337fe700 1 -- 172.21.15.160:0/3523048355 learned_addr learned my addr 172.21.15.160:0/3523048355 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:32:21.449 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:21.447+0000 7f0b337fe700 1 -- 172.21.15.160:0/3523048355 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b28005040 con 0x7f0b340f5410 2023-11-17T05:32:21.449 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:21.447+0000 7f0b337fe700 1 --2- 172.21.15.160:0/3523048355 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b340f5410 0x7f0b34105630 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f0b28013510 tx=0x7f0b2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:32:21.449 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:21.447+0000 7f0b1bfff700 1 -- 172.21.15.160:0/3523048355 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b28009320 con 0x7f0b340f5410 2023-11-17T05:32:21.449 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:21.448+0000 7f0b39aa0700 1 -- 172.21.15.160:0/3523048355 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b34105d20 con 0x7f0b340f5410 2023-11-17T05:32:21.450 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:21.448+0000 7f0b1bfff700 1 -- 172.21.15.160:0/3523048355 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b28013f00 con 0x7f0b340f5410 2023-11-17T05:32:21.450 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:21.448+0000 7f0b1bfff700 1 -- 172.21.15.160:0/3523048355 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b28014d90 con 0x7f0b340f5410 2023-11-17T05:32:21.450 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:21.448+0000 7f0b39aa0700 1 -- 172.21.15.160:0/3523048355 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b341061b0 con 0x7f0b340f5410 2023-11-17T05:32:21.451 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:21.448+0000 7f0b1bfff700 1 -- 172.21.15.160:0/3523048355 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0b28014420 con 0x7f0b340f5410 2023-11-17T05:32:21.451 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:21.449+0000 7f0b39aa0700 1 -- 172.21.15.160:0/3523048355 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b34045ad0 con 0x7f0b340f5410 2023-11-17T05:32:21.451 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:21.449+0000 7f0b1bfff700 1 --2- 172.21.15.160:0/3523048355 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f0b1c02ff60 0x7f0b1c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:32:21.451 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:21.449+0000 7f0b1bfff700 1 -- 172.21.15.160:0/3523048355 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0b28040fb0 con 0x7f0b340f5410 2023-11-17T05:32:21.451 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:21.449+0000 7f0b32ffd700 1 --2- 172.21.15.160:0/3523048355 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f0b1c02ff60 0x7f0b1c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:32:21.451 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:21.449+0000 7f0b32ffd700 1 --2- 172.21.15.160:0/3523048355 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f0b1c02ff60 0x7f0b1c032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f0b240096a0 tx=0x7f0b24006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:32:21.455 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:21.454+0000 7f0b1bfff700 1 -- 172.21.15.160:0/3523048355 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0b280095f0 con 0x7f0b340f5410 2023-11-17T05:32:21.660 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:21.659+0000 7f0b39aa0700 1 -- 172.21.15.160:0/3523048355 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0b340fae60 con 0x7f0b340f5410 2023-11-17T05:32:21.661 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:21.660+0000 7f0b1bfff700 1 -- 172.21.15.160:0/3523048355 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f0b2801db70 con 0x7f0b340f5410 2023-11-17T05:32:21.661 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:32:21.661 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:32:21.663 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:21.662+0000 7f0b39aa0700 1 -- 172.21.15.160:0/3523048355 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f0b1c02ff60 msgr2=0x7f0b1c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:32:21.663 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:21.662+0000 7f0b39aa0700 1 --2- 172.21.15.160:0/3523048355 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f0b1c02ff60 0x7f0b1c032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f0b240096a0 tx=0x7f0b24006b40).stop 2023-11-17T05:32:21.663 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:21.662+0000 7f0b39aa0700 1 -- 172.21.15.160:0/3523048355 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b340f5410 msgr2=0x7f0b34105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:32:21.664 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:21.662+0000 7f0b39aa0700 1 --2- 172.21.15.160:0/3523048355 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b340f5410 0x7f0b34105630 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f0b28013510 tx=0x7f0b2800b040).stop 2023-11-17T05:32:21.664 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:21.662+0000 7f0b39aa0700 1 -- 172.21.15.160:0/3523048355 shutdown_connections 2023-11-17T05:32:21.664 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:21.662+0000 7f0b39aa0700 1 --2- 172.21.15.160:0/3523048355 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f0b1c02ff60 0x7f0b1c032420 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:32:21.664 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:21.662+0000 7f0b39aa0700 1 --2- 172.21.15.160:0/3523048355 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0b340f5410 0x7f0b34105630 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:32:21.664 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:21.662+0000 7f0b39aa0700 1 -- 172.21.15.160:0/3523048355 >> 172.21.15.160:0/3523048355 conn(0x7f0b340f0b70 msgr2=0x7f0b340f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:32:21.665 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:21.662+0000 7f0b39aa0700 1 -- 172.21.15.160:0/3523048355 shutdown_connections 2023-11-17T05:32:21.665 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:21.662+0000 7f0b39aa0700 1 -- 172.21.15.160:0/3523048355 wait complete. 2023-11-17T05:32:21.665 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:32:22.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:22 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:32:21.455606+0000 mgr.smithi003.eddcuc (mgr.14164) 213 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:32:22.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:22 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:32:21.660720+0000 mon.smithi003 (mon.0) 219 : audit [DBG] from='client.? 172.21.15.160:0/3523048355' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:32:23.054 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:32:23.055 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:32:24.271 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:24.269+0000 7ff918806700 1 -- 172.21.15.160:0/1495397215 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9140f5410 msgr2=0x7ff9140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:32:24.271 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:24.269+0000 7ff918806700 1 --2- 172.21.15.160:0/1495397215 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9140f5410 0x7ff9140f5830 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7ff8fc009fb0 tx=0x7ff8fc009b30).stop 2023-11-17T05:32:24.271 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:24.269+0000 7ff918806700 1 -- 172.21.15.160:0/1495397215 shutdown_connections 2023-11-17T05:32:24.271 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:24.269+0000 7ff918806700 1 --2- 172.21.15.160:0/1495397215 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9140f5410 0x7ff9140f5830 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:32:24.289 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:24.269+0000 7ff918806700 1 -- 172.21.15.160:0/1495397215 >> 172.21.15.160:0/1495397215 conn(0x7ff9140f0b70 msgr2=0x7ff9140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:32:24.289 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:24.269+0000 7ff918806700 1 -- 172.21.15.160:0/1495397215 shutdown_connections 2023-11-17T05:32:24.289 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:24.269+0000 7ff918806700 1 -- 172.21.15.160:0/1495397215 wait complete. 2023-11-17T05:32:24.289 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:24.270+0000 7ff918806700 1 Processor -- start 2023-11-17T05:32:24.289 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:24.270+0000 7ff918806700 1 -- start start 2023-11-17T05:32:24.289 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:24.270+0000 7ff918806700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9140f5410 0x7ff914109420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:32:24.290 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:24.270+0000 7ff918806700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff914060220 con 0x7ff9140f5410 2023-11-17T05:32:24.290 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:24.270+0000 7ff91259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9140f5410 0x7ff914109420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:32:24.290 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:24.270+0000 7ff91259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9140f5410 0x7ff914109420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:39772/0 (socket says 172.21.15.160:39772) 2023-11-17T05:32:24.290 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:24.270+0000 7ff91259c700 1 -- 172.21.15.160:0/2015241036 learned_addr learned my addr 172.21.15.160:0/2015241036 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:32:24.290 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:24.270+0000 7ff91259c700 1 -- 172.21.15.160:0/2015241036 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8fc005040 con 0x7ff9140f5410 2023-11-17T05:32:24.290 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:24.271+0000 7ff91259c700 1 --2- 172.21.15.160:0/2015241036 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9140f5410 0x7ff914109420 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7ff8fc013510 tx=0x7ff8fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:32:24.291 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:24.271+0000 7ff90affd700 1 -- 172.21.15.160:0/2015241036 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8fc009320 con 0x7ff9140f5410 2023-11-17T05:32:24.291 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:24.271+0000 7ff918806700 1 -- 172.21.15.160:0/2015241036 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff914060420 con 0x7ff9140f5410 2023-11-17T05:32:24.292 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:24.271+0000 7ff90affd700 1 -- 172.21.15.160:0/2015241036 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff8fc013f00 con 0x7ff9140f5410 2023-11-17T05:32:24.292 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:24.271+0000 7ff90affd700 1 -- 172.21.15.160:0/2015241036 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8fc014d90 con 0x7ff9140f5410 2023-11-17T05:32:24.292 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:24.271+0000 7ff918806700 1 -- 172.21.15.160:0/2015241036 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff914109ad0 con 0x7ff9140f5410 2023-11-17T05:32:24.292 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:24.272+0000 7ff90affd700 1 -- 172.21.15.160:0/2015241036 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff8fc0263a0 con 0x7ff9140f5410 2023-11-17T05:32:24.292 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:24.272+0000 7ff90affd700 1 --2- 172.21.15.160:0/2015241036 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7ff90002ff60 0x7ff900032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:32:24.292 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:24.272+0000 7ff90affd700 1 -- 172.21.15.160:0/2015241036 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff8fc0410f0 con 0x7ff9140f5410 2023-11-17T05:32:24.292 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:24.272+0000 7ff918806700 1 -- 172.21.15.160:0/2015241036 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff914059100 con 0x7ff9140f5410 2023-11-17T05:32:24.292 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:24.272+0000 7ff911d9b700 1 --2- 172.21.15.160:0/2015241036 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7ff90002ff60 0x7ff900032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:32:24.292 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:24.272+0000 7ff911d9b700 1 --2- 172.21.15.160:0/2015241036 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7ff90002ff60 0x7ff900032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7ff9040096a0 tx=0x7ff904006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:32:24.292 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:24.277+0000 7ff90affd700 1 -- 172.21.15.160:0/2015241036 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff8fc012920 con 0x7ff9140f5410 2023-11-17T05:32:24.483 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:24.482+0000 7ff918806700 1 -- 172.21.15.160:0/2015241036 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff914045ad0 con 0x7ff9140f5410 2023-11-17T05:32:24.483 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:24.483+0000 7ff90affd700 1 -- 172.21.15.160:0/2015241036 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7ff8fc02b020 con 0x7ff9140f5410 2023-11-17T05:32:24.484 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:32:24.484 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:32:24.485 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:24.484+0000 7ff918806700 1 -- 172.21.15.160:0/2015241036 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7ff90002ff60 msgr2=0x7ff900032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:32:24.485 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:24.484+0000 7ff918806700 1 --2- 172.21.15.160:0/2015241036 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7ff90002ff60 0x7ff900032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7ff9040096a0 tx=0x7ff904006b40).stop 2023-11-17T05:32:24.485 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:24.484+0000 7ff918806700 1 -- 172.21.15.160:0/2015241036 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9140f5410 msgr2=0x7ff914109420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:32:24.486 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:24.484+0000 7ff918806700 1 --2- 172.21.15.160:0/2015241036 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9140f5410 0x7ff914109420 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7ff8fc013510 tx=0x7ff8fc00b040).stop 2023-11-17T05:32:24.486 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:24.484+0000 7ff918806700 1 -- 172.21.15.160:0/2015241036 shutdown_connections 2023-11-17T05:32:24.486 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:24.484+0000 7ff918806700 1 --2- 172.21.15.160:0/2015241036 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7ff90002ff60 0x7ff900032420 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:32:24.486 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:24.484+0000 7ff918806700 1 --2- 172.21.15.160:0/2015241036 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9140f5410 0x7ff914109420 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:32:24.486 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:24.484+0000 7ff918806700 1 -- 172.21.15.160:0/2015241036 >> 172.21.15.160:0/2015241036 conn(0x7ff9140f0b70 msgr2=0x7ff9140f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:32:24.486 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:24.484+0000 7ff918806700 1 -- 172.21.15.160:0/2015241036 shutdown_connections 2023-11-17T05:32:24.486 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:24.484+0000 7ff918806700 1 -- 172.21.15.160:0/2015241036 wait complete. 2023-11-17T05:32:24.486 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:32:24.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:32:23.455912+0000 mgr.smithi003.eddcuc (mgr.14164) 214 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:32:24.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:32:24.483957+0000 mon.smithi003 (mon.0) 220 : audit [DBG] from='client.? 172.21.15.160:0/2015241036' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:32:25.918 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:32:25.919 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:32:26.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:26 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:32:25.456286+0000 mgr.smithi003.eddcuc (mgr.14164) 215 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:32:27.088 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:27.084+0000 7fc7e7912700 1 -- 172.21.15.160:0/3908827311 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7e00f5410 msgr2=0x7fc7e00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:32:27.088 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:27.084+0000 7fc7e7912700 1 --2- 172.21.15.160:0/3908827311 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7e00f5410 0x7fc7e00f5830 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7fc7d4009fb0 tx=0x7fc7d4009b30).stop 2023-11-17T05:32:27.088 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:27.084+0000 7fc7e7912700 1 -- 172.21.15.160:0/3908827311 shutdown_connections 2023-11-17T05:32:27.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:27.084+0000 7fc7e7912700 1 --2- 172.21.15.160:0/3908827311 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7e00f5410 0x7fc7e00f5830 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:32:27.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:27.084+0000 7fc7e7912700 1 -- 172.21.15.160:0/3908827311 >> 172.21.15.160:0/3908827311 conn(0x7fc7e00f0b70 msgr2=0x7fc7e00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:32:27.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:27.085+0000 7fc7e7912700 1 -- 172.21.15.160:0/3908827311 shutdown_connections 2023-11-17T05:32:27.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:27.085+0000 7fc7e7912700 1 -- 172.21.15.160:0/3908827311 wait complete. 2023-11-17T05:32:27.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:27.085+0000 7fc7e7912700 1 Processor -- start 2023-11-17T05:32:27.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:27.085+0000 7fc7e7912700 1 -- start start 2023-11-17T05:32:27.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:27.085+0000 7fc7e7912700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7e00f5410 0x7fc7e0109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:32:27.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:27.085+0000 7fc7e7912700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e0060220 con 0x7fc7e00f5410 2023-11-17T05:32:27.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:27.085+0000 7fc7e56ae700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7e00f5410 0x7fc7e0109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:32:27.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:27.085+0000 7fc7e56ae700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7e00f5410 0x7fc7e0109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:39796/0 (socket says 172.21.15.160:39796) 2023-11-17T05:32:27.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:27.085+0000 7fc7e56ae700 1 -- 172.21.15.160:0/868455920 learned_addr learned my addr 172.21.15.160:0/868455920 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:32:27.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:27.086+0000 7fc7e56ae700 1 -- 172.21.15.160:0/868455920 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d4005040 con 0x7fc7e00f5410 2023-11-17T05:32:27.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:27.086+0000 7fc7e56ae700 1 --2- 172.21.15.160:0/868455920 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7e00f5410 0x7fc7e0109450 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7fc7d4013510 tx=0x7fc7d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:32:27.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:27.086+0000 7fc7d1ffb700 1 -- 172.21.15.160:0/868455920 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7d4009320 con 0x7fc7e00f5410 2023-11-17T05:32:27.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:27.086+0000 7fc7e7912700 1 -- 172.21.15.160:0/868455920 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7e0060420 con 0x7fc7e00f5410 2023-11-17T05:32:27.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:27.086+0000 7fc7d1ffb700 1 -- 172.21.15.160:0/868455920 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc7d4013f00 con 0x7fc7e00f5410 2023-11-17T05:32:27.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:27.086+0000 7fc7d1ffb700 1 -- 172.21.15.160:0/868455920 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7d4014d90 con 0x7fc7e00f5410 2023-11-17T05:32:27.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:27.086+0000 7fc7e7912700 1 -- 172.21.15.160:0/868455920 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7e0109b00 con 0x7fc7e00f5410 2023-11-17T05:32:27.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:27.087+0000 7fc7e7912700 1 -- 172.21.15.160:0/868455920 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7e0059100 con 0x7fc7e00f5410 2023-11-17T05:32:27.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:27.087+0000 7fc7d1ffb700 1 -- 172.21.15.160:0/868455920 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc7d40095f0 con 0x7fc7e00f5410 2023-11-17T05:32:27.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:27.087+0000 7fc7d1ffb700 1 --2- 172.21.15.160:0/868455920 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc7cc02ff60 0x7fc7cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:32:27.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:27.087+0000 7fc7d1ffb700 1 -- 172.21.15.160:0/868455920 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc7d4041080 con 0x7fc7e00f5410 2023-11-17T05:32:27.091 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:27.087+0000 7fc7e4ead700 1 --2- 172.21.15.160:0/868455920 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc7cc02ff60 0x7fc7cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:32:27.091 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:27.088+0000 7fc7e4ead700 1 --2- 172.21.15.160:0/868455920 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc7cc02ff60 0x7fc7cc032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fc7dc0096a0 tx=0x7fc7dc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:32:27.093 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:27.093+0000 7fc7d1ffb700 1 -- 172.21.15.160:0/868455920 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc7d401d7f0 con 0x7fc7e00f5410 2023-11-17T05:32:27.299 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:27.298+0000 7fc7e7912700 1 -- 172.21.15.160:0/868455920 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc7e0045ad0 con 0x7fc7e00f5410 2023-11-17T05:32:27.300 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:27.299+0000 7fc7d1ffb700 1 -- 172.21.15.160:0/868455920 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fc7d4010230 con 0x7fc7e00f5410 2023-11-17T05:32:27.300 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:32:27.300 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:32:27.301 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:27.301+0000 7fc7e7912700 1 -- 172.21.15.160:0/868455920 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc7cc02ff60 msgr2=0x7fc7cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:32:27.302 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:27.301+0000 7fc7e7912700 1 --2- 172.21.15.160:0/868455920 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc7cc02ff60 0x7fc7cc032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fc7dc0096a0 tx=0x7fc7dc006b40).stop 2023-11-17T05:32:27.302 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:27.301+0000 7fc7e7912700 1 -- 172.21.15.160:0/868455920 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7e00f5410 msgr2=0x7fc7e0109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:32:27.302 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:27.301+0000 7fc7e7912700 1 --2- 172.21.15.160:0/868455920 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7e00f5410 0x7fc7e0109450 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7fc7d4013510 tx=0x7fc7d400b040).stop 2023-11-17T05:32:27.302 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:27.301+0000 7fc7e7912700 1 -- 172.21.15.160:0/868455920 shutdown_connections 2023-11-17T05:32:27.302 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:27.301+0000 7fc7e7912700 1 --2- 172.21.15.160:0/868455920 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc7cc02ff60 0x7fc7cc032420 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:32:27.302 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:27.301+0000 7fc7e7912700 1 --2- 172.21.15.160:0/868455920 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7e00f5410 0x7fc7e0109450 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:32:27.302 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:27.301+0000 7fc7e7912700 1 -- 172.21.15.160:0/868455920 >> 172.21.15.160:0/868455920 conn(0x7fc7e00f0b70 msgr2=0x7fc7e00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:32:27.302 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:27.301+0000 7fc7e7912700 1 -- 172.21.15.160:0/868455920 shutdown_connections 2023-11-17T05:32:27.302 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:27.301+0000 7fc7e7912700 1 -- 172.21.15.160:0/868455920 wait complete. 2023-11-17T05:32:27.303 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:32:27.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:27 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:32:27.300304+0000 mon.smithi003 (mon.0) 221 : audit [DBG] from='client.? 172.21.15.160:0/868455920' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:32:28.683 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:32:28.683 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:32:28.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:32:27.456639+0000 mgr.smithi003.eddcuc (mgr.14164) 216 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:32:28.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:32:27.552988+0000 mon.smithi003 (mon.0) 222 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.eddcuc/mirror_snapshot_schedule"}]: dispatch 2023-11-17T05:32:28.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:32:27.556710+0000 mon.smithi003 (mon.0) 223 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.eddcuc/trash_purge_schedule"}]: dispatch 2023-11-17T05:32:29.987 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:29.983+0000 7f26c9619700 1 -- 172.21.15.160:0/2025557330 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26c40f6510 msgr2=0x7f26c40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:32:29.987 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:29.983+0000 7f26c9619700 1 --2- 172.21.15.160:0/2025557330 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26c40f6510 0x7f26c40f6930 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f26b8009fb0 tx=0x7f26b8009b30).stop 2023-11-17T05:32:29.987 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:29.983+0000 7f26c9619700 1 -- 172.21.15.160:0/2025557330 shutdown_connections 2023-11-17T05:32:29.987 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:29.983+0000 7f26c9619700 1 --2- 172.21.15.160:0/2025557330 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26c40f6510 0x7f26c40f6930 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:32:29.987 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:29.983+0000 7f26c9619700 1 -- 172.21.15.160:0/2025557330 >> 172.21.15.160:0/2025557330 conn(0x7f26c40f1c70 msgr2=0x7f26c40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:32:29.988 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:29.983+0000 7f26c9619700 1 -- 172.21.15.160:0/2025557330 shutdown_connections 2023-11-17T05:32:29.988 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:29.983+0000 7f26c9619700 1 -- 172.21.15.160:0/2025557330 wait complete. 2023-11-17T05:32:29.988 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:29.984+0000 7f26c9619700 1 Processor -- start 2023-11-17T05:32:29.988 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:29.984+0000 7f26c9619700 1 -- start start 2023-11-17T05:32:29.988 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:29.984+0000 7f26c9619700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26c40f6510 0x7f26c4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:32:29.989 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:29.984+0000 7f26c9619700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c4106c20 con 0x7f26c40f6510 2023-11-17T05:32:29.989 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:29.984+0000 7f26c2ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26c40f6510 0x7f26c4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:32:29.989 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:29.984+0000 7f26c2ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26c40f6510 0x7f26c4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:39816/0 (socket says 172.21.15.160:39816) 2023-11-17T05:32:29.989 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:29.984+0000 7f26c2ffd700 1 -- 172.21.15.160:0/1588800889 learned_addr learned my addr 172.21.15.160:0/1588800889 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:32:29.989 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:29.984+0000 7f26c2ffd700 1 -- 172.21.15.160:0/1588800889 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b8005040 con 0x7f26c40f6510 2023-11-17T05:32:29.989 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:29.984+0000 7f26c2ffd700 1 --2- 172.21.15.160:0/1588800889 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26c40f6510 0x7f26c4106730 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f26b8013510 tx=0x7f26b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:32:29.989 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:29.985+0000 7f26ab7fe700 1 -- 172.21.15.160:0/1588800889 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26b8009320 con 0x7f26c40f6510 2023-11-17T05:32:29.989 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:29.985+0000 7f26c9619700 1 -- 172.21.15.160:0/1588800889 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26c4106e20 con 0x7f26c40f6510 2023-11-17T05:32:29.989 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:29.985+0000 7f26ab7fe700 1 -- 172.21.15.160:0/1588800889 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f26b8013f00 con 0x7f26c40f6510 2023-11-17T05:32:29.990 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:29.985+0000 7f26ab7fe700 1 -- 172.21.15.160:0/1588800889 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26b8014d90 con 0x7f26c40f6510 2023-11-17T05:32:29.990 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:29.985+0000 7f26c9619700 1 -- 172.21.15.160:0/1588800889 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26c41072b0 con 0x7f26c40f6510 2023-11-17T05:32:29.990 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:29.985+0000 7f26ab7fe700 1 -- 172.21.15.160:0/1588800889 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f26b8014420 con 0x7f26c40f6510 2023-11-17T05:32:29.990 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:29.985+0000 7f26c9619700 1 -- 172.21.15.160:0/1588800889 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26c4045ad0 con 0x7f26c40f6510 2023-11-17T05:32:29.990 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:29.986+0000 7f26ab7fe700 1 --2- 172.21.15.160:0/1588800889 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f26ac02ff60 0x7f26ac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:32:29.990 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:29.986+0000 7f26ab7fe700 1 -- 172.21.15.160:0/1588800889 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f26b8041070 con 0x7f26c40f6510 2023-11-17T05:32:29.990 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:29.986+0000 7f26c27fc700 1 --2- 172.21.15.160:0/1588800889 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f26ac02ff60 0x7f26ac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:32:29.990 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:29.986+0000 7f26c27fc700 1 --2- 172.21.15.160:0/1588800889 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f26ac02ff60 0x7f26ac032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f26b40096a0 tx=0x7f26b4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:32:29.990 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:29.991+0000 7f26ab7fe700 1 -- 172.21.15.160:0/1588800889 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f26b80095f0 con 0x7f26c40f6510 2023-11-17T05:32:30.194 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:30.194+0000 7f26c9619700 1 -- 172.21.15.160:0/1588800889 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f26c40fbf60 con 0x7f26c40f6510 2023-11-17T05:32:30.194 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:30.195+0000 7f26ab7fe700 1 -- 172.21.15.160:0/1588800889 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f26b801db70 con 0x7f26c40f6510 2023-11-17T05:32:30.195 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:32:30.195 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:32:30.196 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:30.196+0000 7f26c9619700 1 -- 172.21.15.160:0/1588800889 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f26ac02ff60 msgr2=0x7f26ac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:32:30.196 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:30.196+0000 7f26c9619700 1 --2- 172.21.15.160:0/1588800889 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f26ac02ff60 0x7f26ac032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f26b40096a0 tx=0x7f26b4006b40).stop 2023-11-17T05:32:30.196 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:30.196+0000 7f26c9619700 1 -- 172.21.15.160:0/1588800889 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26c40f6510 msgr2=0x7f26c4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:32:30.197 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:30.196+0000 7f26c9619700 1 --2- 172.21.15.160:0/1588800889 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26c40f6510 0x7f26c4106730 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f26b8013510 tx=0x7f26b800b040).stop 2023-11-17T05:32:30.197 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:30.196+0000 7f26c9619700 1 -- 172.21.15.160:0/1588800889 shutdown_connections 2023-11-17T05:32:30.197 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:30.196+0000 7f26c9619700 1 --2- 172.21.15.160:0/1588800889 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f26ac02ff60 0x7f26ac032420 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:32:30.197 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:30.196+0000 7f26c9619700 1 --2- 172.21.15.160:0/1588800889 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26c40f6510 0x7f26c4106730 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:32:30.197 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:30.196+0000 7f26c9619700 1 -- 172.21.15.160:0/1588800889 >> 172.21.15.160:0/1588800889 conn(0x7f26c40f1c70 msgr2=0x7f26c40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:32:30.197 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:30.196+0000 7f26c9619700 1 -- 172.21.15.160:0/1588800889 shutdown_connections 2023-11-17T05:32:30.197 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:32:30.196+0000 7f26c9619700 1 -- 172.21.15.160:0/1588800889 wait complete. 2023-11-17T05:32:30.197 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:32:30.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:29 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:32:29.456909+0000 mgr.smithi003.eddcuc (mgr.14164) 217 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:32:31.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:30 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:32:30.194769+0000 mon.smithi003 (mon.0) 224 : audit [DBG] from='client.? 172.21.15.160:0/1588800889' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:33:26.704 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:33:26.706 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:33:26.725 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:31 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:32:31.457250+0000 mgr.smithi003.eddcuc (mgr.14164) 218 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:26.725 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:34 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:32:33.457612+0000 mgr.smithi003.eddcuc (mgr.14164) 219 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:26.725 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:36 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:32:35.457999+0000 mgr.smithi003.eddcuc (mgr.14164) 220 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:26.726 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:32:37.458350+0000 mgr.smithi003.eddcuc (mgr.14164) 221 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:26.726 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:39 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:32:39.458627+0000 mgr.smithi003.eddcuc (mgr.14164) 222 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:26.726 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:42 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:32:41.458980+0000 mgr.smithi003.eddcuc (mgr.14164) 223 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:26.726 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:44 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:32:43.459388+0000 mgr.smithi003.eddcuc (mgr.14164) 224 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:26.726 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:46 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:32:45.459802+0000 mgr.smithi003.eddcuc (mgr.14164) 225 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:26.726 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:48 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:32:47.460139+0000 mgr.smithi003.eddcuc (mgr.14164) 226 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:26.726 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:49 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:32:49.460439+0000 mgr.smithi003.eddcuc (mgr.14164) 227 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:26.726 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:52 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:32:51.460729+0000 mgr.smithi003.eddcuc (mgr.14164) 228 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:26.727 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:54 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:32:53.461105+0000 mgr.smithi003.eddcuc (mgr.14164) 229 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:26.727 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:56 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:32:55.461454+0000 mgr.smithi003.eddcuc (mgr.14164) 230 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:26.727 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cephadm 2023-11-17T05:32:56.062823+0000 mgr.smithi003.eddcuc (mgr.14164) 231 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi003 ... 2023-11-17T05:33:26.727 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: Verifying port 9100 ... 2023-11-17T05:33:26.727 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: Non-zero exit code 1 from systemctl start ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7@node-exporter.smithi003 2023-11-17T05:33:26.727 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: systemctl: stderr Job for ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7@node-exporter.smithi003.service failed because a timeout was exceeded. 2023-11-17T05:33:26.727 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: systemctl: stderr See "systemctl status ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7@node-exporter.smithi003.service" and "journalctl -xe" for details. 2023-11-17T05:33:26.727 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: Traceback (most recent call last): 2023-11-17T05:33:26.727 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: File "/var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-17T05:33:26.728 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: main() 2023-11-17T05:33:26.728 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: File "/var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-17T05:33:26.728 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: r = ctx.func(ctx) 2023-11-17T05:33:26.728 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: File "/var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-17T05:33:26.789 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: return func(ctx) 2023-11-17T05:33:26.790 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: File "/var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-11-17T05:33:26.790 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: ports=daemon_ports) 2023-11-17T05:33:26.790 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: File "/var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2023-11-17T05:33:26.790 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: c, osd_fsid=osd_fsid, ports=ports) 2023-11-17T05:33:26.790 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: File "/var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2023-11-17T05:33:26.790 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-11-17T05:33:26.836 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: File "/var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-11-17T05:33:26.836 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-11-17T05:33:26.840 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: RuntimeError: Failed command: systemctl start ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7@node-exporter.smithi003 2023-11-17T05:33:26.866 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: Traceback (most recent call last): 2023-11-17T05:33:26.866 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-11-17T05:33:26.866 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: yield (conn, connr) 2023-11-17T05:33:26.866 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-11-17T05:33:26.866 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: code, '\n'.join(err))) 2023-11-17T05:33:26.866 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi003 ... 2023-11-17T05:33:26.866 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: Verifying port 9100 ... 2023-11-17T05:33:26.867 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: Non-zero exit code 1 from systemctl start ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7@node-exporter.smithi003 2023-11-17T05:33:26.867 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: systemctl: stderr Job for ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7@node-exporter.smithi003.service failed because a timeout was exceeded. 2023-11-17T05:33:26.867 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: systemctl: stderr See "systemctl status ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7@node-exporter.smithi003.service" and "journalctl -xe" for details. 2023-11-17T05:33:26.867 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: Traceback (most recent call last): 2023-11-17T05:33:26.867 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: File "/var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-17T05:33:26.867 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: main() 2023-11-17T05:33:26.867 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: File "/var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-17T05:33:26.867 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: r = ctx.func(ctx) 2023-11-17T05:33:26.867 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: File "/var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-17T05:33:26.867 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: return func(ctx) 2023-11-17T05:33:26.867 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: File "/var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-11-17T05:33:26.867 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: ports=daemon_ports) 2023-11-17T05:33:26.867 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: File "/var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2023-11-17T05:33:26.869 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: c, osd_fsid=osd_fsid, ports=ports) 2023-11-17T05:33:26.869 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: File "/var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2023-11-17T05:33:26.869 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-11-17T05:33:26.869 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: File "/var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-11-17T05:33:26.869 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-11-17T05:33:26.869 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: RuntimeError: Failed command: systemctl start ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7@node-exporter.smithi003 2023-11-17T05:33:26.870 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cephadm 2023-11-17T05:32:56.068702+0000 mgr.smithi003.eddcuc (mgr.14164) 232 : cephadm [INF] Deploying daemon prometheus.smithi003 on smithi003 2023-11-17T05:33:26.870 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:58 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:32:57.461781+0000 mgr.smithi003.eddcuc (mgr.14164) 233 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:26.870 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:32:59 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:32:59.462093+0000 mgr.smithi003.eddcuc (mgr.14164) 234 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:26.870 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:33:02 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:33:01.462458+0000 mgr.smithi003.eddcuc (mgr.14164) 235 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:26.870 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:33:04 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:33:03.462788+0000 mgr.smithi003.eddcuc (mgr.14164) 236 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:26.870 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:33:06 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:33:05.463068+0000 mgr.smithi003.eddcuc (mgr.14164) 237 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:26.870 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:33:08 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:33:07.463331+0000 mgr.smithi003.eddcuc (mgr.14164) 238 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:26.870 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:33:09 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:33:09.463683+0000 mgr.smithi003.eddcuc (mgr.14164) 239 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:26.870 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:33:12 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:33:11.463986+0000 mgr.smithi003.eddcuc (mgr.14164) 240 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:26.870 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:33:14 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:33:13.464235+0000 mgr.smithi003.eddcuc (mgr.14164) 241 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:26.871 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:33:16 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:33:15.464550+0000 mgr.smithi003.eddcuc (mgr.14164) 242 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:26.871 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:33:18 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:33:17.464872+0000 mgr.smithi003.eddcuc (mgr.14164) 243 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:26.871 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:33:19 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:33:19.465231+0000 mgr.smithi003.eddcuc (mgr.14164) 244 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:26.871 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:33:22 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:33:21.465575+0000 mgr.smithi003.eddcuc (mgr.14164) 245 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:26.871 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:33:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:33:23.465927+0000 mgr.smithi003.eddcuc (mgr.14164) 246 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:27.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:33:26 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:33:25.466308+0000 mgr.smithi003.eddcuc (mgr.14164) 247 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:27.962 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:27.960+0000 7fcb20e5a700 1 -- 172.21.15.160:0/841529798 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb1c0f6510 msgr2=0x7fcb1c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:33:27.962 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:27.960+0000 7fcb20e5a700 1 --2- 172.21.15.160:0/841529798 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb1c0f6510 0x7fcb1c0f6930 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7fcb04009fb0 tx=0x7fcb04009b30).stop 2023-11-17T05:33:27.962 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:27.960+0000 7fcb20e5a700 1 -- 172.21.15.160:0/841529798 shutdown_connections 2023-11-17T05:33:27.962 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:27.960+0000 7fcb20e5a700 1 --2- 172.21.15.160:0/841529798 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb1c0f6510 0x7fcb1c0f6930 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:33:27.962 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:27.960+0000 7fcb20e5a700 1 -- 172.21.15.160:0/841529798 >> 172.21.15.160:0/841529798 conn(0x7fcb1c0f1c70 msgr2=0x7fcb1c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:33:27.962 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:27.960+0000 7fcb20e5a700 1 -- 172.21.15.160:0/841529798 shutdown_connections 2023-11-17T05:33:27.962 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:27.960+0000 7fcb20e5a700 1 -- 172.21.15.160:0/841529798 wait complete. 2023-11-17T05:33:27.962 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:27.961+0000 7fcb20e5a700 1 Processor -- start 2023-11-17T05:33:27.962 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:27.961+0000 7fcb20e5a700 1 -- start start 2023-11-17T05:33:27.963 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:27.961+0000 7fcb20e5a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb1c0f6510 0x7fcb1c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:33:27.963 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:27.961+0000 7fcb20e5a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1c106c20 con 0x7fcb1c0f6510 2023-11-17T05:33:27.963 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:27.961+0000 7fcb1a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb1c0f6510 0x7fcb1c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:33:27.963 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:27.961+0000 7fcb1a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb1c0f6510 0x7fcb1c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:57832/0 (socket says 172.21.15.160:57832) 2023-11-17T05:33:27.963 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:27.961+0000 7fcb1a59c700 1 -- 172.21.15.160:0/476181916 learned_addr learned my addr 172.21.15.160:0/476181916 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:33:27.964 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:27.961+0000 7fcb1a59c700 1 -- 172.21.15.160:0/476181916 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb04005040 con 0x7fcb1c0f6510 2023-11-17T05:33:27.964 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:27.961+0000 7fcb1a59c700 1 --2- 172.21.15.160:0/476181916 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb1c0f6510 0x7fcb1c106730 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fcb04013510 tx=0x7fcb0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:33:27.964 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:27.962+0000 7fcb12ffd700 1 -- 172.21.15.160:0/476181916 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb04009320 con 0x7fcb1c0f6510 2023-11-17T05:33:27.964 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:27.962+0000 7fcb20e5a700 1 -- 172.21.15.160:0/476181916 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb1c106e20 con 0x7fcb1c0f6510 2023-11-17T05:33:27.964 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:27.962+0000 7fcb12ffd700 1 -- 172.21.15.160:0/476181916 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcb04013f00 con 0x7fcb1c0f6510 2023-11-17T05:33:27.965 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:27.962+0000 7fcb12ffd700 1 -- 172.21.15.160:0/476181916 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb04014d90 con 0x7fcb1c0f6510 2023-11-17T05:33:27.965 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:27.962+0000 7fcb20e5a700 1 -- 172.21.15.160:0/476181916 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb1c1072b0 con 0x7fcb1c0f6510 2023-11-17T05:33:27.965 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:27.962+0000 7fcb12ffd700 1 -- 172.21.15.160:0/476181916 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcb040095f0 con 0x7fcb1c0f6510 2023-11-17T05:33:27.965 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:27.962+0000 7fcb20e5a700 1 -- 172.21.15.160:0/476181916 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb1c045ad0 con 0x7fcb1c0f6510 2023-11-17T05:33:27.965 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:27.963+0000 7fcb12ffd700 1 --2- 172.21.15.160:0/476181916 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fcb0802ff60 0x7fcb08032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:33:27.965 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:27.963+0000 7fcb12ffd700 1 -- 172.21.15.160:0/476181916 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcb04040fe0 con 0x7fcb1c0f6510 2023-11-17T05:33:27.965 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:27.963+0000 7fcb19d9b700 1 --2- 172.21.15.160:0/476181916 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fcb0802ff60 0x7fcb08032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:33:27.965 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:27.963+0000 7fcb19d9b700 1 --2- 172.21.15.160:0/476181916 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fcb0802ff60 0x7fcb08032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fcb0c0096a0 tx=0x7fcb0c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:33:27.968 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:27.968+0000 7fcb12ffd700 1 -- 172.21.15.160:0/476181916 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcb0401d7f0 con 0x7fcb1c0f6510 2023-11-17T05:33:28.173 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:28.172+0000 7fcb20e5a700 1 -- 172.21.15.160:0/476181916 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcb1c0fbf60 con 0x7fcb1c0f6510 2023-11-17T05:33:28.173 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:28.173+0000 7fcb12ffd700 1 -- 172.21.15.160:0/476181916 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fcb04010230 con 0x7fcb1c0f6510 2023-11-17T05:33:28.174 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:33:28.174 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:33:28.175 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:28.174+0000 7fcb20e5a700 1 -- 172.21.15.160:0/476181916 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fcb0802ff60 msgr2=0x7fcb08032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:33:28.175 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:28.174+0000 7fcb20e5a700 1 --2- 172.21.15.160:0/476181916 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fcb0802ff60 0x7fcb08032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fcb0c0096a0 tx=0x7fcb0c006b40).stop 2023-11-17T05:33:28.176 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:28.174+0000 7fcb20e5a700 1 -- 172.21.15.160:0/476181916 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb1c0f6510 msgr2=0x7fcb1c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:33:28.176 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:28.174+0000 7fcb20e5a700 1 --2- 172.21.15.160:0/476181916 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb1c0f6510 0x7fcb1c106730 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fcb04013510 tx=0x7fcb0400b040).stop 2023-11-17T05:33:28.176 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:28.175+0000 7fcb20e5a700 1 -- 172.21.15.160:0/476181916 shutdown_connections 2023-11-17T05:33:28.176 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:28.175+0000 7fcb20e5a700 1 --2- 172.21.15.160:0/476181916 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fcb0802ff60 0x7fcb08032420 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:33:28.176 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:28.175+0000 7fcb20e5a700 1 --2- 172.21.15.160:0/476181916 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcb1c0f6510 0x7fcb1c106730 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:33:28.176 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:28.175+0000 7fcb20e5a700 1 -- 172.21.15.160:0/476181916 >> 172.21.15.160:0/476181916 conn(0x7fcb1c0f1c70 msgr2=0x7fcb1c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:33:28.176 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:28.175+0000 7fcb20e5a700 1 -- 172.21.15.160:0/476181916 shutdown_connections 2023-11-17T05:33:28.176 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:28.175+0000 7fcb20e5a700 1 -- 172.21.15.160:0/476181916 wait complete. 2023-11-17T05:33:28.176 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:33:29.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:33:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:33:27.466670+0000 mgr.smithi003.eddcuc (mgr.14164) 248 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:29.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:33:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:33:27.554074+0000 mon.smithi003 (mon.0) 225 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.eddcuc/mirror_snapshot_schedule"}]: dispatch 2023-11-17T05:33:29.021 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:33:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:33:27.557742+0000 mon.smithi003 (mon.0) 226 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.eddcuc/trash_purge_schedule"}]: dispatch 2023-11-17T05:33:29.021 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:33:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:33:28.173839+0000 mon.smithi003 (mon.0) 227 : audit [DBG] from='client.? 172.21.15.160:0/476181916' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:33:29.589 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:33:29.590 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:33:30.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:33:29 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:33:29.467029+0000 mgr.smithi003.eddcuc (mgr.14164) 249 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:30.819 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:30.814+0000 7f6d50a0c700 1 -- 172.21.15.160:0/1226772035 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d4c0f6510 msgr2=0x7f6d4c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:33:30.819 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:30.814+0000 7f6d50a0c700 1 --2- 172.21.15.160:0/1226772035 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d4c0f6510 0x7f6d4c0f6930 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f6d44009fb0 tx=0x7f6d44009b30).stop 2023-11-17T05:33:30.820 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:30.814+0000 7f6d50a0c700 1 -- 172.21.15.160:0/1226772035 shutdown_connections 2023-11-17T05:33:30.820 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:30.814+0000 7f6d50a0c700 1 --2- 172.21.15.160:0/1226772035 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d4c0f6510 0x7f6d4c0f6930 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:33:30.820 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:30.814+0000 7f6d50a0c700 1 -- 172.21.15.160:0/1226772035 >> 172.21.15.160:0/1226772035 conn(0x7f6d4c0f1c70 msgr2=0x7f6d4c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:33:30.820 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:30.814+0000 7f6d50a0c700 1 -- 172.21.15.160:0/1226772035 shutdown_connections 2023-11-17T05:33:30.821 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:30.814+0000 7f6d50a0c700 1 -- 172.21.15.160:0/1226772035 wait complete. 2023-11-17T05:33:30.821 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:30.815+0000 7f6d50a0c700 1 Processor -- start 2023-11-17T05:33:30.821 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:30.815+0000 7f6d50a0c700 1 -- start start 2023-11-17T05:33:30.821 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:30.815+0000 7f6d50a0c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d4c0f6510 0x7f6d4c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:33:30.822 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:30.815+0000 7f6d50a0c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4c106c20 con 0x7f6d4c0f6510 2023-11-17T05:33:30.822 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:30.815+0000 7f6d4a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d4c0f6510 0x7f6d4c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:33:30.822 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:30.815+0000 7f6d4a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d4c0f6510 0x7f6d4c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:37950/0 (socket says 172.21.15.160:37950) 2023-11-17T05:33:30.822 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:30.815+0000 7f6d4a59c700 1 -- 172.21.15.160:0/2025425825 learned_addr learned my addr 172.21.15.160:0/2025425825 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:33:30.823 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:30.815+0000 7f6d4a59c700 1 -- 172.21.15.160:0/2025425825 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d44005040 con 0x7f6d4c0f6510 2023-11-17T05:33:30.823 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:30.815+0000 7f6d4a59c700 1 --2- 172.21.15.160:0/2025425825 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d4c0f6510 0x7f6d4c106730 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f6d44013510 tx=0x7f6d4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:33:30.823 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:30.816+0000 7f6d42ffd700 1 -- 172.21.15.160:0/2025425825 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d44009320 con 0x7f6d4c0f6510 2023-11-17T05:33:30.823 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:30.816+0000 7f6d42ffd700 1 -- 172.21.15.160:0/2025425825 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6d44013f00 con 0x7f6d4c0f6510 2023-11-17T05:33:30.824 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:30.816+0000 7f6d50a0c700 1 -- 172.21.15.160:0/2025425825 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d4c106e20 con 0x7f6d4c0f6510 2023-11-17T05:33:30.824 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:30.816+0000 7f6d42ffd700 1 -- 172.21.15.160:0/2025425825 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d44014d90 con 0x7f6d4c0f6510 2023-11-17T05:33:30.824 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:30.816+0000 7f6d50a0c700 1 -- 172.21.15.160:0/2025425825 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d4c1072b0 con 0x7f6d4c0f6510 2023-11-17T05:33:30.824 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:30.816+0000 7f6d42ffd700 1 -- 172.21.15.160:0/2025425825 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6d440095f0 con 0x7f6d4c0f6510 2023-11-17T05:33:30.824 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:30.816+0000 7f6d50a0c700 1 -- 172.21.15.160:0/2025425825 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d4c045ad0 con 0x7f6d4c0f6510 2023-11-17T05:33:30.825 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:30.817+0000 7f6d42ffd700 1 --2- 172.21.15.160:0/2025425825 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f6d3802ff60 0x7f6d38032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:33:30.825 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:30.817+0000 7f6d42ffd700 1 -- 172.21.15.160:0/2025425825 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6d440410c0 con 0x7f6d4c0f6510 2023-11-17T05:33:30.825 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:30.817+0000 7f6d49d9b700 1 --2- 172.21.15.160:0/2025425825 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f6d3802ff60 0x7f6d38032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:33:30.825 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:30.817+0000 7f6d49d9b700 1 --2- 172.21.15.160:0/2025425825 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f6d3802ff60 0x7f6d38032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f6d340096a0 tx=0x7f6d34006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:33:30.826 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:30.821+0000 7f6d42ffd700 1 -- 172.21.15.160:0/2025425825 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6d44014420 con 0x7f6d4c0f6510 2023-11-17T05:33:31.030 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:31.029+0000 7f6d50a0c700 1 -- 172.21.15.160:0/2025425825 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6d4c0fbf60 con 0x7f6d4c0f6510 2023-11-17T05:33:31.030 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:31.029+0000 7f6d42ffd700 1 -- 172.21.15.160:0/2025425825 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f6d44010230 con 0x7f6d4c0f6510 2023-11-17T05:33:31.030 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:33:31.031 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:33:31.032 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:31.031+0000 7f6d50a0c700 1 -- 172.21.15.160:0/2025425825 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f6d3802ff60 msgr2=0x7f6d38032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:33:31.032 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:31.031+0000 7f6d50a0c700 1 --2- 172.21.15.160:0/2025425825 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f6d3802ff60 0x7f6d38032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f6d340096a0 tx=0x7f6d34006b40).stop 2023-11-17T05:33:31.032 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:31.031+0000 7f6d50a0c700 1 -- 172.21.15.160:0/2025425825 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d4c0f6510 msgr2=0x7f6d4c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:33:31.032 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:31.031+0000 7f6d50a0c700 1 --2- 172.21.15.160:0/2025425825 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d4c0f6510 0x7f6d4c106730 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f6d44013510 tx=0x7f6d4400b040).stop 2023-11-17T05:33:31.032 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:31.031+0000 7f6d50a0c700 1 -- 172.21.15.160:0/2025425825 shutdown_connections 2023-11-17T05:33:31.033 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:31.031+0000 7f6d50a0c700 1 --2- 172.21.15.160:0/2025425825 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f6d3802ff60 0x7f6d38032420 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:33:31.033 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:31.031+0000 7f6d50a0c700 1 --2- 172.21.15.160:0/2025425825 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d4c0f6510 0x7f6d4c106730 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:33:31.033 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:31.031+0000 7f6d50a0c700 1 -- 172.21.15.160:0/2025425825 >> 172.21.15.160:0/2025425825 conn(0x7f6d4c0f1c70 msgr2=0x7f6d4c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:33:31.033 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:31.031+0000 7f6d50a0c700 1 -- 172.21.15.160:0/2025425825 shutdown_connections 2023-11-17T05:33:31.033 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:31.031+0000 7f6d50a0c700 1 -- 172.21.15.160:0/2025425825 wait complete. 2023-11-17T05:33:31.033 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:33:31.520 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:33:31 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:33:31.030667+0000 mon.smithi003 (mon.0) 228 : audit [DBG] from='client.? 172.21.15.160:0/2025425825' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:33:32.410 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:33:32.411 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:33:32.527 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:33:32 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:33:31.467307+0000 mgr.smithi003.eddcuc (mgr.14164) 250 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:33.655 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:33.654+0000 7fab7121e700 1 -- 172.21.15.160:0/1765932360 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab6c060220 msgr2=0x7fab6c0f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:33:33.655 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:33.654+0000 7fab7121e700 1 --2- 172.21.15.160:0/1765932360 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab6c060220 0x7fab6c0f9150 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7fab60009fb0 tx=0x7fab60009b30).stop 2023-11-17T05:33:33.655 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:33.654+0000 7fab7121e700 1 -- 172.21.15.160:0/1765932360 shutdown_connections 2023-11-17T05:33:33.655 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:33.654+0000 7fab7121e700 1 --2- 172.21.15.160:0/1765932360 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab6c060220 0x7fab6c0f9150 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:33:33.655 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:33.654+0000 7fab7121e700 1 -- 172.21.15.160:0/1765932360 >> 172.21.15.160:0/1765932360 conn(0x7fab6c0f0b70 msgr2=0x7fab6c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:33:33.655 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:33.654+0000 7fab7121e700 1 -- 172.21.15.160:0/1765932360 shutdown_connections 2023-11-17T05:33:33.656 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:33.654+0000 7fab7121e700 1 -- 172.21.15.160:0/1765932360 wait complete. 2023-11-17T05:33:33.657 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:33.655+0000 7fab7121e700 1 Processor -- start 2023-11-17T05:33:33.657 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:33.655+0000 7fab7121e700 1 -- start start 2023-11-17T05:33:33.657 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:33.655+0000 7fab7121e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab6c060220 0x7fab6c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:33:33.657 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:33.655+0000 7fab7121e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab6c106bd0 con 0x7fab6c060220 2023-11-17T05:33:33.657 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:33.655+0000 7fab6ad9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab6c060220 0x7fab6c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:33:33.657 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:33.655+0000 7fab6ad9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab6c060220 0x7fab6c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:37972/0 (socket says 172.21.15.160:37972) 2023-11-17T05:33:33.658 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:33.655+0000 7fab6ad9d700 1 -- 172.21.15.160:0/3073634310 learned_addr learned my addr 172.21.15.160:0/3073634310 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:33:33.658 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:33.655+0000 7fab6ad9d700 1 -- 172.21.15.160:0/3073634310 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab60005040 con 0x7fab6c060220 2023-11-17T05:33:33.658 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:33.655+0000 7fab6ad9d700 1 --2- 172.21.15.160:0/3073634310 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab6c060220 0x7fab6c1066e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fab60013510 tx=0x7fab6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:33:33.658 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:33.656+0000 7fab537fe700 1 -- 172.21.15.160:0/3073634310 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab60009320 con 0x7fab6c060220 2023-11-17T05:33:33.658 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:33.656+0000 7fab537fe700 1 -- 172.21.15.160:0/3073634310 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fab60013f00 con 0x7fab6c060220 2023-11-17T05:33:33.658 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:33.656+0000 7fab7121e700 1 -- 172.21.15.160:0/3073634310 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab6c106dd0 con 0x7fab6c060220 2023-11-17T05:33:33.658 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:33.656+0000 7fab537fe700 1 -- 172.21.15.160:0/3073634310 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab60014dc0 con 0x7fab6c060220 2023-11-17T05:33:33.658 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:33.656+0000 7fab7121e700 1 -- 172.21.15.160:0/3073634310 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab6c107260 con 0x7fab6c060220 2023-11-17T05:33:33.658 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:33.657+0000 7fab537fe700 1 -- 172.21.15.160:0/3073634310 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fab600095f0 con 0x7fab6c060220 2023-11-17T05:33:33.658 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:33.657+0000 7fab7121e700 1 -- 172.21.15.160:0/3073634310 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab6c045ad0 con 0x7fab6c060220 2023-11-17T05:33:33.659 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:33.657+0000 7fab537fe700 1 --2- 172.21.15.160:0/3073634310 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fab5402ff60 0x7fab54032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:33:33.659 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:33.657+0000 7fab537fe700 1 -- 172.21.15.160:0/3073634310 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fab600410c0 con 0x7fab6c060220 2023-11-17T05:33:33.659 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:33.657+0000 7fab6a59c700 1 --2- 172.21.15.160:0/3073634310 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fab5402ff60 0x7fab54032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:33:33.659 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:33.657+0000 7fab6a59c700 1 --2- 172.21.15.160:0/3073634310 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fab5402ff60 0x7fab54032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fab5c0096a0 tx=0x7fab5c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:33:33.662 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:33.662+0000 7fab537fe700 1 -- 172.21.15.160:0/3073634310 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fab6001c790 con 0x7fab6c060220 2023-11-17T05:33:33.869 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:33.868+0000 7fab7121e700 1 -- 172.21.15.160:0/3073634310 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fab6c108110 con 0x7fab6c060220 2023-11-17T05:33:33.870 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:33.869+0000 7fab537fe700 1 -- 172.21.15.160:0/3073634310 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fab60010230 con 0x7fab6c060220 2023-11-17T05:33:33.870 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:33:33.871 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:33:33.872 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:33.871+0000 7fab7121e700 1 -- 172.21.15.160:0/3073634310 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fab5402ff60 msgr2=0x7fab54032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:33:33.872 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:33.871+0000 7fab7121e700 1 --2- 172.21.15.160:0/3073634310 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fab5402ff60 0x7fab54032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fab5c0096a0 tx=0x7fab5c006b40).stop 2023-11-17T05:33:33.872 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:33.871+0000 7fab7121e700 1 -- 172.21.15.160:0/3073634310 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab6c060220 msgr2=0x7fab6c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:33:33.872 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:33.871+0000 7fab7121e700 1 --2- 172.21.15.160:0/3073634310 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab6c060220 0x7fab6c1066e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fab60013510 tx=0x7fab6000b040).stop 2023-11-17T05:33:33.872 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:33.871+0000 7fab7121e700 1 -- 172.21.15.160:0/3073634310 shutdown_connections 2023-11-17T05:33:33.872 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:33.871+0000 7fab7121e700 1 --2- 172.21.15.160:0/3073634310 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fab5402ff60 0x7fab54032420 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:33:33.872 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:33.871+0000 7fab7121e700 1 --2- 172.21.15.160:0/3073634310 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab6c060220 0x7fab6c1066e0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:33:33.872 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:33.871+0000 7fab7121e700 1 -- 172.21.15.160:0/3073634310 >> 172.21.15.160:0/3073634310 conn(0x7fab6c0f0b70 msgr2=0x7fab6c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:33:33.872 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:33.871+0000 7fab7121e700 1 -- 172.21.15.160:0/3073634310 shutdown_connections 2023-11-17T05:33:33.872 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:33.871+0000 7fab7121e700 1 -- 172.21.15.160:0/3073634310 wait complete. 2023-11-17T05:33:33.873 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:33:34.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:33:34 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:33:33.467579+0000 mgr.smithi003.eddcuc (mgr.14164) 251 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:34.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:33:34 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:33:33.870370+0000 mon.smithi003 (mon.0) 229 : audit [DBG] from='client.? 172.21.15.160:0/3073634310' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:33:35.275 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:33:35.275 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:33:36.537 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:36.535+0000 7fa5ffbaa700 1 -- 172.21.15.160:0/1720458247 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5f80f6510 msgr2=0x7fa5f80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:33:36.537 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:36.535+0000 7fa5ffbaa700 1 --2- 172.21.15.160:0/1720458247 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5f80f6510 0x7fa5f80f6930 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7fa5e8009fb0 tx=0x7fa5e8009b30).stop 2023-11-17T05:33:36.537 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:36.535+0000 7fa5ffbaa700 1 -- 172.21.15.160:0/1720458247 shutdown_connections 2023-11-17T05:33:36.537 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:36.535+0000 7fa5ffbaa700 1 --2- 172.21.15.160:0/1720458247 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5f80f6510 0x7fa5f80f6930 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:33:36.537 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:36.535+0000 7fa5ffbaa700 1 -- 172.21.15.160:0/1720458247 >> 172.21.15.160:0/1720458247 conn(0x7fa5f80f1c70 msgr2=0x7fa5f80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:33:36.538 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:36.535+0000 7fa5ffbaa700 1 -- 172.21.15.160:0/1720458247 shutdown_connections 2023-11-17T05:33:36.538 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:36.535+0000 7fa5ffbaa700 1 -- 172.21.15.160:0/1720458247 wait complete. 2023-11-17T05:33:36.538 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:36.536+0000 7fa5ffbaa700 1 Processor -- start 2023-11-17T05:33:36.538 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:36.536+0000 7fa5ffbaa700 1 -- start start 2023-11-17T05:33:36.538 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:36.536+0000 7fa5ffbaa700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5f80f6510 0x7fa5f8108910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:33:36.538 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:36.536+0000 7fa5ffbaa700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f8108e00 con 0x7fa5f80f6510 2023-11-17T05:33:36.538 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:36.536+0000 7fa5fd946700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5f80f6510 0x7fa5f8108910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:33:36.538 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:36.536+0000 7fa5fd946700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5f80f6510 0x7fa5f8108910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:37984/0 (socket says 172.21.15.160:37984) 2023-11-17T05:33:36.538 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:36.536+0000 7fa5fd946700 1 -- 172.21.15.160:0/1023778184 learned_addr learned my addr 172.21.15.160:0/1023778184 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:33:36.539 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:36.537+0000 7fa5fd946700 1 -- 172.21.15.160:0/1023778184 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e8005040 con 0x7fa5f80f6510 2023-11-17T05:33:36.540 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:36.537+0000 7fa5fd946700 1 --2- 172.21.15.160:0/1023778184 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5f80f6510 0x7fa5f8108910 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7fa5e8014000 tx=0x7fa5e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:33:36.540 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:36.537+0000 7fa5ee7fc700 1 -- 172.21.15.160:0/1023778184 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5e80095f0 con 0x7fa5f80f6510 2023-11-17T05:33:36.540 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:36.537+0000 7fa5ffbaa700 1 -- 172.21.15.160:0/1023778184 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5f8109000 con 0x7fa5f80f6510 2023-11-17T05:33:36.540 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:36.537+0000 7fa5ee7fc700 1 -- 172.21.15.160:0/1023778184 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa5e8007e70 con 0x7fa5f80f6510 2023-11-17T05:33:36.540 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:36.537+0000 7fa5ee7fc700 1 -- 172.21.15.160:0/1023778184 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5e8014ea0 con 0x7fa5f80f6510 2023-11-17T05:33:36.540 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:36.537+0000 7fa5ffbaa700 1 -- 172.21.15.160:0/1023778184 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5f8109490 con 0x7fa5f80f6510 2023-11-17T05:33:36.540 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:36.538+0000 7fa5ee7fc700 1 -- 172.21.15.160:0/1023778184 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa5e8025070 con 0x7fa5f80f6510 2023-11-17T05:33:36.540 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:36.538+0000 7fa5ffbaa700 1 -- 172.21.15.160:0/1023778184 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5f8045ad0 con 0x7fa5f80f6510 2023-11-17T05:33:36.540 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:36.538+0000 7fa5ee7fc700 1 --2- 172.21.15.160:0/1023778184 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa5e402ff60 0x7fa5e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:33:36.540 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:36.538+0000 7fa5ee7fc700 1 -- 172.21.15.160:0/1023778184 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa5e8040a30 con 0x7fa5f80f6510 2023-11-17T05:33:36.540 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:36.538+0000 7fa5fd145700 1 --2- 172.21.15.160:0/1023778184 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa5e402ff60 0x7fa5e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:33:36.540 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:36.539+0000 7fa5fd145700 1 --2- 172.21.15.160:0/1023778184 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa5e402ff60 0x7fa5e4032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fa5f40096a0 tx=0x7fa5f4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:33:36.544 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:36.544+0000 7fa5ee7fc700 1 -- 172.21.15.160:0/1023778184 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa5e80079c0 con 0x7fa5f80f6510 2023-11-17T05:33:36.746 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:36.746+0000 7fa5ffbaa700 1 -- 172.21.15.160:0/1023778184 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa5f810a2e0 con 0x7fa5f80f6510 2023-11-17T05:33:36.747 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:36.746+0000 7fa5ee7fc700 1 -- 172.21.15.160:0/1023778184 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fa5e800f300 con 0x7fa5f80f6510 2023-11-17T05:33:36.747 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:33:36.747 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:33:36.749 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:36.748+0000 7fa5ffbaa700 1 -- 172.21.15.160:0/1023778184 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa5e402ff60 msgr2=0x7fa5e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:33:36.749 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:36.748+0000 7fa5ffbaa700 1 --2- 172.21.15.160:0/1023778184 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa5e402ff60 0x7fa5e4032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fa5f40096a0 tx=0x7fa5f4006b40).stop 2023-11-17T05:33:36.749 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:36.748+0000 7fa5ffbaa700 1 -- 172.21.15.160:0/1023778184 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5f80f6510 msgr2=0x7fa5f8108910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:33:36.749 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:36.748+0000 7fa5ffbaa700 1 --2- 172.21.15.160:0/1023778184 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5f80f6510 0x7fa5f8108910 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7fa5e8014000 tx=0x7fa5e800b040).stop 2023-11-17T05:33:36.749 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:36.748+0000 7fa5ffbaa700 1 -- 172.21.15.160:0/1023778184 shutdown_connections 2023-11-17T05:33:36.749 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:36.748+0000 7fa5ffbaa700 1 --2- 172.21.15.160:0/1023778184 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa5e402ff60 0x7fa5e4032420 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:33:36.749 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:36.748+0000 7fa5ffbaa700 1 --2- 172.21.15.160:0/1023778184 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5f80f6510 0x7fa5f8108910 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:33:36.749 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:36.748+0000 7fa5ffbaa700 1 -- 172.21.15.160:0/1023778184 >> 172.21.15.160:0/1023778184 conn(0x7fa5f80f1c70 msgr2=0x7fa5f80f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:33:36.749 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:36.748+0000 7fa5ffbaa700 1 -- 172.21.15.160:0/1023778184 shutdown_connections 2023-11-17T05:33:36.749 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:36.748+0000 7fa5ffbaa700 1 -- 172.21.15.160:0/1023778184 wait complete. 2023-11-17T05:33:36.750 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:33:37.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:33:36 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:33:35.467892+0000 mgr.smithi003.eddcuc (mgr.14164) 252 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:38.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:33:37 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:33:36.747571+0000 mon.smithi003 (mon.0) 230 : audit [DBG] from='client.? 172.21.15.160:0/1023778184' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:33:38.145 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:33:38.145 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:33:39.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:33:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:33:37.468216+0000 mgr.smithi003.eddcuc (mgr.14164) 253 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:39.305 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:39.304+0000 7ff9b37de700 1 -- 172.21.15.160:0/839617229 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9ac0f5410 msgr2=0x7ff9ac0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:33:39.306 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:39.304+0000 7ff9b37de700 1 --2- 172.21.15.160:0/839617229 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9ac0f5410 0x7ff9ac0f5830 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7ff9a8009fb0 tx=0x7ff9a8009b30).stop 2023-11-17T05:33:39.306 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:39.304+0000 7ff9b37de700 1 -- 172.21.15.160:0/839617229 shutdown_connections 2023-11-17T05:33:39.306 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:39.304+0000 7ff9b37de700 1 --2- 172.21.15.160:0/839617229 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9ac0f5410 0x7ff9ac0f5830 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:33:39.306 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:39.304+0000 7ff9b37de700 1 -- 172.21.15.160:0/839617229 >> 172.21.15.160:0/839617229 conn(0x7ff9ac0f0b70 msgr2=0x7ff9ac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:33:39.306 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:39.304+0000 7ff9b37de700 1 -- 172.21.15.160:0/839617229 shutdown_connections 2023-11-17T05:33:39.306 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:39.304+0000 7ff9b37de700 1 -- 172.21.15.160:0/839617229 wait complete. 2023-11-17T05:33:39.306 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:39.305+0000 7ff9b37de700 1 Processor -- start 2023-11-17T05:33:39.306 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:39.305+0000 7ff9b37de700 1 -- start start 2023-11-17T05:33:39.306 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:39.305+0000 7ff9b37de700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9ac0f5410 0x7ff9ac105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:33:39.306 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:39.305+0000 7ff9b37de700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ac105af0 con 0x7ff9ac0f5410 2023-11-17T05:33:39.307 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:39.305+0000 7ff9b157a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9ac0f5410 0x7ff9ac105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:33:39.307 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:39.305+0000 7ff9b157a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9ac0f5410 0x7ff9ac105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:38006/0 (socket says 172.21.15.160:38006) 2023-11-17T05:33:39.307 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:39.305+0000 7ff9b157a700 1 -- 172.21.15.160:0/1084406584 learned_addr learned my addr 172.21.15.160:0/1084406584 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:33:39.307 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:39.305+0000 7ff9b157a700 1 -- 172.21.15.160:0/1084406584 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a8005040 con 0x7ff9ac0f5410 2023-11-17T05:33:39.307 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:39.305+0000 7ff9b157a700 1 --2- 172.21.15.160:0/1084406584 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9ac0f5410 0x7ff9ac105600 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7ff9a8013510 tx=0x7ff9a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:33:39.307 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:39.306+0000 7ff9a1ffb700 1 -- 172.21.15.160:0/1084406584 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9a8009320 con 0x7ff9ac0f5410 2023-11-17T05:33:39.307 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:39.306+0000 7ff9b37de700 1 -- 172.21.15.160:0/1084406584 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9ac105cf0 con 0x7ff9ac0f5410 2023-11-17T05:33:39.307 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:39.306+0000 7ff9a1ffb700 1 -- 172.21.15.160:0/1084406584 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff9a8013f00 con 0x7ff9ac0f5410 2023-11-17T05:33:39.307 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:39.306+0000 7ff9a1ffb700 1 -- 172.21.15.160:0/1084406584 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9a8014d90 con 0x7ff9ac0f5410 2023-11-17T05:33:39.307 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:39.306+0000 7ff9b37de700 1 -- 172.21.15.160:0/1084406584 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9ac106180 con 0x7ff9ac0f5410 2023-11-17T05:33:39.308 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:39.307+0000 7ff9a1ffb700 1 -- 172.21.15.160:0/1084406584 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff9a80095f0 con 0x7ff9ac0f5410 2023-11-17T05:33:39.308 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:39.307+0000 7ff9b37de700 1 -- 172.21.15.160:0/1084406584 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9ac045ad0 con 0x7ff9ac0f5410 2023-11-17T05:33:39.308 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:39.307+0000 7ff9a1ffb700 1 --2- 172.21.15.160:0/1084406584 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7ff99802ff60 0x7ff998032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:33:39.308 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:39.307+0000 7ff9a1ffb700 1 -- 172.21.15.160:0/1084406584 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff9a8040fe0 con 0x7ff9ac0f5410 2023-11-17T05:33:39.308 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:39.307+0000 7ff9b0d79700 1 --2- 172.21.15.160:0/1084406584 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7ff99802ff60 0x7ff998032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:33:39.309 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:39.307+0000 7ff9b0d79700 1 --2- 172.21.15.160:0/1084406584 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7ff99802ff60 0x7ff998032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7ff99c0096a0 tx=0x7ff99c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:33:39.311 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:39.312+0000 7ff9a1ffb700 1 -- 172.21.15.160:0/1084406584 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff9a8014420 con 0x7ff9ac0f5410 2023-11-17T05:33:39.517 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:39.517+0000 7ff9b37de700 1 -- 172.21.15.160:0/1084406584 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff9ac0fae60 con 0x7ff9ac0f5410 2023-11-17T05:33:39.517 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:39.517+0000 7ff9a1ffb700 1 -- 172.21.15.160:0/1084406584 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7ff9a8010230 con 0x7ff9ac0f5410 2023-11-17T05:33:39.518 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:33:39.518 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:33:39.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:39.519+0000 7ff9b37de700 1 -- 172.21.15.160:0/1084406584 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7ff99802ff60 msgr2=0x7ff998032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:33:39.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:39.519+0000 7ff9b37de700 1 --2- 172.21.15.160:0/1084406584 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7ff99802ff60 0x7ff998032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7ff99c0096a0 tx=0x7ff99c006b40).stop 2023-11-17T05:33:39.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:39.519+0000 7ff9b37de700 1 -- 172.21.15.160:0/1084406584 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9ac0f5410 msgr2=0x7ff9ac105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:33:39.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:39.519+0000 7ff9b37de700 1 --2- 172.21.15.160:0/1084406584 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9ac0f5410 0x7ff9ac105600 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7ff9a8013510 tx=0x7ff9a800b040).stop 2023-11-17T05:33:39.519 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:39.519+0000 7ff9b37de700 1 -- 172.21.15.160:0/1084406584 shutdown_connections 2023-11-17T05:33:39.520 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:39.519+0000 7ff9b37de700 1 --2- 172.21.15.160:0/1084406584 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7ff99802ff60 0x7ff998032420 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:33:39.520 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:39.519+0000 7ff9b37de700 1 --2- 172.21.15.160:0/1084406584 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9ac0f5410 0x7ff9ac105600 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:33:39.520 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:39.519+0000 7ff9b37de700 1 -- 172.21.15.160:0/1084406584 >> 172.21.15.160:0/1084406584 conn(0x7ff9ac0f0b70 msgr2=0x7ff9ac0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:33:39.520 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:39.519+0000 7ff9b37de700 1 -- 172.21.15.160:0/1084406584 shutdown_connections 2023-11-17T05:33:39.520 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:39.519+0000 7ff9b37de700 1 -- 172.21.15.160:0/1084406584 wait complete. 2023-11-17T05:33:39.520 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:33:40.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:33:39 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:33:39.468564+0000 mgr.smithi003.eddcuc (mgr.14164) 254 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:40.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:33:39 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:33:39.517760+0000 mon.smithi003 (mon.0) 231 : audit [DBG] from='client.? 172.21.15.160:0/1084406584' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:33:40.921 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:33:40.921 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:33:42.165 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:42.163+0000 7fe34e5a5700 1 -- 172.21.15.160:0/1865314879 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3480f5410 msgr2=0x7fe3480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:33:42.165 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:42.163+0000 7fe34e5a5700 1 --2- 172.21.15.160:0/1865314879 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3480f5410 0x7fe3480f5830 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7fe338009fb0 tx=0x7fe338009b30).stop 2023-11-17T05:33:42.165 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:42.163+0000 7fe34e5a5700 1 -- 172.21.15.160:0/1865314879 shutdown_connections 2023-11-17T05:33:42.166 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:42.163+0000 7fe34e5a5700 1 --2- 172.21.15.160:0/1865314879 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3480f5410 0x7fe3480f5830 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:33:42.166 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:42.163+0000 7fe34e5a5700 1 -- 172.21.15.160:0/1865314879 >> 172.21.15.160:0/1865314879 conn(0x7fe3480f0b70 msgr2=0x7fe3480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:33:42.166 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:42.163+0000 7fe34e5a5700 1 -- 172.21.15.160:0/1865314879 shutdown_connections 2023-11-17T05:33:42.166 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:42.163+0000 7fe34e5a5700 1 -- 172.21.15.160:0/1865314879 wait complete. 2023-11-17T05:33:42.166 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:42.164+0000 7fe34e5a5700 1 Processor -- start 2023-11-17T05:33:42.166 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:42.164+0000 7fe34e5a5700 1 -- start start 2023-11-17T05:33:42.166 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:42.164+0000 7fe34e5a5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3480f5410 0x7fe348105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:33:42.166 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:42.164+0000 7fe34e5a5700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe348105b20 con 0x7fe3480f5410 2023-11-17T05:33:42.166 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:42.165+0000 7fe347fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3480f5410 0x7fe348105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:33:42.166 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:42.165+0000 7fe347fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3480f5410 0x7fe348105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:43788/0 (socket says 172.21.15.160:43788) 2023-11-17T05:33:42.167 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:42.165+0000 7fe347fff700 1 -- 172.21.15.160:0/2985738848 learned_addr learned my addr 172.21.15.160:0/2985738848 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:33:42.168 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:42.165+0000 7fe347fff700 1 -- 172.21.15.160:0/2985738848 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe338005040 con 0x7fe3480f5410 2023-11-17T05:33:42.168 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:42.165+0000 7fe347fff700 1 --2- 172.21.15.160:0/2985738848 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3480f5410 0x7fe348105630 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fe338013510 tx=0x7fe33800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:33:42.168 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:42.165+0000 7fe344ff9700 1 -- 172.21.15.160:0/2985738848 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe338009320 con 0x7fe3480f5410 2023-11-17T05:33:42.168 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:42.165+0000 7fe34e5a5700 1 -- 172.21.15.160:0/2985738848 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe348105d20 con 0x7fe3480f5410 2023-11-17T05:33:42.169 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:42.165+0000 7fe344ff9700 1 -- 172.21.15.160:0/2985738848 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe338013f00 con 0x7fe3480f5410 2023-11-17T05:33:42.169 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:42.165+0000 7fe344ff9700 1 -- 172.21.15.160:0/2985738848 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe338014d90 con 0x7fe3480f5410 2023-11-17T05:33:42.169 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:42.165+0000 7fe34e5a5700 1 -- 172.21.15.160:0/2985738848 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3481061b0 con 0x7fe3480f5410 2023-11-17T05:33:42.169 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:42.166+0000 7fe344ff9700 1 -- 172.21.15.160:0/2985738848 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe3380095f0 con 0x7fe3480f5410 2023-11-17T05:33:42.169 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:42.166+0000 7fe34e5a5700 1 -- 172.21.15.160:0/2985738848 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe348045ad0 con 0x7fe3480f5410 2023-11-17T05:33:42.169 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:42.166+0000 7fe344ff9700 1 --2- 172.21.15.160:0/2985738848 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe33402ffb0 0x7fe334032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:33:42.169 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:42.166+0000 7fe344ff9700 1 -- 172.21.15.160:0/2985738848 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe338041050 con 0x7fe3480f5410 2023-11-17T05:33:42.169 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:42.166+0000 7fe3477fe700 1 --2- 172.21.15.160:0/2985738848 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe33402ffb0 0x7fe334032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:33:42.169 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:42.167+0000 7fe3477fe700 1 --2- 172.21.15.160:0/2985738848 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe33402ffb0 0x7fe334032470 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fe3300096a0 tx=0x7fe330006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:33:42.172 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:42.171+0000 7fe344ff9700 1 -- 172.21.15.160:0/2985738848 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe338014420 con 0x7fe3480f5410 2023-11-17T05:33:42.377 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:42.374+0000 7fe34e5a5700 1 -- 172.21.15.160:0/2985738848 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe3480fae60 con 0x7fe3480f5410 2023-11-17T05:33:42.377 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:42.375+0000 7fe344ff9700 1 -- 172.21.15.160:0/2985738848 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fe338010230 con 0x7fe3480f5410 2023-11-17T05:33:42.378 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:33:42.378 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:33:42.378 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:42.376+0000 7fe34e5a5700 1 -- 172.21.15.160:0/2985738848 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe33402ffb0 msgr2=0x7fe334032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:33:42.378 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:42.376+0000 7fe34e5a5700 1 --2- 172.21.15.160:0/2985738848 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe33402ffb0 0x7fe334032470 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fe3300096a0 tx=0x7fe330006b40).stop 2023-11-17T05:33:42.379 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:42.376+0000 7fe34e5a5700 1 -- 172.21.15.160:0/2985738848 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3480f5410 msgr2=0x7fe348105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:33:42.379 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:42.376+0000 7fe34e5a5700 1 --2- 172.21.15.160:0/2985738848 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3480f5410 0x7fe348105630 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fe338013510 tx=0x7fe33800b040).stop 2023-11-17T05:33:42.379 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:42.376+0000 7fe34e5a5700 1 -- 172.21.15.160:0/2985738848 shutdown_connections 2023-11-17T05:33:42.379 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:42.376+0000 7fe34e5a5700 1 --2- 172.21.15.160:0/2985738848 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe33402ffb0 0x7fe334032470 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:33:42.379 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:42.376+0000 7fe34e5a5700 1 --2- 172.21.15.160:0/2985738848 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3480f5410 0x7fe348105630 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:33:42.379 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:42.376+0000 7fe34e5a5700 1 -- 172.21.15.160:0/2985738848 >> 172.21.15.160:0/2985738848 conn(0x7fe3480f0b70 msgr2=0x7fe3480f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:33:42.379 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:42.376+0000 7fe34e5a5700 1 -- 172.21.15.160:0/2985738848 shutdown_connections 2023-11-17T05:33:42.379 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:42.376+0000 7fe34e5a5700 1 -- 172.21.15.160:0/2985738848 wait complete. 2023-11-17T05:33:42.379 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:33:42.831 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:33:42 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:33:41.468876+0000 mgr.smithi003.eddcuc (mgr.14164) 255 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:42.831 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:33:42 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:33:42.375977+0000 mon.smithi003 (mon.0) 232 : audit [DBG] from='client.? 172.21.15.160:0/2985738848' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:33:43.795 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:33:43.796 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:33:44.975 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:44.973+0000 7fddedcbc700 1 -- 172.21.15.160:0/1723484206 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdde80f51a0 msgr2=0x7fdde80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:33:44.975 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:44.973+0000 7fddedcbc700 1 --2- 172.21.15.160:0/1723484206 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdde80f51a0 0x7fdde80f55c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7fdddc009fb0 tx=0x7fdddc009b30).stop 2023-11-17T05:33:44.975 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:44.973+0000 7fddedcbc700 1 -- 172.21.15.160:0/1723484206 shutdown_connections 2023-11-17T05:33:44.975 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:44.973+0000 7fddedcbc700 1 --2- 172.21.15.160:0/1723484206 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdde80f51a0 0x7fdde80f55c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:33:44.975 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:44.973+0000 7fddedcbc700 1 -- 172.21.15.160:0/1723484206 >> 172.21.15.160:0/1723484206 conn(0x7fdde80f09a0 msgr2=0x7fdde80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:33:44.975 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:44.973+0000 7fddedcbc700 1 -- 172.21.15.160:0/1723484206 shutdown_connections 2023-11-17T05:33:44.975 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:44.973+0000 7fddedcbc700 1 -- 172.21.15.160:0/1723484206 wait complete. 2023-11-17T05:33:44.975 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:44.973+0000 7fddedcbc700 1 Processor -- start 2023-11-17T05:33:44.975 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:44.973+0000 7fddedcbc700 1 -- start start 2023-11-17T05:33:44.975 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:44.973+0000 7fddedcbc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdde80f51a0 0x7fdde81091f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:33:44.975 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:44.973+0000 7fddedcbc700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde8060220 con 0x7fdde80f51a0 2023-11-17T05:33:44.976 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:44.974+0000 7fddeccba700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdde80f51a0 0x7fdde81091f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:33:44.976 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:44.974+0000 7fddeccba700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdde80f51a0 0x7fdde81091f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:43806/0 (socket says 172.21.15.160:43806) 2023-11-17T05:33:44.976 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:44.974+0000 7fddeccba700 1 -- 172.21.15.160:0/3233052890 learned_addr learned my addr 172.21.15.160:0/3233052890 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:33:44.976 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:44.974+0000 7fddeccba700 1 -- 172.21.15.160:0/3233052890 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdddc005040 con 0x7fdde80f51a0 2023-11-17T05:33:44.976 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:44.974+0000 7fddeccba700 1 --2- 172.21.15.160:0/3233052890 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdde80f51a0 0x7fdde81091f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7fdddc013510 tx=0x7fdddc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:33:44.977 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:44.974+0000 7fdde57fa700 1 -- 172.21.15.160:0/3233052890 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdddc009320 con 0x7fdde80f51a0 2023-11-17T05:33:44.977 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:44.974+0000 7fddedcbc700 1 -- 172.21.15.160:0/3233052890 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdde8060420 con 0x7fdde80f51a0 2023-11-17T05:33:44.977 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:44.974+0000 7fdde57fa700 1 -- 172.21.15.160:0/3233052890 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdddc013f00 con 0x7fdde80f51a0 2023-11-17T05:33:44.977 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:44.974+0000 7fdde57fa700 1 -- 172.21.15.160:0/3233052890 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdddc014dc0 con 0x7fdde80f51a0 2023-11-17T05:33:44.977 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:44.974+0000 7fddedcbc700 1 -- 172.21.15.160:0/3233052890 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdde81098a0 con 0x7fdde80f51a0 2023-11-17T05:33:44.977 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:44.975+0000 7fdde57fa700 1 -- 172.21.15.160:0/3233052890 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdddc014420 con 0x7fdde80f51a0 2023-11-17T05:33:44.977 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:44.975+0000 7fddedcbc700 1 -- 172.21.15.160:0/3233052890 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdde8059100 con 0x7fdde80f51a0 2023-11-17T05:33:44.978 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:44.975+0000 7fdde57fa700 1 --2- 172.21.15.160:0/3233052890 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fddd802ff60 0x7fddd8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:33:44.978 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:44.976+0000 7fdde57fa700 1 -- 172.21.15.160:0/3233052890 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdddc041050 con 0x7fdde80f51a0 2023-11-17T05:33:44.978 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:44.976+0000 7fdde7fff700 1 --2- 172.21.15.160:0/3233052890 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fddd802ff60 0x7fddd8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:33:44.978 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:44.976+0000 7fdde7fff700 1 --2- 172.21.15.160:0/3233052890 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fddd802ff60 0x7fddd8032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fddd40096a0 tx=0x7fddd4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:33:44.981 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:44.980+0000 7fdde57fa700 1 -- 172.21.15.160:0/3233052890 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdddc0095f0 con 0x7fdde80f51a0 2023-11-17T05:33:45.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:33:44 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:33:43.469253+0000 mgr.smithi003.eddcuc (mgr.14164) 256 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:45.187 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:45.185+0000 7fddedcbc700 1 -- 172.21.15.160:0/3233052890 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdde8045ad0 con 0x7fdde80f51a0 2023-11-17T05:33:45.187 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:45.186+0000 7fdde57fa700 1 -- 172.21.15.160:0/3233052890 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fdddc01cb10 con 0x7fdde80f51a0 2023-11-17T05:33:45.187 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:33:45.187 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:33:45.188 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:45.187+0000 7fddedcbc700 1 -- 172.21.15.160:0/3233052890 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fddd802ff60 msgr2=0x7fddd8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:33:45.188 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:45.187+0000 7fddedcbc700 1 --2- 172.21.15.160:0/3233052890 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fddd802ff60 0x7fddd8032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fddd40096a0 tx=0x7fddd4006b40).stop 2023-11-17T05:33:45.189 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:45.187+0000 7fddedcbc700 1 -- 172.21.15.160:0/3233052890 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdde80f51a0 msgr2=0x7fdde81091f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:33:45.189 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:45.187+0000 7fddedcbc700 1 --2- 172.21.15.160:0/3233052890 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdde80f51a0 0x7fdde81091f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7fdddc013510 tx=0x7fdddc00b040).stop 2023-11-17T05:33:45.189 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:45.188+0000 7fddedcbc700 1 -- 172.21.15.160:0/3233052890 shutdown_connections 2023-11-17T05:33:45.189 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:45.188+0000 7fddedcbc700 1 --2- 172.21.15.160:0/3233052890 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fddd802ff60 0x7fddd8032420 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:33:45.189 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:45.188+0000 7fddedcbc700 1 --2- 172.21.15.160:0/3233052890 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdde80f51a0 0x7fdde81091f0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:33:45.189 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:45.188+0000 7fddedcbc700 1 -- 172.21.15.160:0/3233052890 >> 172.21.15.160:0/3233052890 conn(0x7fdde80f09a0 msgr2=0x7fdde80f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:33:45.189 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:45.188+0000 7fddedcbc700 1 -- 172.21.15.160:0/3233052890 shutdown_connections 2023-11-17T05:33:45.189 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:45.188+0000 7fddedcbc700 1 -- 172.21.15.160:0/3233052890 wait complete. 2023-11-17T05:33:45.189 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:33:46.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:33:45 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:33:45.187104+0000 mon.smithi003 (mon.0) 233 : audit [DBG] from='client.? 172.21.15.160:0/3233052890' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:33:46.566 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:33:46.566 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:33:47.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:33:46 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:33:45.469608+0000 mgr.smithi003.eddcuc (mgr.14164) 257 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:47.835 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:47.833+0000 7f6e36d06700 1 -- 172.21.15.160:0/2993672783 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e300f5410 msgr2=0x7f6e300f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:33:47.835 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:47.833+0000 7f6e36d06700 1 --2- 172.21.15.160:0/2993672783 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e300f5410 0x7f6e300f5830 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f6e24009fb0 tx=0x7f6e24009b30).stop 2023-11-17T05:33:47.835 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:47.833+0000 7f6e36d06700 1 -- 172.21.15.160:0/2993672783 shutdown_connections 2023-11-17T05:33:47.835 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:47.833+0000 7f6e36d06700 1 --2- 172.21.15.160:0/2993672783 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e300f5410 0x7f6e300f5830 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:33:47.835 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:47.833+0000 7f6e36d06700 1 -- 172.21.15.160:0/2993672783 >> 172.21.15.160:0/2993672783 conn(0x7f6e300f0b70 msgr2=0x7f6e300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:33:47.835 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:47.833+0000 7f6e36d06700 1 -- 172.21.15.160:0/2993672783 shutdown_connections 2023-11-17T05:33:47.835 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:47.833+0000 7f6e36d06700 1 -- 172.21.15.160:0/2993672783 wait complete. 2023-11-17T05:33:47.835 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:47.834+0000 7f6e36d06700 1 Processor -- start 2023-11-17T05:33:47.835 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:47.834+0000 7f6e36d06700 1 -- start start 2023-11-17T05:33:47.837 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:47.834+0000 7f6e36d06700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e300f5410 0x7f6e30105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:33:47.837 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:47.834+0000 7f6e36d06700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e30105af0 con 0x7f6e300f5410 2023-11-17T05:33:47.837 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:47.835+0000 7f6e34aa2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e300f5410 0x7f6e30105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:33:47.837 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:47.835+0000 7f6e34aa2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e300f5410 0x7f6e30105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:43818/0 (socket says 172.21.15.160:43818) 2023-11-17T05:33:47.837 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:47.835+0000 7f6e34aa2700 1 -- 172.21.15.160:0/1027357954 learned_addr learned my addr 172.21.15.160:0/1027357954 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:33:47.837 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:47.835+0000 7f6e34aa2700 1 -- 172.21.15.160:0/1027357954 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e24005040 con 0x7f6e300f5410 2023-11-17T05:33:47.837 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:47.835+0000 7f6e34aa2700 1 --2- 172.21.15.160:0/1027357954 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e300f5410 0x7f6e30105600 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f6e24013510 tx=0x7f6e2400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:33:47.837 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:47.835+0000 7f6e2d7fa700 1 -- 172.21.15.160:0/1027357954 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e24009320 con 0x7f6e300f5410 2023-11-17T05:33:47.837 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:47.835+0000 7f6e36d06700 1 -- 172.21.15.160:0/1027357954 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e30105cf0 con 0x7f6e300f5410 2023-11-17T05:33:47.837 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:47.835+0000 7f6e2d7fa700 1 -- 172.21.15.160:0/1027357954 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6e24013f00 con 0x7f6e300f5410 2023-11-17T05:33:47.837 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:47.836+0000 7f6e2d7fa700 1 -- 172.21.15.160:0/1027357954 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e24014d90 con 0x7f6e300f5410 2023-11-17T05:33:47.838 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:47.836+0000 7f6e36d06700 1 -- 172.21.15.160:0/1027357954 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e30106180 con 0x7f6e300f5410 2023-11-17T05:33:47.838 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:47.836+0000 7f6e2d7fa700 1 -- 172.21.15.160:0/1027357954 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6e240095f0 con 0x7f6e300f5410 2023-11-17T05:33:47.838 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:47.836+0000 7f6e36d06700 1 -- 172.21.15.160:0/1027357954 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e30045ad0 con 0x7f6e300f5410 2023-11-17T05:33:47.839 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:47.837+0000 7f6e2d7fa700 1 --2- 172.21.15.160:0/1027357954 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f6e2002ff60 0x7f6e20032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:33:47.839 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:47.837+0000 7f6e2d7fa700 1 -- 172.21.15.160:0/1027357954 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6e24040fe0 con 0x7f6e300f5410 2023-11-17T05:33:47.839 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:47.837+0000 7f6e2ffff700 1 --2- 172.21.15.160:0/1027357954 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f6e2002ff60 0x7f6e20032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:33:47.839 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:47.837+0000 7f6e2ffff700 1 --2- 172.21.15.160:0/1027357954 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f6e2002ff60 0x7f6e20032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f6e1c0096a0 tx=0x7f6e1c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:33:47.842 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:47.842+0000 7f6e2d7fa700 1 -- 172.21.15.160:0/1027357954 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6e2401d7f0 con 0x7f6e300f5410 2023-11-17T05:33:48.046 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:48.045+0000 7f6e36d06700 1 -- 172.21.15.160:0/1027357954 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6e300fae60 con 0x7f6e300f5410 2023-11-17T05:33:48.047 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:48.046+0000 7f6e2d7fa700 1 -- 172.21.15.160:0/1027357954 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f6e24010230 con 0x7f6e300f5410 2023-11-17T05:33:48.047 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:33:48.047 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:33:48.049 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:48.048+0000 7f6e36d06700 1 -- 172.21.15.160:0/1027357954 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f6e2002ff60 msgr2=0x7f6e20032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:33:48.049 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:48.048+0000 7f6e36d06700 1 --2- 172.21.15.160:0/1027357954 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f6e2002ff60 0x7f6e20032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f6e1c0096a0 tx=0x7f6e1c006b40).stop 2023-11-17T05:33:48.049 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:48.048+0000 7f6e36d06700 1 -- 172.21.15.160:0/1027357954 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e300f5410 msgr2=0x7f6e30105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:33:48.049 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:48.048+0000 7f6e36d06700 1 --2- 172.21.15.160:0/1027357954 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e300f5410 0x7f6e30105600 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f6e24013510 tx=0x7f6e2400b040).stop 2023-11-17T05:33:48.049 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:48.048+0000 7f6e36d06700 1 -- 172.21.15.160:0/1027357954 shutdown_connections 2023-11-17T05:33:48.049 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:48.048+0000 7f6e36d06700 1 --2- 172.21.15.160:0/1027357954 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f6e2002ff60 0x7f6e20032420 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:33:48.049 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:48.048+0000 7f6e36d06700 1 --2- 172.21.15.160:0/1027357954 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e300f5410 0x7f6e30105600 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:33:48.049 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:48.048+0000 7f6e36d06700 1 -- 172.21.15.160:0/1027357954 >> 172.21.15.160:0/1027357954 conn(0x7f6e300f0b70 msgr2=0x7f6e300f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:33:48.049 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:48.048+0000 7f6e36d06700 1 -- 172.21.15.160:0/1027357954 shutdown_connections 2023-11-17T05:33:48.049 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:48.048+0000 7f6e36d06700 1 -- 172.21.15.160:0/1027357954 wait complete. 2023-11-17T05:33:48.050 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:33:49.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:33:48 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:33:47.469992+0000 mgr.smithi003.eddcuc (mgr.14164) 258 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:49.471 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:33:48 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:33:48.047286+0000 mon.smithi003 (mon.0) 234 : audit [DBG] from='client.? 172.21.15.160:0/1027357954' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:33:49.476 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:33:49.476 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:33:50.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:33:49 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:33:49.470387+0000 mgr.smithi003.eddcuc (mgr.14164) 259 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:50.679 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:50.673+0000 7fc833e81700 1 -- 172.21.15.160:0/3940677741 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc82c0f5410 msgr2=0x7fc82c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:33:50.679 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:50.673+0000 7fc833e81700 1 --2- 172.21.15.160:0/3940677741 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc82c0f5410 0x7fc82c0f5830 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7fc81c009fb0 tx=0x7fc81c009b30).stop 2023-11-17T05:33:50.679 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:50.673+0000 7fc833e81700 1 -- 172.21.15.160:0/3940677741 shutdown_connections 2023-11-17T05:33:50.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:50.673+0000 7fc833e81700 1 --2- 172.21.15.160:0/3940677741 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc82c0f5410 0x7fc82c0f5830 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:33:50.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:50.673+0000 7fc833e81700 1 -- 172.21.15.160:0/3940677741 >> 172.21.15.160:0/3940677741 conn(0x7fc82c0f0b70 msgr2=0x7fc82c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:33:50.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:50.674+0000 7fc833e81700 1 -- 172.21.15.160:0/3940677741 shutdown_connections 2023-11-17T05:33:50.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:50.674+0000 7fc833e81700 1 -- 172.21.15.160:0/3940677741 wait complete. 2023-11-17T05:33:50.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:50.674+0000 7fc833e81700 1 Processor -- start 2023-11-17T05:33:50.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:50.674+0000 7fc833e81700 1 -- start start 2023-11-17T05:33:50.681 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:50.674+0000 7fc833e81700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc82c0f5410 0x7fc82c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:33:50.681 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:50.674+0000 7fc833e81700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc82c105b20 con 0x7fc82c0f5410 2023-11-17T05:33:50.681 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:50.675+0000 7fc831c1d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc82c0f5410 0x7fc82c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:33:50.681 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:50.675+0000 7fc831c1d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc82c0f5410 0x7fc82c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:49548/0 (socket says 172.21.15.160:49548) 2023-11-17T05:33:50.681 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:50.675+0000 7fc831c1d700 1 -- 172.21.15.160:0/2287962469 learned_addr learned my addr 172.21.15.160:0/2287962469 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:33:50.682 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:50.675+0000 7fc831c1d700 1 -- 172.21.15.160:0/2287962469 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc81c005040 con 0x7fc82c0f5410 2023-11-17T05:33:50.686 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:50.675+0000 7fc831c1d700 1 --2- 172.21.15.160:0/2287962469 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc82c0f5410 0x7fc82c105630 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7fc81c013510 tx=0x7fc81c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:33:50.687 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:50.675+0000 7fc8227fc700 1 -- 172.21.15.160:0/2287962469 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc81c009320 con 0x7fc82c0f5410 2023-11-17T05:33:50.687 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:50.675+0000 7fc833e81700 1 -- 172.21.15.160:0/2287962469 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc82c105d20 con 0x7fc82c0f5410 2023-11-17T05:33:50.687 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:50.675+0000 7fc8227fc700 1 -- 172.21.15.160:0/2287962469 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc81c013f00 con 0x7fc82c0f5410 2023-11-17T05:33:50.687 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:50.675+0000 7fc8227fc700 1 -- 172.21.15.160:0/2287962469 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc81c014d90 con 0x7fc82c0f5410 2023-11-17T05:33:50.687 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:50.675+0000 7fc833e81700 1 -- 172.21.15.160:0/2287962469 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc82c1061b0 con 0x7fc82c0f5410 2023-11-17T05:33:50.688 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:50.676+0000 7fc8227fc700 1 -- 172.21.15.160:0/2287962469 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc81c0263a0 con 0x7fc82c0f5410 2023-11-17T05:33:50.688 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:50.676+0000 7fc8227fc700 1 --2- 172.21.15.160:0/2287962469 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc81802ff60 0x7fc818032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:33:50.688 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:50.676+0000 7fc8227fc700 1 -- 172.21.15.160:0/2287962469 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc81c0410c0 con 0x7fc82c0f5410 2023-11-17T05:33:50.688 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:50.676+0000 7fc833e81700 1 -- 172.21.15.160:0/2287962469 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc82c045ad0 con 0x7fc82c0f5410 2023-11-17T05:33:50.688 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:50.677+0000 7fc83141c700 1 --2- 172.21.15.160:0/2287962469 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc81802ff60 0x7fc818032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:33:50.688 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:50.677+0000 7fc83141c700 1 --2- 172.21.15.160:0/2287962469 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc81802ff60 0x7fc818032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fc8280096a0 tx=0x7fc828006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:33:50.689 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:50.681+0000 7fc8227fc700 1 -- 172.21.15.160:0/2287962469 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc81c012920 con 0x7fc82c0f5410 2023-11-17T05:33:50.887 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:50.886+0000 7fc833e81700 1 -- 172.21.15.160:0/2287962469 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc82c106ed0 con 0x7fc82c0f5410 2023-11-17T05:33:50.887 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:50.887+0000 7fc8227fc700 1 -- 172.21.15.160:0/2287962469 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fc81c02b020 con 0x7fc82c0f5410 2023-11-17T05:33:50.888 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:33:50.888 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:33:50.889 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:50.888+0000 7fc833e81700 1 -- 172.21.15.160:0/2287962469 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc81802ff60 msgr2=0x7fc818032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:33:50.889 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:50.888+0000 7fc833e81700 1 --2- 172.21.15.160:0/2287962469 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc81802ff60 0x7fc818032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fc8280096a0 tx=0x7fc828006b40).stop 2023-11-17T05:33:50.889 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:50.888+0000 7fc833e81700 1 -- 172.21.15.160:0/2287962469 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc82c0f5410 msgr2=0x7fc82c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:33:50.890 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:50.888+0000 7fc833e81700 1 --2- 172.21.15.160:0/2287962469 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc82c0f5410 0x7fc82c105630 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7fc81c013510 tx=0x7fc81c00b040).stop 2023-11-17T05:33:50.890 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:50.888+0000 7fc833e81700 1 -- 172.21.15.160:0/2287962469 shutdown_connections 2023-11-17T05:33:50.890 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:50.888+0000 7fc833e81700 1 --2- 172.21.15.160:0/2287962469 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc81802ff60 0x7fc818032420 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:33:50.890 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:50.888+0000 7fc833e81700 1 --2- 172.21.15.160:0/2287962469 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc82c0f5410 0x7fc82c105630 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:33:50.890 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:50.888+0000 7fc833e81700 1 -- 172.21.15.160:0/2287962469 >> 172.21.15.160:0/2287962469 conn(0x7fc82c0f0b70 msgr2=0x7fc82c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:33:50.890 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:50.888+0000 7fc833e81700 1 -- 172.21.15.160:0/2287962469 shutdown_connections 2023-11-17T05:33:50.890 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:50.888+0000 7fc833e81700 1 -- 172.21.15.160:0/2287962469 wait complete. 2023-11-17T05:33:50.890 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:33:51.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:33:50 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:33:50.887874+0000 mon.smithi003 (mon.0) 235 : audit [DBG] from='client.? 172.21.15.160:0/2287962469' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:33:52.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:33:51 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:33:51.470774+0000 mgr.smithi003.eddcuc (mgr.14164) 260 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:52.319 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:33:52.320 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:33:53.521 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:53.520+0000 7f3f62903700 1 -- 172.21.15.160:0/2749025194 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f5c060280 msgr2=0x7f3f5c0f90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:33:53.522 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:53.520+0000 7f3f62903700 1 --2- 172.21.15.160:0/2749025194 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f5c060280 0x7f3f5c0f90d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f3f4c009fb0 tx=0x7f3f4c009b30).stop 2023-11-17T05:33:53.522 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:53.520+0000 7f3f62903700 1 -- 172.21.15.160:0/2749025194 shutdown_connections 2023-11-17T05:33:53.522 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:53.520+0000 7f3f62903700 1 --2- 172.21.15.160:0/2749025194 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f5c060280 0x7f3f5c0f90d0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:33:53.522 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:53.521+0000 7f3f62903700 1 -- 172.21.15.160:0/2749025194 >> 172.21.15.160:0/2749025194 conn(0x7f3f5c0f0b70 msgr2=0x7f3f5c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:33:53.522 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:53.521+0000 7f3f62903700 1 -- 172.21.15.160:0/2749025194 shutdown_connections 2023-11-17T05:33:53.522 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:53.521+0000 7f3f62903700 1 -- 172.21.15.160:0/2749025194 wait complete. 2023-11-17T05:33:53.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:53.521+0000 7f3f62903700 1 Processor -- start 2023-11-17T05:33:53.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:53.521+0000 7f3f62903700 1 -- start start 2023-11-17T05:33:53.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:53.521+0000 7f3f62903700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f5c060280 0x7f3f5c106690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:33:53.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:53.521+0000 7f3f62903700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5c106b80 con 0x7f3f5c060280 2023-11-17T05:33:53.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:53.522+0000 7f3f5bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f5c060280 0x7f3f5c106690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:33:53.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:53.522+0000 7f3f5bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f5c060280 0x7f3f5c106690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:49570/0 (socket says 172.21.15.160:49570) 2023-11-17T05:33:53.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:53.522+0000 7f3f5bfff700 1 -- 172.21.15.160:0/4034220921 learned_addr learned my addr 172.21.15.160:0/4034220921 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:33:53.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:53.522+0000 7f3f5bfff700 1 -- 172.21.15.160:0/4034220921 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f4c005040 con 0x7f3f5c060280 2023-11-17T05:33:53.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:53.522+0000 7f3f5bfff700 1 --2- 172.21.15.160:0/4034220921 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f5c060280 0x7f3f5c106690 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f3f4c013510 tx=0x7f3f4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:33:53.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:53.523+0000 7f3f58ff9700 1 -- 172.21.15.160:0/4034220921 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f4c009320 con 0x7f3f5c060280 2023-11-17T05:33:53.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:53.523+0000 7f3f58ff9700 1 -- 172.21.15.160:0/4034220921 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3f4c013f00 con 0x7f3f5c060280 2023-11-17T05:33:53.524 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:53.523+0000 7f3f62903700 1 -- 172.21.15.160:0/4034220921 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f5c106d80 con 0x7f3f5c060280 2023-11-17T05:33:53.525 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:53.523+0000 7f3f58ff9700 1 -- 172.21.15.160:0/4034220921 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f4c014dc0 con 0x7f3f5c060280 2023-11-17T05:33:53.525 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:53.523+0000 7f3f62903700 1 -- 172.21.15.160:0/4034220921 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f5c107210 con 0x7f3f5c060280 2023-11-17T05:33:53.525 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:53.524+0000 7f3f58ff9700 1 -- 172.21.15.160:0/4034220921 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3f4c014420 con 0x7f3f5c060280 2023-11-17T05:33:53.526 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:53.524+0000 7f3f62903700 1 -- 172.21.15.160:0/4034220921 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f5c045ad0 con 0x7f3f5c060280 2023-11-17T05:33:53.526 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:53.524+0000 7f3f58ff9700 1 --2- 172.21.15.160:0/4034220921 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f3f4402ffb0 0x7f3f44032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:33:53.526 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:53.524+0000 7f3f58ff9700 1 -- 172.21.15.160:0/4034220921 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3f4c040ff0 con 0x7f3f5c060280 2023-11-17T05:33:53.526 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:53.524+0000 7f3f5b7fe700 1 --2- 172.21.15.160:0/4034220921 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f3f4402ffb0 0x7f3f44032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:33:53.526 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:53.524+0000 7f3f5b7fe700 1 --2- 172.21.15.160:0/4034220921 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f3f4402ffb0 0x7f3f44032470 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f3f500096a0 tx=0x7f3f50006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:33:53.529 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:53.529+0000 7f3f58ff9700 1 -- 172.21.15.160:0/4034220921 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3f4c06d2b0 con 0x7f3f5c060280 2023-11-17T05:33:53.733 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:53.732+0000 7f3f62903700 1 -- 172.21.15.160:0/4034220921 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3f5c1080c0 con 0x7f3f5c060280 2023-11-17T05:33:53.734 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:53.733+0000 7f3f58ff9700 1 -- 172.21.15.160:0/4034220921 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f3f4c0095f0 con 0x7f3f5c060280 2023-11-17T05:33:53.734 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:33:53.734 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:33:53.736 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:53.735+0000 7f3f62903700 1 -- 172.21.15.160:0/4034220921 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f3f4402ffb0 msgr2=0x7f3f44032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:33:53.736 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:53.735+0000 7f3f62903700 1 --2- 172.21.15.160:0/4034220921 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f3f4402ffb0 0x7f3f44032470 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f3f500096a0 tx=0x7f3f50006b40).stop 2023-11-17T05:33:53.736 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:53.735+0000 7f3f62903700 1 -- 172.21.15.160:0/4034220921 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f5c060280 msgr2=0x7f3f5c106690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:33:53.736 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:53.735+0000 7f3f62903700 1 --2- 172.21.15.160:0/4034220921 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f5c060280 0x7f3f5c106690 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f3f4c013510 tx=0x7f3f4c00b040).stop 2023-11-17T05:33:53.736 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:53.735+0000 7f3f62903700 1 -- 172.21.15.160:0/4034220921 shutdown_connections 2023-11-17T05:33:53.736 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:53.735+0000 7f3f62903700 1 --2- 172.21.15.160:0/4034220921 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f3f4402ffb0 0x7f3f44032470 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:33:53.737 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:53.735+0000 7f3f62903700 1 --2- 172.21.15.160:0/4034220921 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f5c060280 0x7f3f5c106690 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:33:53.737 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:53.735+0000 7f3f62903700 1 -- 172.21.15.160:0/4034220921 >> 172.21.15.160:0/4034220921 conn(0x7f3f5c0f0b70 msgr2=0x7f3f5c0f1600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:33:53.737 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:53.735+0000 7f3f62903700 1 -- 172.21.15.160:0/4034220921 shutdown_connections 2023-11-17T05:33:53.737 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:53.735+0000 7f3f62903700 1 -- 172.21.15.160:0/4034220921 wait complete. 2023-11-17T05:33:53.737 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:33:55.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:33:54 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:33:53.471078+0000 mgr.smithi003.eddcuc (mgr.14164) 261 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:55.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:33:54 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:33:53.734343+0000 mon.smithi003 (mon.0) 236 : audit [DBG] from='client.? 172.21.15.160:0/4034220921' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:33:55.141 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:33:55.141 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:33:56.399 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:56.397+0000 7fce8d299700 1 -- 172.21.15.160:0/3534960471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fce880f5410 msgr2=0x7fce880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:33:56.399 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:56.397+0000 7fce8d299700 1 --2- 172.21.15.160:0/3534960471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fce880f5410 0x7fce880f5830 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7fce70009fb0 tx=0x7fce70009b30).stop 2023-11-17T05:33:56.399 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:56.397+0000 7fce8d299700 1 -- 172.21.15.160:0/3534960471 shutdown_connections 2023-11-17T05:33:56.399 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:56.397+0000 7fce8d299700 1 --2- 172.21.15.160:0/3534960471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fce880f5410 0x7fce880f5830 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:33:56.399 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:56.397+0000 7fce8d299700 1 -- 172.21.15.160:0/3534960471 >> 172.21.15.160:0/3534960471 conn(0x7fce880f0b70 msgr2=0x7fce880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:33:56.399 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:56.397+0000 7fce8d299700 1 -- 172.21.15.160:0/3534960471 shutdown_connections 2023-11-17T05:33:56.399 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:56.397+0000 7fce8d299700 1 -- 172.21.15.160:0/3534960471 wait complete. 2023-11-17T05:33:56.400 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:56.398+0000 7fce8d299700 1 Processor -- start 2023-11-17T05:33:56.400 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:56.398+0000 7fce8d299700 1 -- start start 2023-11-17T05:33:56.400 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:56.398+0000 7fce8d299700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fce880f5410 0x7fce88106760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:33:56.400 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:56.398+0000 7fce8d299700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce88106c50 con 0x7fce880f5410 2023-11-17T05:33:56.401 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:56.398+0000 7fce86ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fce880f5410 0x7fce88106760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:33:56.402 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:56.398+0000 7fce86ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fce880f5410 0x7fce88106760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:49586/0 (socket says 172.21.15.160:49586) 2023-11-17T05:33:56.402 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:56.398+0000 7fce86ffd700 1 -- 172.21.15.160:0/746087697 learned_addr learned my addr 172.21.15.160:0/746087697 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:33:56.402 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:56.399+0000 7fce86ffd700 1 -- 172.21.15.160:0/746087697 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce70005040 con 0x7fce880f5410 2023-11-17T05:33:56.402 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:56.399+0000 7fce86ffd700 1 --2- 172.21.15.160:0/746087697 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fce880f5410 0x7fce88106760 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fce70009f80 tx=0x7fce7000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:33:56.402 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:56.399+0000 7fce7f7fe700 1 -- 172.21.15.160:0/746087697 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce70009320 con 0x7fce880f5410 2023-11-17T05:33:56.402 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:56.399+0000 7fce8d299700 1 -- 172.21.15.160:0/746087697 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce88106e50 con 0x7fce880f5410 2023-11-17T05:33:56.402 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:56.399+0000 7fce7f7fe700 1 -- 172.21.15.160:0/746087697 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce70013ec0 con 0x7fce880f5410 2023-11-17T05:33:56.402 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:56.399+0000 7fce7f7fe700 1 -- 172.21.15.160:0/746087697 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce7001ece0 con 0x7fce880f5410 2023-11-17T05:33:56.403 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:56.399+0000 7fce8d299700 1 -- 172.21.15.160:0/746087697 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce881072e0 con 0x7fce880f5410 2023-11-17T05:33:56.403 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:56.400+0000 7fce7f7fe700 1 -- 172.21.15.160:0/746087697 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fce7001e420 con 0x7fce880f5410 2023-11-17T05:33:56.403 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:56.400+0000 7fce8d299700 1 -- 172.21.15.160:0/746087697 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce88045ad0 con 0x7fce880f5410 2023-11-17T05:33:56.404 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:56.400+0000 7fce7f7fe700 1 --2- 172.21.15.160:0/746087697 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fce7402ff60 0x7fce74032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:33:56.404 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:56.400+0000 7fce7f7fe700 1 -- 172.21.15.160:0/746087697 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fce70040d70 con 0x7fce880f5410 2023-11-17T05:33:56.404 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:56.400+0000 7fce867fc700 1 --2- 172.21.15.160:0/746087697 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fce7402ff60 0x7fce74032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:33:56.404 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:56.401+0000 7fce867fc700 1 --2- 172.21.15.160:0/746087697 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fce7402ff60 0x7fce74032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fce780096a0 tx=0x7fce78006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:33:56.406 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:56.405+0000 7fce7f7fe700 1 -- 172.21.15.160:0/746087697 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fce7001e730 con 0x7fce880f5410 2023-11-17T05:33:56.609 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:56.608+0000 7fce8d299700 1 -- 172.21.15.160:0/746087697 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fce88108190 con 0x7fce880f5410 2023-11-17T05:33:56.610 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:56.609+0000 7fce7f7fe700 1 -- 172.21.15.160:0/746087697 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fce7002c2c0 con 0x7fce880f5410 2023-11-17T05:33:56.610 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:33:56.610 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:33:56.612 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:56.611+0000 7fce8d299700 1 -- 172.21.15.160:0/746087697 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fce7402ff60 msgr2=0x7fce74032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:33:56.612 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:56.611+0000 7fce8d299700 1 --2- 172.21.15.160:0/746087697 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fce7402ff60 0x7fce74032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fce780096a0 tx=0x7fce78006b40).stop 2023-11-17T05:33:56.612 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:56.611+0000 7fce8d299700 1 -- 172.21.15.160:0/746087697 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fce880f5410 msgr2=0x7fce88106760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:33:56.612 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:56.611+0000 7fce8d299700 1 --2- 172.21.15.160:0/746087697 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fce880f5410 0x7fce88106760 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fce70009f80 tx=0x7fce7000b040).stop 2023-11-17T05:33:56.612 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:56.611+0000 7fce8d299700 1 -- 172.21.15.160:0/746087697 shutdown_connections 2023-11-17T05:33:56.612 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:56.611+0000 7fce8d299700 1 --2- 172.21.15.160:0/746087697 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fce7402ff60 0x7fce74032420 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:33:56.612 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:56.611+0000 7fce8d299700 1 --2- 172.21.15.160:0/746087697 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fce880f5410 0x7fce88106760 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:33:56.612 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:56.611+0000 7fce8d299700 1 -- 172.21.15.160:0/746087697 >> 172.21.15.160:0/746087697 conn(0x7fce880f0b70 msgr2=0x7fce8805fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:33:56.612 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:56.611+0000 7fce8d299700 1 -- 172.21.15.160:0/746087697 shutdown_connections 2023-11-17T05:33:56.612 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:56.611+0000 7fce8d299700 1 -- 172.21.15.160:0/746087697 wait complete. 2023-11-17T05:33:56.613 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:33:57.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:33:56 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:33:55.471378+0000 mgr.smithi003.eddcuc (mgr.14164) 262 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:58.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:33:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:33:56.609771+0000 mon.smithi003 (mon.0) 237 : audit [DBG] from='client.? 172.21.15.160:0/746087697' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:33:58.162 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:33:58.162 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:33:59.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:33:58 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:33:57.471737+0000 mgr.smithi003.eddcuc (mgr.14164) 263 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:33:59.419 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:59.418+0000 7fc7019ac700 1 -- 172.21.15.160:0/3878937775 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6fc0f6510 msgr2=0x7fc6fc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:33:59.419 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:59.418+0000 7fc7019ac700 1 --2- 172.21.15.160:0/3878937775 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6fc0f6510 0x7fc6fc0f6930 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7fc6f0009fb0 tx=0x7fc6f0009b30).stop 2023-11-17T05:33:59.419 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:59.418+0000 7fc7019ac700 1 -- 172.21.15.160:0/3878937775 shutdown_connections 2023-11-17T05:33:59.419 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:59.418+0000 7fc7019ac700 1 --2- 172.21.15.160:0/3878937775 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6fc0f6510 0x7fc6fc0f6930 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:33:59.420 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:59.418+0000 7fc7019ac700 1 -- 172.21.15.160:0/3878937775 >> 172.21.15.160:0/3878937775 conn(0x7fc6fc0f1c70 msgr2=0x7fc6fc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:33:59.420 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:59.418+0000 7fc7019ac700 1 -- 172.21.15.160:0/3878937775 shutdown_connections 2023-11-17T05:33:59.420 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:59.418+0000 7fc7019ac700 1 -- 172.21.15.160:0/3878937775 wait complete. 2023-11-17T05:33:59.421 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:59.419+0000 7fc7019ac700 1 Processor -- start 2023-11-17T05:33:59.421 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:59.419+0000 7fc7019ac700 1 -- start start 2023-11-17T05:33:59.421 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:59.419+0000 7fc7019ac700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6fc0f6510 0x7fc6fc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:33:59.421 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:59.419+0000 7fc7019ac700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6fc106c20 con 0x7fc6fc0f6510 2023-11-17T05:33:59.421 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:59.419+0000 7fc6faffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6fc0f6510 0x7fc6fc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:33:59.421 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:59.419+0000 7fc6faffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6fc0f6510 0x7fc6fc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:49610/0 (socket says 172.21.15.160:49610) 2023-11-17T05:33:59.421 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:59.419+0000 7fc6faffd700 1 -- 172.21.15.160:0/399639058 learned_addr learned my addr 172.21.15.160:0/399639058 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:33:59.422 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:59.420+0000 7fc6faffd700 1 -- 172.21.15.160:0/399639058 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f0005040 con 0x7fc6fc0f6510 2023-11-17T05:33:59.422 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:59.420+0000 7fc6faffd700 1 --2- 172.21.15.160:0/399639058 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6fc0f6510 0x7fc6fc106730 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7fc6f0013510 tx=0x7fc6f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:33:59.422 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:59.420+0000 7fc6e37fe700 1 -- 172.21.15.160:0/399639058 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc6f0009320 con 0x7fc6fc0f6510 2023-11-17T05:33:59.422 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:59.420+0000 7fc7019ac700 1 -- 172.21.15.160:0/399639058 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6fc106e20 con 0x7fc6fc0f6510 2023-11-17T05:33:59.422 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:59.420+0000 7fc6e37fe700 1 -- 172.21.15.160:0/399639058 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc6f0013f00 con 0x7fc6fc0f6510 2023-11-17T05:33:59.422 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:59.420+0000 7fc6e37fe700 1 -- 172.21.15.160:0/399639058 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc6f0014d90 con 0x7fc6fc0f6510 2023-11-17T05:33:59.422 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:59.420+0000 7fc7019ac700 1 -- 172.21.15.160:0/399639058 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6fc1072b0 con 0x7fc6fc0f6510 2023-11-17T05:33:59.423 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:59.421+0000 7fc6e37fe700 1 -- 172.21.15.160:0/399639058 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc6f0014420 con 0x7fc6fc0f6510 2023-11-17T05:33:59.423 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:59.421+0000 7fc7019ac700 1 -- 172.21.15.160:0/399639058 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6fc045ad0 con 0x7fc6fc0f6510 2023-11-17T05:33:59.423 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:59.421+0000 7fc6e37fe700 1 --2- 172.21.15.160:0/399639058 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc6e402ff60 0x7fc6e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:33:59.423 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:59.421+0000 7fc6e37fe700 1 -- 172.21.15.160:0/399639058 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc6f0040fb0 con 0x7fc6fc0f6510 2023-11-17T05:33:59.423 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:59.421+0000 7fc6fa7fc700 1 --2- 172.21.15.160:0/399639058 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc6e402ff60 0x7fc6e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:33:59.423 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:59.422+0000 7fc6fa7fc700 1 --2- 172.21.15.160:0/399639058 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc6e402ff60 0x7fc6e4032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fc6ec0096a0 tx=0x7fc6ec006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:33:59.426 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:59.426+0000 7fc6e37fe700 1 -- 172.21.15.160:0/399639058 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc6f00095f0 con 0x7fc6fc0f6510 2023-11-17T05:33:59.630 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:59.630+0000 7fc7019ac700 1 -- 172.21.15.160:0/399639058 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc6fc0fbf60 con 0x7fc6fc0f6510 2023-11-17T05:33:59.631 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:59.630+0000 7fc6e37fe700 1 -- 172.21.15.160:0/399639058 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fc6f001db70 con 0x7fc6fc0f6510 2023-11-17T05:33:59.631 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:33:59.631 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:33:59.632 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:59.632+0000 7fc7019ac700 1 -- 172.21.15.160:0/399639058 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc6e402ff60 msgr2=0x7fc6e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:33:59.633 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:59.632+0000 7fc7019ac700 1 --2- 172.21.15.160:0/399639058 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc6e402ff60 0x7fc6e4032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fc6ec0096a0 tx=0x7fc6ec006b40).stop 2023-11-17T05:33:59.633 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:59.632+0000 7fc7019ac700 1 -- 172.21.15.160:0/399639058 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6fc0f6510 msgr2=0x7fc6fc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:33:59.633 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:59.632+0000 7fc7019ac700 1 --2- 172.21.15.160:0/399639058 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6fc0f6510 0x7fc6fc106730 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7fc6f0013510 tx=0x7fc6f000b040).stop 2023-11-17T05:33:59.633 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:59.632+0000 7fc7019ac700 1 -- 172.21.15.160:0/399639058 shutdown_connections 2023-11-17T05:33:59.633 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:59.632+0000 7fc7019ac700 1 --2- 172.21.15.160:0/399639058 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc6e402ff60 0x7fc6e4032420 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:33:59.633 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:59.632+0000 7fc7019ac700 1 --2- 172.21.15.160:0/399639058 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6fc0f6510 0x7fc6fc106730 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:33:59.633 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:59.632+0000 7fc7019ac700 1 -- 172.21.15.160:0/399639058 >> 172.21.15.160:0/399639058 conn(0x7fc6fc0f1c70 msgr2=0x7fc6fc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:33:59.633 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:59.632+0000 7fc7019ac700 1 -- 172.21.15.160:0/399639058 shutdown_connections 2023-11-17T05:33:59.633 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:33:59.632+0000 7fc7019ac700 1 -- 172.21.15.160:0/399639058 wait complete. 2023-11-17T05:33:59.634 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:34:00.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:33:59 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:33:59.472153+0000 mgr.smithi003.eddcuc (mgr.14164) 264 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:00.271 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:33:59 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:33:59.631425+0000 mon.smithi003 (mon.0) 238 : audit [DBG] from='client.? 172.21.15.160:0/399639058' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:34:01.030 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:34:01.033 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:34:02.159 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:02.151+0000 7faee9fd6700 1 -- 172.21.15.160:0/2887456131 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faee40f5410 msgr2=0x7faee40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:02.160 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:02.151+0000 7faee9fd6700 1 --2- 172.21.15.160:0/2887456131 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faee40f5410 0x7faee40f5830 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7faed8009fb0 tx=0x7faed8009b30).stop 2023-11-17T05:34:02.160 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:02.151+0000 7faee9fd6700 1 -- 172.21.15.160:0/2887456131 shutdown_connections 2023-11-17T05:34:02.160 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:02.151+0000 7faee9fd6700 1 --2- 172.21.15.160:0/2887456131 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faee40f5410 0x7faee40f5830 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:02.160 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:02.151+0000 7faee9fd6700 1 -- 172.21.15.160:0/2887456131 >> 172.21.15.160:0/2887456131 conn(0x7faee40f0b70 msgr2=0x7faee40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:34:02.160 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:02.151+0000 7faee9fd6700 1 -- 172.21.15.160:0/2887456131 shutdown_connections 2023-11-17T05:34:02.160 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:02.151+0000 7faee9fd6700 1 -- 172.21.15.160:0/2887456131 wait complete. 2023-11-17T05:34:02.160 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:02.152+0000 7faee9fd6700 1 Processor -- start 2023-11-17T05:34:02.160 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:02.152+0000 7faee9fd6700 1 -- start start 2023-11-17T05:34:02.160 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:02.152+0000 7faee9fd6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faee40f5410 0x7faee4109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:34:02.160 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:02.152+0000 7faee9fd6700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee4060220 con 0x7faee40f5410 2023-11-17T05:34:02.160 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:02.152+0000 7faee37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faee40f5410 0x7faee4109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:34:02.160 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:02.152+0000 7faee37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faee40f5410 0x7faee4109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:36798/0 (socket says 172.21.15.160:36798) 2023-11-17T05:34:02.160 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:02.152+0000 7faee37fe700 1 -- 172.21.15.160:0/2579688309 learned_addr learned my addr 172.21.15.160:0/2579688309 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:34:02.161 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:02.153+0000 7faee37fe700 1 -- 172.21.15.160:0/2579688309 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faed8005040 con 0x7faee40f5410 2023-11-17T05:34:02.161 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:02.153+0000 7faee37fe700 1 --2- 172.21.15.160:0/2579688309 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faee40f5410 0x7faee4109450 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7faed8013510 tx=0x7faed800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:34:02.161 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:02.153+0000 7faecbfff700 1 -- 172.21.15.160:0/2579688309 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faed8009320 con 0x7faee40f5410 2023-11-17T05:34:02.161 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:02.153+0000 7faee9fd6700 1 -- 172.21.15.160:0/2579688309 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faee4060420 con 0x7faee40f5410 2023-11-17T05:34:02.161 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:02.153+0000 7faee9fd6700 1 -- 172.21.15.160:0/2579688309 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faee4109b00 con 0x7faee40f5410 2023-11-17T05:34:02.161 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:02.153+0000 7faecbfff700 1 -- 172.21.15.160:0/2579688309 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faed8013f00 con 0x7faee40f5410 2023-11-17T05:34:02.161 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:02.153+0000 7faecbfff700 1 -- 172.21.15.160:0/2579688309 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faed8014d60 con 0x7faee40f5410 2023-11-17T05:34:02.161 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:02.154+0000 7faecbfff700 1 -- 172.21.15.160:0/2579688309 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faed8014420 con 0x7faee40f5410 2023-11-17T05:34:02.161 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:02.154+0000 7faee9fd6700 1 -- 172.21.15.160:0/2579688309 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faee4059100 con 0x7faee40f5410 2023-11-17T05:34:02.161 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:02.154+0000 7faecbfff700 1 --2- 172.21.15.160:0/2579688309 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7faecc02ff60 0x7faecc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:34:02.161 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:02.154+0000 7faecbfff700 1 -- 172.21.15.160:0/2579688309 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faed8041390 con 0x7faee40f5410 2023-11-17T05:34:02.161 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:02.154+0000 7faee2ffd700 1 --2- 172.21.15.160:0/2579688309 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7faecc02ff60 0x7faecc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:34:02.161 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:02.155+0000 7faee2ffd700 1 --2- 172.21.15.160:0/2579688309 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7faecc02ff60 0x7faecc032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7faed40096a0 tx=0x7faed4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:34:02.162 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:02.159+0000 7faecbfff700 1 -- 172.21.15.160:0/2579688309 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faed8012b20 con 0x7faee40f5410 2023-11-17T05:34:02.365 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:02.365+0000 7faee9fd6700 1 -- 172.21.15.160:0/2579688309 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faee4045ad0 con 0x7faee40f5410 2023-11-17T05:34:02.366 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:02.365+0000 7faecbfff700 1 -- 172.21.15.160:0/2579688309 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7faed8012b20 con 0x7faee40f5410 2023-11-17T05:34:02.366 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:34:02.366 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:34:02.368 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:02.367+0000 7faee9fd6700 1 -- 172.21.15.160:0/2579688309 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7faecc02ff60 msgr2=0x7faecc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:02.368 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:02.367+0000 7faee9fd6700 1 --2- 172.21.15.160:0/2579688309 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7faecc02ff60 0x7faecc032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7faed40096a0 tx=0x7faed4006b40).stop 2023-11-17T05:34:02.368 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:02.367+0000 7faee9fd6700 1 -- 172.21.15.160:0/2579688309 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faee40f5410 msgr2=0x7faee4109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:02.368 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:02.367+0000 7faee9fd6700 1 --2- 172.21.15.160:0/2579688309 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faee40f5410 0x7faee4109450 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7faed8013510 tx=0x7faed800b040).stop 2023-11-17T05:34:02.368 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:02.367+0000 7faee9fd6700 1 -- 172.21.15.160:0/2579688309 shutdown_connections 2023-11-17T05:34:02.368 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:02.367+0000 7faee9fd6700 1 --2- 172.21.15.160:0/2579688309 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7faecc02ff60 0x7faecc032420 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:02.368 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:02.367+0000 7faee9fd6700 1 --2- 172.21.15.160:0/2579688309 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faee40f5410 0x7faee4109450 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:02.368 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:02.367+0000 7faee9fd6700 1 -- 172.21.15.160:0/2579688309 >> 172.21.15.160:0/2579688309 conn(0x7faee40f0b70 msgr2=0x7faee40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:34:02.368 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:02.367+0000 7faee9fd6700 1 -- 172.21.15.160:0/2579688309 shutdown_connections 2023-11-17T05:34:02.368 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:02.367+0000 7faee9fd6700 1 -- 172.21.15.160:0/2579688309 wait complete. 2023-11-17T05:34:02.369 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:34:03.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:02 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:34:01.472428+0000 mgr.smithi003.eddcuc (mgr.14164) 265 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:03.021 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:02 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:34:02.366479+0000 mon.smithi003 (mon.0) 239 : audit [DBG] from='client.? 172.21.15.160:0/2579688309' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:34:03.747 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:34:03.747 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:34:05.015 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:05.013+0000 7f224a194700 1 -- 172.21.15.160:0/481395299 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f22440f62a0 msgr2=0x7f22440f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:05.015 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:05.013+0000 7f224a194700 1 --2- 172.21.15.160:0/481395299 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f22440f62a0 0x7f22440f66c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f2234009fb0 tx=0x7f2234009b30).stop 2023-11-17T05:34:05.015 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:05.013+0000 7f224a194700 1 -- 172.21.15.160:0/481395299 shutdown_connections 2023-11-17T05:34:05.015 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:05.013+0000 7f224a194700 1 --2- 172.21.15.160:0/481395299 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f22440f62a0 0x7f22440f66c0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:05.016 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:05.013+0000 7f224a194700 1 -- 172.21.15.160:0/481395299 >> 172.21.15.160:0/481395299 conn(0x7f22440f1ac0 msgr2=0x7f22440f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:34:05.016 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:05.013+0000 7f224a194700 1 -- 172.21.15.160:0/481395299 shutdown_connections 2023-11-17T05:34:05.016 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:05.013+0000 7f224a194700 1 -- 172.21.15.160:0/481395299 wait complete. 2023-11-17T05:34:05.016 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:05.013+0000 7f224a194700 1 Processor -- start 2023-11-17T05:34:05.016 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:05.013+0000 7f224a194700 1 -- start start 2023-11-17T05:34:05.016 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:05.014+0000 7f224a194700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f22440f62a0 0x7f224406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:34:05.016 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:05.014+0000 7f224a194700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2244069e20 con 0x7f22440f62a0 2023-11-17T05:34:05.016 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:05.014+0000 7f2249192700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f22440f62a0 0x7f224406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:34:05.016 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:05.014+0000 7f2249192700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f22440f62a0 0x7f224406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:36818/0 (socket says 172.21.15.160:36818) 2023-11-17T05:34:05.016 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:05.014+0000 7f2249192700 1 -- 172.21.15.160:0/3205889678 learned_addr learned my addr 172.21.15.160:0/3205889678 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:34:05.016 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:05.014+0000 7f2249192700 1 -- 172.21.15.160:0/3205889678 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2234005040 con 0x7f22440f62a0 2023-11-17T05:34:05.016 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:05.014+0000 7f2249192700 1 --2- 172.21.15.160:0/3205889678 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f22440f62a0 0x7f224406b9a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f2234013510 tx=0x7f223400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:34:05.016 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:05.014+0000 7f2239ffb700 1 -- 172.21.15.160:0/3205889678 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2234009320 con 0x7f22440f62a0 2023-11-17T05:34:05.017 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:05.014+0000 7f224a194700 1 -- 172.21.15.160:0/3205889678 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2244069fc0 con 0x7f22440f62a0 2023-11-17T05:34:05.017 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:05.014+0000 7f2239ffb700 1 -- 172.21.15.160:0/3205889678 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2234013f00 con 0x7f22440f62a0 2023-11-17T05:34:05.017 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:05.014+0000 7f2239ffb700 1 -- 172.21.15.160:0/3205889678 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2234014dc0 con 0x7f22440f62a0 2023-11-17T05:34:05.017 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:05.015+0000 7f224a194700 1 -- 172.21.15.160:0/3205889678 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f224406a450 con 0x7f22440f62a0 2023-11-17T05:34:05.017 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:05.015+0000 7f2239ffb700 1 -- 172.21.15.160:0/3205889678 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f22340095f0 con 0x7f22440f62a0 2023-11-17T05:34:05.018 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:05.015+0000 7f224a194700 1 -- 172.21.15.160:0/3205889678 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2244059100 con 0x7f22440f62a0 2023-11-17T05:34:05.018 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:05.015+0000 7f2239ffb700 1 --2- 172.21.15.160:0/3205889678 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f223002ff60 0x7f2230032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:34:05.018 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:05.016+0000 7f2239ffb700 1 -- 172.21.15.160:0/3205889678 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2234041100 con 0x7f22440f62a0 2023-11-17T05:34:05.018 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:05.016+0000 7f2248991700 1 --2- 172.21.15.160:0/3205889678 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f223002ff60 0x7f2230032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:34:05.018 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:05.016+0000 7f2248991700 1 --2- 172.21.15.160:0/3205889678 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f223002ff60 0x7f2230032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f22400096a0 tx=0x7f2240006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:34:05.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:04 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:34:03.472676+0000 mgr.smithi003.eddcuc (mgr.14164) 266 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:05.021 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:05.020+0000 7f2239ffb700 1 -- 172.21.15.160:0/3205889678 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2234014420 con 0x7f22440f62a0 2023-11-17T05:34:05.228 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:05.227+0000 7f224a194700 1 -- 172.21.15.160:0/3205889678 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f22440fbce0 con 0x7f22440f62a0 2023-11-17T05:34:05.228 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:05.227+0000 7f2239ffb700 1 -- 172.21.15.160:0/3205889678 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f2234010230 con 0x7f22440f62a0 2023-11-17T05:34:05.228 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:34:05.228 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:34:05.230 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:05.229+0000 7f224a194700 1 -- 172.21.15.160:0/3205889678 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f223002ff60 msgr2=0x7f2230032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:05.230 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:05.229+0000 7f224a194700 1 --2- 172.21.15.160:0/3205889678 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f223002ff60 0x7f2230032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f22400096a0 tx=0x7f2240006b40).stop 2023-11-17T05:34:05.230 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:05.229+0000 7f224a194700 1 -- 172.21.15.160:0/3205889678 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f22440f62a0 msgr2=0x7f224406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:05.230 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:05.229+0000 7f224a194700 1 --2- 172.21.15.160:0/3205889678 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f22440f62a0 0x7f224406b9a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f2234013510 tx=0x7f223400b040).stop 2023-11-17T05:34:05.230 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:05.229+0000 7f224a194700 1 -- 172.21.15.160:0/3205889678 shutdown_connections 2023-11-17T05:34:05.230 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:05.229+0000 7f224a194700 1 --2- 172.21.15.160:0/3205889678 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f223002ff60 0x7f2230032420 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:05.230 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:05.229+0000 7f224a194700 1 --2- 172.21.15.160:0/3205889678 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f22440f62a0 0x7f224406b9a0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:05.231 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:05.229+0000 7f224a194700 1 -- 172.21.15.160:0/3205889678 >> 172.21.15.160:0/3205889678 conn(0x7f22440f1ac0 msgr2=0x7f22440f2530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:34:05.231 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:05.229+0000 7f224a194700 1 -- 172.21.15.160:0/3205889678 shutdown_connections 2023-11-17T05:34:05.231 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:05.229+0000 7f224a194700 1 -- 172.21.15.160:0/3205889678 wait complete. 2023-11-17T05:34:05.231 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:34:06.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:05 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:34:05.228673+0000 mon.smithi003 (mon.0) 240 : audit [DBG] from='client.? 172.21.15.160:0/3205889678' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:34:06.637 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:34:06.637 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:34:07.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:06 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:34:05.473056+0000 mgr.smithi003.eddcuc (mgr.14164) 267 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:07.844 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:07.843+0000 7f27402d7700 1 -- 172.21.15.160:0/2400548771 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27380f6510 msgr2=0x7f27380f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:07.844 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:07.843+0000 7f27402d7700 1 --2- 172.21.15.160:0/2400548771 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27380f6510 0x7f27380f6930 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f2734009fb0 tx=0x7f2734009b30).stop 2023-11-17T05:34:07.844 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:07.843+0000 7f27402d7700 1 -- 172.21.15.160:0/2400548771 shutdown_connections 2023-11-17T05:34:07.844 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:07.843+0000 7f27402d7700 1 --2- 172.21.15.160:0/2400548771 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27380f6510 0x7f27380f6930 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:07.844 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:07.843+0000 7f27402d7700 1 -- 172.21.15.160:0/2400548771 >> 172.21.15.160:0/2400548771 conn(0x7f27380f1c70 msgr2=0x7f27380f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:34:07.845 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:07.843+0000 7f27402d7700 1 -- 172.21.15.160:0/2400548771 shutdown_connections 2023-11-17T05:34:07.845 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:07.843+0000 7f27402d7700 1 -- 172.21.15.160:0/2400548771 wait complete. 2023-11-17T05:34:07.845 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:07.843+0000 7f27402d7700 1 Processor -- start 2023-11-17T05:34:07.845 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:07.843+0000 7f27402d7700 1 -- start start 2023-11-17T05:34:07.845 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:07.843+0000 7f27402d7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27380f6510 0x7f2738106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:34:07.845 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:07.843+0000 7f27402d7700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2738106c20 con 0x7f27380f6510 2023-11-17T05:34:07.846 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:07.844+0000 7f273e073700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27380f6510 0x7f2738106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:34:07.847 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:07.844+0000 7f273e073700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27380f6510 0x7f2738106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:36832/0 (socket says 172.21.15.160:36832) 2023-11-17T05:34:07.847 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:07.844+0000 7f273e073700 1 -- 172.21.15.160:0/3149630268 learned_addr learned my addr 172.21.15.160:0/3149630268 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:34:07.847 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:07.844+0000 7f273e073700 1 -- 172.21.15.160:0/3149630268 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2734005040 con 0x7f27380f6510 2023-11-17T05:34:07.847 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:07.844+0000 7f273e073700 1 --2- 172.21.15.160:0/3149630268 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27380f6510 0x7f2738106730 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f2734013510 tx=0x7f273400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:34:07.847 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:07.844+0000 7f272effd700 1 -- 172.21.15.160:0/3149630268 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2734009320 con 0x7f27380f6510 2023-11-17T05:34:07.847 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:07.844+0000 7f27402d7700 1 -- 172.21.15.160:0/3149630268 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2738106e20 con 0x7f27380f6510 2023-11-17T05:34:07.847 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:07.844+0000 7f272effd700 1 -- 172.21.15.160:0/3149630268 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2734013f00 con 0x7f27380f6510 2023-11-17T05:34:07.847 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:07.844+0000 7f272effd700 1 -- 172.21.15.160:0/3149630268 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2734014d90 con 0x7f27380f6510 2023-11-17T05:34:07.847 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:07.844+0000 7f27402d7700 1 -- 172.21.15.160:0/3149630268 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27381072b0 con 0x7f27380f6510 2023-11-17T05:34:07.847 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:07.845+0000 7f272effd700 1 -- 172.21.15.160:0/3149630268 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f27340095f0 con 0x7f27380f6510 2023-11-17T05:34:07.847 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:07.845+0000 7f27402d7700 1 -- 172.21.15.160:0/3149630268 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2738045ad0 con 0x7f27380f6510 2023-11-17T05:34:07.847 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:07.845+0000 7f272effd700 1 --2- 172.21.15.160:0/3149630268 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f272402ff60 0x7f2724032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:34:07.848 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:07.845+0000 7f272effd700 1 -- 172.21.15.160:0/3149630268 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f27340410c0 con 0x7f27380f6510 2023-11-17T05:34:07.848 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:07.846+0000 7f273d872700 1 --2- 172.21.15.160:0/3149630268 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f272402ff60 0x7f2724032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:34:07.848 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:07.846+0000 7f273d872700 1 --2- 172.21.15.160:0/3149630268 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f272402ff60 0x7f2724032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f27280096a0 tx=0x7f2728006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:34:07.851 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:07.850+0000 7f272effd700 1 -- 172.21.15.160:0/3149630268 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2734014420 con 0x7f27380f6510 2023-11-17T05:34:08.055 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:08.054+0000 7f27402d7700 1 -- 172.21.15.160:0/3149630268 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f27380fbf60 con 0x7f27380f6510 2023-11-17T05:34:08.055 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:08.055+0000 7f272effd700 1 -- 172.21.15.160:0/3149630268 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f2734010230 con 0x7f27380f6510 2023-11-17T05:34:08.056 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:34:08.056 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:34:08.057 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:08.056+0000 7f27402d7700 1 -- 172.21.15.160:0/3149630268 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f272402ff60 msgr2=0x7f2724032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:08.057 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:08.056+0000 7f27402d7700 1 --2- 172.21.15.160:0/3149630268 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f272402ff60 0x7f2724032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f27280096a0 tx=0x7f2728006b40).stop 2023-11-17T05:34:08.057 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:08.056+0000 7f27402d7700 1 -- 172.21.15.160:0/3149630268 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27380f6510 msgr2=0x7f2738106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:08.058 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:08.056+0000 7f27402d7700 1 --2- 172.21.15.160:0/3149630268 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27380f6510 0x7f2738106730 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f2734013510 tx=0x7f273400b040).stop 2023-11-17T05:34:08.058 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:08.057+0000 7f27402d7700 1 -- 172.21.15.160:0/3149630268 shutdown_connections 2023-11-17T05:34:08.058 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:08.057+0000 7f27402d7700 1 --2- 172.21.15.160:0/3149630268 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f272402ff60 0x7f2724032420 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:08.058 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:08.057+0000 7f27402d7700 1 --2- 172.21.15.160:0/3149630268 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f27380f6510 0x7f2738106730 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:08.058 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:08.057+0000 7f27402d7700 1 -- 172.21.15.160:0/3149630268 >> 172.21.15.160:0/3149630268 conn(0x7f27380f1c70 msgr2=0x7f27380f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:34:08.058 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:08.057+0000 7f27402d7700 1 -- 172.21.15.160:0/3149630268 shutdown_connections 2023-11-17T05:34:08.058 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:08.057+0000 7f27402d7700 1 -- 172.21.15.160:0/3149630268 wait complete. 2023-11-17T05:34:08.058 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:34:09.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:08 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:34:07.473416+0000 mgr.smithi003.eddcuc (mgr.14164) 268 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:16.170 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:08 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:34:08.056070+0000 mon.smithi003 (mon.0) 241 : audit [DBG] from='client.? 172.21.15.160:0/3149630268' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:34:16.170 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:34:16.171 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:34:16.172 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:09 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:34:09.473815+0000 mgr.smithi003.eddcuc (mgr.14164) 269 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:16.172 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:12 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:34:11.474139+0000 mgr.smithi003.eddcuc (mgr.14164) 270 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:16.172 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:14 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:34:13.474400+0000 mgr.smithi003.eddcuc (mgr.14164) 271 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:17.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:16 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:34:15.474740+0000 mgr.smithi003.eddcuc (mgr.14164) 272 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:17.393 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:17.391+0000 7f8b36806700 1 -- 172.21.15.160:0/247188801 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b300f5410 msgr2=0x7f8b300f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:17.393 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:17.391+0000 7f8b36806700 1 --2- 172.21.15.160:0/247188801 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b300f5410 0x7f8b300f5830 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f8b20009fb0 tx=0x7f8b20009b30).stop 2023-11-17T05:34:17.393 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:17.391+0000 7f8b36806700 1 -- 172.21.15.160:0/247188801 shutdown_connections 2023-11-17T05:34:17.393 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:17.391+0000 7f8b36806700 1 --2- 172.21.15.160:0/247188801 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b300f5410 0x7f8b300f5830 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:17.393 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:17.391+0000 7f8b36806700 1 -- 172.21.15.160:0/247188801 >> 172.21.15.160:0/247188801 conn(0x7f8b300f0b70 msgr2=0x7f8b300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:34:17.393 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:17.391+0000 7f8b36806700 1 -- 172.21.15.160:0/247188801 shutdown_connections 2023-11-17T05:34:17.394 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:17.391+0000 7f8b36806700 1 -- 172.21.15.160:0/247188801 wait complete. 2023-11-17T05:34:17.394 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:17.391+0000 7f8b36806700 1 Processor -- start 2023-11-17T05:34:17.394 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:17.392+0000 7f8b36806700 1 -- start start 2023-11-17T05:34:17.394 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:17.392+0000 7f8b36806700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b300f5410 0x7f8b30105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:34:17.394 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:17.392+0000 7f8b36806700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b30105af0 con 0x7f8b300f5410 2023-11-17T05:34:17.394 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:17.392+0000 7f8b2ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b300f5410 0x7f8b30105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:34:17.394 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:17.392+0000 7f8b2ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b300f5410 0x7f8b30105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:49694/0 (socket says 172.21.15.160:49694) 2023-11-17T05:34:17.394 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:17.392+0000 7f8b2ffff700 1 -- 172.21.15.160:0/191076701 learned_addr learned my addr 172.21.15.160:0/191076701 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:34:17.394 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:17.392+0000 7f8b2ffff700 1 -- 172.21.15.160:0/191076701 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b20005040 con 0x7f8b300f5410 2023-11-17T05:34:17.394 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:17.392+0000 7f8b2ffff700 1 --2- 172.21.15.160:0/191076701 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b300f5410 0x7f8b30105600 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f8b20014000 tx=0x7f8b2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:34:17.395 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:17.393+0000 7f8b2cff9700 1 -- 172.21.15.160:0/191076701 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b200095f0 con 0x7f8b300f5410 2023-11-17T05:34:17.396 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:17.393+0000 7f8b36806700 1 -- 172.21.15.160:0/191076701 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b30105cf0 con 0x7f8b300f5410 2023-11-17T05:34:17.396 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:17.393+0000 7f8b2cff9700 1 -- 172.21.15.160:0/191076701 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8b20007e70 con 0x7f8b300f5410 2023-11-17T05:34:17.396 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:17.393+0000 7f8b2cff9700 1 -- 172.21.15.160:0/191076701 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b20014ea0 con 0x7f8b300f5410 2023-11-17T05:34:17.396 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:17.393+0000 7f8b36806700 1 -- 172.21.15.160:0/191076701 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b30106180 con 0x7f8b300f5410 2023-11-17T05:34:17.396 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:17.393+0000 7f8b2cff9700 1 -- 172.21.15.160:0/191076701 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8b20025070 con 0x7f8b300f5410 2023-11-17T05:34:17.396 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:17.393+0000 7f8b36806700 1 -- 172.21.15.160:0/191076701 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b30045ad0 con 0x7f8b300f5410 2023-11-17T05:34:17.396 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:17.394+0000 7f8b2cff9700 1 --2- 172.21.15.160:0/191076701 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f8b1802ff60 0x7f8b18032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:34:17.396 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:17.394+0000 7f8b2cff9700 1 -- 172.21.15.160:0/191076701 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8b20040a30 con 0x7f8b300f5410 2023-11-17T05:34:17.396 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:17.394+0000 7f8b2f7fe700 1 --2- 172.21.15.160:0/191076701 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f8b1802ff60 0x7f8b18032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:34:17.396 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:17.394+0000 7f8b2f7fe700 1 --2- 172.21.15.160:0/191076701 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f8b1802ff60 0x7f8b18032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f8b240096a0 tx=0x7f8b24006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:34:17.399 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:17.399+0000 7f8b2cff9700 1 -- 172.21.15.160:0/191076701 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8b200079c0 con 0x7f8b300f5410 2023-11-17T05:34:17.604 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:17.604+0000 7f8b36806700 1 -- 172.21.15.160:0/191076701 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8b300fae60 con 0x7f8b300f5410 2023-11-17T05:34:17.605 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:17.604+0000 7f8b2cff9700 1 -- 172.21.15.160:0/191076701 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f8b2000f300 con 0x7f8b300f5410 2023-11-17T05:34:17.605 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:34:17.605 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:34:17.607 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:17.606+0000 7f8b36806700 1 -- 172.21.15.160:0/191076701 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f8b1802ff60 msgr2=0x7f8b18032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:17.607 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:17.606+0000 7f8b36806700 1 --2- 172.21.15.160:0/191076701 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f8b1802ff60 0x7f8b18032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f8b240096a0 tx=0x7f8b24006b40).stop 2023-11-17T05:34:17.607 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:17.606+0000 7f8b36806700 1 -- 172.21.15.160:0/191076701 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b300f5410 msgr2=0x7f8b30105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:17.607 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:17.606+0000 7f8b36806700 1 --2- 172.21.15.160:0/191076701 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b300f5410 0x7f8b30105600 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f8b20014000 tx=0x7f8b2000b040).stop 2023-11-17T05:34:17.607 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:17.606+0000 7f8b36806700 1 -- 172.21.15.160:0/191076701 shutdown_connections 2023-11-17T05:34:17.607 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:17.606+0000 7f8b36806700 1 --2- 172.21.15.160:0/191076701 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f8b1802ff60 0x7f8b18032420 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:17.607 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:17.606+0000 7f8b36806700 1 --2- 172.21.15.160:0/191076701 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b300f5410 0x7f8b30105600 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:17.607 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:17.606+0000 7f8b36806700 1 -- 172.21.15.160:0/191076701 >> 172.21.15.160:0/191076701 conn(0x7f8b300f0b70 msgr2=0x7f8b300f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:34:17.607 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:17.606+0000 7f8b36806700 1 -- 172.21.15.160:0/191076701 shutdown_connections 2023-11-17T05:34:17.607 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:17.606+0000 7f8b36806700 1 -- 172.21.15.160:0/191076701 wait complete. 2023-11-17T05:34:17.608 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:34:19.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:18 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:34:17.475052+0000 mgr.smithi003.eddcuc (mgr.14164) 273 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:19.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:18 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:34:17.605405+0000 mon.smithi003 (mon.0) 242 : audit [DBG] from='client.? 172.21.15.160:0/191076701' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:34:19.028 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:34:19.029 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:34:20.184 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:20.181+0000 7f885756a700 1 -- 172.21.15.160:0/2847166145 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88500601e0 msgr2=0x7f88500f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:20.184 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:20.181+0000 7f885756a700 1 --2- 172.21.15.160:0/2847166145 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88500601e0 0x7f88500f8f20 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f884c009fb0 tx=0x7f884c009b30).stop 2023-11-17T05:34:20.184 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:20.181+0000 7f885756a700 1 -- 172.21.15.160:0/2847166145 shutdown_connections 2023-11-17T05:34:20.184 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:20.181+0000 7f885756a700 1 --2- 172.21.15.160:0/2847166145 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88500601e0 0x7f88500f8f20 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:20.184 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:20.181+0000 7f885756a700 1 -- 172.21.15.160:0/2847166145 >> 172.21.15.160:0/2847166145 conn(0x7f88500f09a0 msgr2=0x7f88500f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:34:20.184 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:20.181+0000 7f885756a700 1 -- 172.21.15.160:0/2847166145 shutdown_connections 2023-11-17T05:34:20.185 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:20.181+0000 7f885756a700 1 -- 172.21.15.160:0/2847166145 wait complete. 2023-11-17T05:34:20.185 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:20.182+0000 7f885756a700 1 Processor -- start 2023-11-17T05:34:20.186 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:20.182+0000 7f885756a700 1 -- start start 2023-11-17T05:34:20.186 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:20.182+0000 7f885756a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88500601e0 0x7f88500f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:34:20.186 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:20.182+0000 7f885756a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88500f4e20 con 0x7f88500601e0 2023-11-17T05:34:20.186 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:20.182+0000 7f8856568700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88500601e0 0x7f88500f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:34:20.186 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:20.182+0000 7f8856568700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88500601e0 0x7f88500f69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:49714/0 (socket says 172.21.15.160:49714) 2023-11-17T05:34:20.186 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:20.182+0000 7f8856568700 1 -- 172.21.15.160:0/527917070 learned_addr learned my addr 172.21.15.160:0/527917070 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:34:20.186 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:20.183+0000 7f8856568700 1 -- 172.21.15.160:0/527917070 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f884c005040 con 0x7f88500601e0 2023-11-17T05:34:20.188 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:20.183+0000 7f8856568700 1 --2- 172.21.15.160:0/527917070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88500601e0 0x7f88500f69a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f884c014000 tx=0x7f884c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:34:20.188 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:20.183+0000 7f8846ffd700 1 -- 172.21.15.160:0/527917070 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f884c0095f0 con 0x7f88500601e0 2023-11-17T05:34:20.188 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:20.183+0000 7f885756a700 1 -- 172.21.15.160:0/527917070 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88500f4fc0 con 0x7f88500601e0 2023-11-17T05:34:20.189 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:20.183+0000 7f8846ffd700 1 -- 172.21.15.160:0/527917070 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f884c007e70 con 0x7f88500601e0 2023-11-17T05:34:20.190 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:20.183+0000 7f8846ffd700 1 -- 172.21.15.160:0/527917070 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f884c014ea0 con 0x7f88500601e0 2023-11-17T05:34:20.191 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:20.183+0000 7f885756a700 1 -- 172.21.15.160:0/527917070 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88500f5450 con 0x7f88500601e0 2023-11-17T05:34:20.194 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:20.184+0000 7f885756a700 1 -- 172.21.15.160:0/527917070 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8850059100 con 0x7f88500601e0 2023-11-17T05:34:20.195 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:20.184+0000 7f8846ffd700 1 -- 172.21.15.160:0/527917070 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f884c025070 con 0x7f88500601e0 2023-11-17T05:34:20.196 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:20.184+0000 7f8846ffd700 1 --2- 172.21.15.160:0/527917070 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f883c02ff60 0x7f883c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:34:20.196 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:20.184+0000 7f8846ffd700 1 -- 172.21.15.160:0/527917070 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f884c040a30 con 0x7f88500601e0 2023-11-17T05:34:20.197 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:20.184+0000 7f8855d67700 1 --2- 172.21.15.160:0/527917070 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f883c02ff60 0x7f883c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:34:20.198 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:20.185+0000 7f8855d67700 1 --2- 172.21.15.160:0/527917070 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f883c02ff60 0x7f883c032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f88400096a0 tx=0x7f8840006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:34:20.198 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:20.189+0000 7f8846ffd700 1 -- 172.21.15.160:0/527917070 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f884c0079c0 con 0x7f88500601e0 2023-11-17T05:34:20.271 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:19 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:34:19.475400+0000 mgr.smithi003.eddcuc (mgr.14164) 274 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:20.398 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:20.397+0000 7f885756a700 1 -- 172.21.15.160:0/527917070 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f88500fbcd0 con 0x7f88500601e0 2023-11-17T05:34:20.398 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:20.398+0000 7f8846ffd700 1 -- 172.21.15.160:0/527917070 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f884c00f300 con 0x7f88500601e0 2023-11-17T05:34:20.399 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:34:20.399 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:34:20.400 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:20.399+0000 7f885756a700 1 -- 172.21.15.160:0/527917070 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f883c02ff60 msgr2=0x7f883c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:20.400 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:20.399+0000 7f885756a700 1 --2- 172.21.15.160:0/527917070 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f883c02ff60 0x7f883c032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f88400096a0 tx=0x7f8840006b40).stop 2023-11-17T05:34:20.400 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:20.399+0000 7f885756a700 1 -- 172.21.15.160:0/527917070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88500601e0 msgr2=0x7f88500f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:20.401 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:20.399+0000 7f885756a700 1 --2- 172.21.15.160:0/527917070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88500601e0 0x7f88500f69a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f884c014000 tx=0x7f884c00b040).stop 2023-11-17T05:34:20.401 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:20.399+0000 7f885756a700 1 -- 172.21.15.160:0/527917070 shutdown_connections 2023-11-17T05:34:20.401 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:20.399+0000 7f885756a700 1 --2- 172.21.15.160:0/527917070 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f883c02ff60 0x7f883c032420 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:20.401 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:20.399+0000 7f885756a700 1 --2- 172.21.15.160:0/527917070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f88500601e0 0x7f88500f69a0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:20.401 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:20.399+0000 7f885756a700 1 -- 172.21.15.160:0/527917070 >> 172.21.15.160:0/527917070 conn(0x7f88500f09a0 msgr2=0x7f88500f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:34:20.401 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:20.399+0000 7f885756a700 1 -- 172.21.15.160:0/527917070 shutdown_connections 2023-11-17T05:34:20.401 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:20.399+0000 7f885756a700 1 -- 172.21.15.160:0/527917070 wait complete. 2023-11-17T05:34:20.401 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:34:21.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:20 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:34:20.398999+0000 mon.smithi003 (mon.0) 243 : audit [DBG] from='client.? 172.21.15.160:0/527917070' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:34:21.784 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:34:21.784 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:34:22.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:21 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:34:21.475762+0000 mgr.smithi003.eddcuc (mgr.14164) 275 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:22.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:22.997+0000 7f03f359e700 1 -- 172.21.15.160:0/1888493943 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03f40f51a0 msgr2=0x7f03f40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:22.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:22.997+0000 7f03f359e700 1 --2- 172.21.15.160:0/1888493943 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03f40f51a0 0x7f03f40f55c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f03e4009fb0 tx=0x7f03e4009b30).stop 2023-11-17T05:34:22.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:22.997+0000 7f03f359e700 1 -- 172.21.15.160:0/1888493943 shutdown_connections 2023-11-17T05:34:22.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:22.997+0000 7f03f359e700 1 --2- 172.21.15.160:0/1888493943 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03f40f51a0 0x7f03f40f55c0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:23.000 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:22.997+0000 7f03f359e700 1 -- 172.21.15.160:0/1888493943 >> 172.21.15.160:0/1888493943 conn(0x7f03f40f09a0 msgr2=0x7f03f40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:34:23.000 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:22.997+0000 7f03f359e700 1 -- 172.21.15.160:0/1888493943 shutdown_connections 2023-11-17T05:34:23.000 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:22.997+0000 7f03f359e700 1 -- 172.21.15.160:0/1888493943 wait complete. 2023-11-17T05:34:23.000 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:22.998+0000 7f03f359e700 1 Processor -- start 2023-11-17T05:34:23.000 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:22.998+0000 7f03f359e700 1 -- start start 2023-11-17T05:34:23.000 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:22.998+0000 7f03f359e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03f40f51a0 0x7f03f4109170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:34:23.000 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:22.998+0000 7f03f359e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f4060280 con 0x7f03f40f51a0 2023-11-17T05:34:23.000 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:22.998+0000 7f03f259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03f40f51a0 0x7f03f4109170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:34:23.000 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:22.998+0000 7f03f259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03f40f51a0 0x7f03f4109170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:54330/0 (socket says 172.21.15.160:54330) 2023-11-17T05:34:23.000 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:22.998+0000 7f03f259c700 1 -- 172.21.15.160:0/1520054413 learned_addr learned my addr 172.21.15.160:0/1520054413 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:34:23.000 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:22.999+0000 7f03f259c700 1 -- 172.21.15.160:0/1520054413 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03e4005040 con 0x7f03f40f51a0 2023-11-17T05:34:23.000 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:22.999+0000 7f03f259c700 1 --2- 172.21.15.160:0/1520054413 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03f40f51a0 0x7f03f4109170 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f03e4013510 tx=0x7f03e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:34:23.002 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:22.999+0000 7f03e2ffd700 1 -- 172.21.15.160:0/1520054413 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03e4009320 con 0x7f03f40f51a0 2023-11-17T05:34:23.002 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:22.999+0000 7f03f359e700 1 -- 172.21.15.160:0/1520054413 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03f4060480 con 0x7f03f40f51a0 2023-11-17T05:34:23.002 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:22.999+0000 7f03e2ffd700 1 -- 172.21.15.160:0/1520054413 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f03e4013f00 con 0x7f03f40f51a0 2023-11-17T05:34:23.002 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:22.999+0000 7f03e2ffd700 1 -- 172.21.15.160:0/1520054413 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03e4014dc0 con 0x7f03f40f51a0 2023-11-17T05:34:23.002 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:22.999+0000 7f03f359e700 1 -- 172.21.15.160:0/1520054413 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03f41098a0 con 0x7f03f40f51a0 2023-11-17T05:34:23.002 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:23.000+0000 7f03f359e700 1 -- 172.21.15.160:0/1520054413 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03f4045ad0 con 0x7f03f40f51a0 2023-11-17T05:34:23.002 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:23.000+0000 7f03e2ffd700 1 -- 172.21.15.160:0/1520054413 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f03e40095f0 con 0x7f03f40f51a0 2023-11-17T05:34:23.002 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:23.000+0000 7f03e2ffd700 1 --2- 172.21.15.160:0/1520054413 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f03dc02ff60 0x7f03dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:34:23.002 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:23.000+0000 7f03e2ffd700 1 -- 172.21.15.160:0/1520054413 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f03e40410c0 con 0x7f03f40f51a0 2023-11-17T05:34:23.002 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:23.000+0000 7f03f1d9b700 1 --2- 172.21.15.160:0/1520054413 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f03dc02ff60 0x7f03dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:34:23.002 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:23.001+0000 7f03f1d9b700 1 --2- 172.21.15.160:0/1520054413 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f03dc02ff60 0x7f03dc032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f03e80096a0 tx=0x7f03e8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:34:23.005 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:23.005+0000 7f03e2ffd700 1 -- 172.21.15.160:0/1520054413 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f03e4014420 con 0x7f03f40f51a0 2023-11-17T05:34:23.210 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:23.209+0000 7f03f359e700 1 -- 172.21.15.160:0/1520054413 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f03f40fabe0 con 0x7f03f40f51a0 2023-11-17T05:34:23.210 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:23.210+0000 7f03e2ffd700 1 -- 172.21.15.160:0/1520054413 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f03e4010230 con 0x7f03f40f51a0 2023-11-17T05:34:23.211 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:34:23.211 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:34:23.212 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:23.211+0000 7f03f359e700 1 -- 172.21.15.160:0/1520054413 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f03dc02ff60 msgr2=0x7f03dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:23.212 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:23.211+0000 7f03f359e700 1 --2- 172.21.15.160:0/1520054413 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f03dc02ff60 0x7f03dc032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f03e80096a0 tx=0x7f03e8006b40).stop 2023-11-17T05:34:23.213 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:23.211+0000 7f03f359e700 1 -- 172.21.15.160:0/1520054413 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03f40f51a0 msgr2=0x7f03f4109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:23.213 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:23.211+0000 7f03f359e700 1 --2- 172.21.15.160:0/1520054413 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03f40f51a0 0x7f03f4109170 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f03e4013510 tx=0x7f03e400b040).stop 2023-11-17T05:34:23.213 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:23.211+0000 7f03f359e700 1 -- 172.21.15.160:0/1520054413 shutdown_connections 2023-11-17T05:34:23.213 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:23.211+0000 7f03f359e700 1 --2- 172.21.15.160:0/1520054413 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f03dc02ff60 0x7f03dc032420 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:23.213 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:23.211+0000 7f03f359e700 1 --2- 172.21.15.160:0/1520054413 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03f40f51a0 0x7f03f4109170 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:23.213 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:23.211+0000 7f03f359e700 1 -- 172.21.15.160:0/1520054413 >> 172.21.15.160:0/1520054413 conn(0x7f03f40f09a0 msgr2=0x7f03f40f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:34:23.213 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:23.212+0000 7f03f359e700 1 -- 172.21.15.160:0/1520054413 shutdown_connections 2023-11-17T05:34:23.213 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:23.212+0000 7f03f359e700 1 -- 172.21.15.160:0/1520054413 wait complete. 2023-11-17T05:34:23.213 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:34:23.520 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:23 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:34:23.211074+0000 mon.smithi003 (mon.0) 244 : audit [DBG] from='client.? 172.21.15.160:0/1520054413' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:34:24.527 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:34:23.476029+0000 mgr.smithi003.eddcuc (mgr.14164) 276 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:24.689 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:34:24.689 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:34:25.917 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:25.914+0000 7f429ccde700 1 -- 172.21.15.160:0/833859298 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42980f5410 msgr2=0x7f42980f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:25.917 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:25.914+0000 7f429ccde700 1 --2- 172.21.15.160:0/833859298 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42980f5410 0x7f42980f5830 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f4280009fb0 tx=0x7f4280009b30).stop 2023-11-17T05:34:25.917 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:25.914+0000 7f429ccde700 1 -- 172.21.15.160:0/833859298 shutdown_connections 2023-11-17T05:34:25.918 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:25.914+0000 7f429ccde700 1 --2- 172.21.15.160:0/833859298 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42980f5410 0x7f42980f5830 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:25.918 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:25.914+0000 7f429ccde700 1 -- 172.21.15.160:0/833859298 >> 172.21.15.160:0/833859298 conn(0x7f42980f0b70 msgr2=0x7f42980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:34:25.918 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:25.914+0000 7f429ccde700 1 -- 172.21.15.160:0/833859298 shutdown_connections 2023-11-17T05:34:25.918 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:25.915+0000 7f429ccde700 1 -- 172.21.15.160:0/833859298 wait complete. 2023-11-17T05:34:25.918 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:25.915+0000 7f429ccde700 1 Processor -- start 2023-11-17T05:34:25.918 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:25.915+0000 7f429ccde700 1 -- start start 2023-11-17T05:34:25.918 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:25.915+0000 7f429ccde700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42980f5410 0x7f42980fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:34:25.918 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:25.915+0000 7f429ccde700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42980fb820 con 0x7f42980f5410 2023-11-17T05:34:25.918 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:25.915+0000 7f429659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42980f5410 0x7f42980fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:34:25.918 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:25.915+0000 7f429659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42980f5410 0x7f42980fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:54352/0 (socket says 172.21.15.160:54352) 2023-11-17T05:34:25.918 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:25.915+0000 7f429659c700 1 -- 172.21.15.160:0/422054400 learned_addr learned my addr 172.21.15.160:0/422054400 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:34:25.919 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:25.916+0000 7f429659c700 1 -- 172.21.15.160:0/422054400 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4280005040 con 0x7f42980f5410 2023-11-17T05:34:25.919 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:25.916+0000 7f429659c700 1 --2- 172.21.15.160:0/422054400 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42980f5410 0x7f42980fd3a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f4280013510 tx=0x7f428000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:34:25.919 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:25.916+0000 7f428effd700 1 -- 172.21.15.160:0/422054400 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4280009320 con 0x7f42980f5410 2023-11-17T05:34:25.919 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:25.916+0000 7f428effd700 1 -- 172.21.15.160:0/422054400 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4280013f00 con 0x7f42980f5410 2023-11-17T05:34:25.921 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:25.916+0000 7f429ccde700 1 -- 172.21.15.160:0/422054400 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42980fb9c0 con 0x7f42980f5410 2023-11-17T05:34:25.921 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:25.916+0000 7f428effd700 1 -- 172.21.15.160:0/422054400 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4280014d90 con 0x7f42980f5410 2023-11-17T05:34:25.921 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:25.916+0000 7f429ccde700 1 -- 172.21.15.160:0/422054400 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42980fbe50 con 0x7f42980f5410 2023-11-17T05:34:25.921 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:25.917+0000 7f428effd700 1 -- 172.21.15.160:0/422054400 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f42800095f0 con 0x7f42980f5410 2023-11-17T05:34:25.921 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:25.917+0000 7f429ccde700 1 -- 172.21.15.160:0/422054400 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4298059100 con 0x7f42980f5410 2023-11-17T05:34:25.921 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:25.917+0000 7f428effd700 1 --2- 172.21.15.160:0/422054400 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f428402ff60 0x7f4284032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:34:25.921 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:25.917+0000 7f428effd700 1 -- 172.21.15.160:0/422054400 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4280041000 con 0x7f42980f5410 2023-11-17T05:34:25.921 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:25.917+0000 7f4295d9b700 1 --2- 172.21.15.160:0/422054400 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f428402ff60 0x7f4284032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:34:25.921 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:25.918+0000 7f4295d9b700 1 --2- 172.21.15.160:0/422054400 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f428402ff60 0x7f4284032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f42880096a0 tx=0x7f4288006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:34:25.924 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:25.922+0000 7f428effd700 1 -- 172.21.15.160:0/422054400 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4280014420 con 0x7f42980f5410 2023-11-17T05:34:26.128 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:26.127+0000 7f429ccde700 1 -- 172.21.15.160:0/422054400 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f42980fae60 con 0x7f42980f5410 2023-11-17T05:34:26.128 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:26.128+0000 7f428effd700 1 -- 172.21.15.160:0/422054400 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f4280010230 con 0x7f42980f5410 2023-11-17T05:34:26.128 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:34:26.128 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:34:26.130 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:26.129+0000 7f429ccde700 1 -- 172.21.15.160:0/422054400 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f428402ff60 msgr2=0x7f4284032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:26.130 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:26.129+0000 7f429ccde700 1 --2- 172.21.15.160:0/422054400 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f428402ff60 0x7f4284032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f42880096a0 tx=0x7f4288006b40).stop 2023-11-17T05:34:26.130 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:26.129+0000 7f429ccde700 1 -- 172.21.15.160:0/422054400 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42980f5410 msgr2=0x7f42980fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:26.130 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:26.129+0000 7f429ccde700 1 --2- 172.21.15.160:0/422054400 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42980f5410 0x7f42980fd3a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f4280013510 tx=0x7f428000b040).stop 2023-11-17T05:34:26.130 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:26.129+0000 7f429ccde700 1 -- 172.21.15.160:0/422054400 shutdown_connections 2023-11-17T05:34:26.130 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:26.129+0000 7f429ccde700 1 --2- 172.21.15.160:0/422054400 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f428402ff60 0x7f4284032420 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:26.130 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:26.129+0000 7f429ccde700 1 --2- 172.21.15.160:0/422054400 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f42980f5410 0x7f42980fd3a0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:26.131 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:26.129+0000 7f429ccde700 1 -- 172.21.15.160:0/422054400 >> 172.21.15.160:0/422054400 conn(0x7f42980f0b70 msgr2=0x7f429805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:34:26.131 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:26.129+0000 7f429ccde700 1 -- 172.21.15.160:0/422054400 shutdown_connections 2023-11-17T05:34:26.131 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:26.129+0000 7f429ccde700 1 -- 172.21.15.160:0/422054400 wait complete. 2023-11-17T05:34:26.131 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:34:27.021 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:26 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:34:25.476361+0000 mgr.smithi003.eddcuc (mgr.14164) 277 : cluster [DBG] pgmap v263: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:27.021 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:26 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:34:26.128726+0000 mon.smithi003 (mon.0) 245 : audit [DBG] from='client.? 172.21.15.160:0/422054400' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:34:27.557 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:34:27.558 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:34:28.799 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:28.798+0000 7fa0488f2700 1 -- 172.21.15.160:0/138857414 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0440f51a0 msgr2=0x7fa0440f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:28.799 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:28.798+0000 7fa0488f2700 1 --2- 172.21.15.160:0/138857414 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0440f51a0 0x7fa0440f55c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7fa02c009fb0 tx=0x7fa02c009b30).stop 2023-11-17T05:34:28.799 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:28.798+0000 7fa0488f2700 1 -- 172.21.15.160:0/138857414 shutdown_connections 2023-11-17T05:34:28.799 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:28.798+0000 7fa0488f2700 1 --2- 172.21.15.160:0/138857414 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0440f51a0 0x7fa0440f55c0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:28.799 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:28.798+0000 7fa0488f2700 1 -- 172.21.15.160:0/138857414 >> 172.21.15.160:0/138857414 conn(0x7fa0440f09a0 msgr2=0x7fa0440f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:34:28.800 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:28.798+0000 7fa0488f2700 1 -- 172.21.15.160:0/138857414 shutdown_connections 2023-11-17T05:34:28.800 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:28.798+0000 7fa0488f2700 1 -- 172.21.15.160:0/138857414 wait complete. 2023-11-17T05:34:28.801 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:28.799+0000 7fa0488f2700 1 Processor -- start 2023-11-17T05:34:28.801 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:28.799+0000 7fa0488f2700 1 -- start start 2023-11-17T05:34:28.801 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:28.799+0000 7fa0488f2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0440f51a0 0x7fa0441053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:34:28.801 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:28.799+0000 7fa0488f2700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0441058a0 con 0x7fa0440f51a0 2023-11-17T05:34:28.801 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:28.799+0000 7fa0437fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0440f51a0 0x7fa0441053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:34:28.801 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:28.799+0000 7fa0437fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0440f51a0 0x7fa0441053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:54376/0 (socket says 172.21.15.160:54376) 2023-11-17T05:34:28.801 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:28.799+0000 7fa0437fe700 1 -- 172.21.15.160:0/2671880304 learned_addr learned my addr 172.21.15.160:0/2671880304 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:34:28.802 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:28.800+0000 7fa0437fe700 1 -- 172.21.15.160:0/2671880304 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa02c005040 con 0x7fa0440f51a0 2023-11-17T05:34:28.802 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:28.800+0000 7fa0437fe700 1 --2- 172.21.15.160:0/2671880304 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0440f51a0 0x7fa0441053b0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7fa02c009830 tx=0x7fa02c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:34:28.802 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:28.800+0000 7fa03bfff700 1 -- 172.21.15.160:0/2671880304 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa02c009320 con 0x7fa0440f51a0 2023-11-17T05:34:28.802 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:28.800+0000 7fa0488f2700 1 -- 172.21.15.160:0/2671880304 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa044105aa0 con 0x7fa0440f51a0 2023-11-17T05:34:28.802 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:28.800+0000 7fa03bfff700 1 -- 172.21.15.160:0/2671880304 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa02c0136c0 con 0x7fa0440f51a0 2023-11-17T05:34:28.802 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:28.800+0000 7fa03bfff700 1 -- 172.21.15.160:0/2671880304 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa02c01ee90 con 0x7fa0440f51a0 2023-11-17T05:34:28.802 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:28.800+0000 7fa0488f2700 1 -- 172.21.15.160:0/2671880304 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa044105f30 con 0x7fa0440f51a0 2023-11-17T05:34:28.803 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:28.801+0000 7fa03bfff700 1 -- 172.21.15.160:0/2671880304 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa02c01e420 con 0x7fa0440f51a0 2023-11-17T05:34:28.803 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:28.801+0000 7fa0488f2700 1 -- 172.21.15.160:0/2671880304 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa044045ad0 con 0x7fa0440f51a0 2023-11-17T05:34:28.803 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:28.801+0000 7fa03bfff700 1 --2- 172.21.15.160:0/2671880304 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa030032120 0x7fa0300345e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:34:28.803 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:28.801+0000 7fa03bfff700 1 -- 172.21.15.160:0/2671880304 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa02c041010 con 0x7fa0440f51a0 2023-11-17T05:34:28.804 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:28.801+0000 7fa042ffd700 1 --2- 172.21.15.160:0/2671880304 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa030032120 0x7fa0300345e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:34:28.804 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:28.802+0000 7fa042ffd700 1 --2- 172.21.15.160:0/2671880304 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa030032120 0x7fa0300345e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fa0340096a0 tx=0x7fa034006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:34:28.806 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:28.806+0000 7fa03bfff700 1 -- 172.21.15.160:0/2671880304 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa02c0095f0 con 0x7fa0440f51a0 2023-11-17T05:34:29.017 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:29.014+0000 7fa0488f2700 1 -- 172.21.15.160:0/2671880304 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa044106de0 con 0x7fa0440f51a0 2023-11-17T05:34:29.017 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:29.015+0000 7fa03bfff700 1 -- 172.21.15.160:0/2671880304 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fa02c01cb10 con 0x7fa0440f51a0 2023-11-17T05:34:29.019 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:34:29.020 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:34:29.022 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:29.016+0000 7fa0488f2700 1 -- 172.21.15.160:0/2671880304 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa030032120 msgr2=0x7fa0300345e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:29.022 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:29.016+0000 7fa0488f2700 1 --2- 172.21.15.160:0/2671880304 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa030032120 0x7fa0300345e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fa0340096a0 tx=0x7fa034006b40).stop 2023-11-17T05:34:29.022 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:29.016+0000 7fa0488f2700 1 -- 172.21.15.160:0/2671880304 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0440f51a0 msgr2=0x7fa0441053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:29.023 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:29.016+0000 7fa0488f2700 1 --2- 172.21.15.160:0/2671880304 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0440f51a0 0x7fa0441053b0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7fa02c009830 tx=0x7fa02c00b040).stop 2023-11-17T05:34:29.023 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:29.016+0000 7fa0488f2700 1 -- 172.21.15.160:0/2671880304 shutdown_connections 2023-11-17T05:34:29.023 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:29.016+0000 7fa0488f2700 1 --2- 172.21.15.160:0/2671880304 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa030032120 0x7fa0300345e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:29.023 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:29.016+0000 7fa0488f2700 1 --2- 172.21.15.160:0/2671880304 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0440f51a0 0x7fa0441053b0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:29.023 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:29.016+0000 7fa0488f2700 1 -- 172.21.15.160:0/2671880304 >> 172.21.15.160:0/2671880304 conn(0x7fa0440f09a0 msgr2=0x7fa0440f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:34:29.023 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:29.016+0000 7fa0488f2700 1 -- 172.21.15.160:0/2671880304 shutdown_connections 2023-11-17T05:34:29.023 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:29.016+0000 7fa0488f2700 1 -- 172.21.15.160:0/2671880304 wait complete. 2023-11-17T05:34:29.023 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:34:29.023 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:34:27.476647+0000 mgr.smithi003.eddcuc (mgr.14164) 278 : cluster [DBG] pgmap v264: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:29.024 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:34:27.555209+0000 mon.smithi003 (mon.0) 246 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.eddcuc/mirror_snapshot_schedule"}]: dispatch 2023-11-17T05:34:29.024 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:34:27.558907+0000 mon.smithi003 (mon.0) 247 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.eddcuc/trash_purge_schedule"}]: dispatch 2023-11-17T05:34:30.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:29 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:34:29.015910+0000 mon.smithi003 (mon.0) 248 : audit [DBG] from='client.? 172.21.15.160:0/2671880304' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:34:30.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:29 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:34:29.476937+0000 mgr.smithi003.eddcuc (mgr.14164) 279 : cluster [DBG] pgmap v265: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:30.365 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:34:30.365 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:34:31.575 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:31.574+0000 7f16824ed700 1 -- 172.21.15.160:0/4009146740 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f167c0f6510 msgr2=0x7f167c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:31.575 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:31.574+0000 7f16824ed700 1 --2- 172.21.15.160:0/4009146740 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f167c0f6510 0x7f167c0f6930 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f166c009fb0 tx=0x7f166c009b30).stop 2023-11-17T05:34:31.575 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:31.574+0000 7f16824ed700 1 -- 172.21.15.160:0/4009146740 shutdown_connections 2023-11-17T05:34:31.575 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:31.574+0000 7f16824ed700 1 --2- 172.21.15.160:0/4009146740 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f167c0f6510 0x7f167c0f6930 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:31.575 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:31.574+0000 7f16824ed700 1 -- 172.21.15.160:0/4009146740 >> 172.21.15.160:0/4009146740 conn(0x7f167c0f1c70 msgr2=0x7f167c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:34:31.575 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:31.574+0000 7f16824ed700 1 -- 172.21.15.160:0/4009146740 shutdown_connections 2023-11-17T05:34:31.575 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:31.574+0000 7f16824ed700 1 -- 172.21.15.160:0/4009146740 wait complete. 2023-11-17T05:34:31.577 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:31.574+0000 7f16824ed700 1 Processor -- start 2023-11-17T05:34:31.577 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:31.575+0000 7f16824ed700 1 -- start start 2023-11-17T05:34:31.577 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:31.575+0000 7f16824ed700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f167c0f6510 0x7f167c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:34:31.577 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:31.575+0000 7f16824ed700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f167c106c20 con 0x7f167c0f6510 2023-11-17T05:34:31.577 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:31.575+0000 7f167bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f167c0f6510 0x7f167c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:34:31.577 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:31.575+0000 7f167bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f167c0f6510 0x7f167c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:36944/0 (socket says 172.21.15.160:36944) 2023-11-17T05:34:31.577 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:31.575+0000 7f167bfff700 1 -- 172.21.15.160:0/2491133710 learned_addr learned my addr 172.21.15.160:0/2491133710 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:34:31.577 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:31.575+0000 7f167bfff700 1 -- 172.21.15.160:0/2491133710 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f166c005040 con 0x7f167c0f6510 2023-11-17T05:34:31.577 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:31.576+0000 7f167bfff700 1 --2- 172.21.15.160:0/2491133710 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f167c0f6510 0x7f167c106730 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f166c013510 tx=0x7f166c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:34:31.577 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:31.576+0000 7f1678ff9700 1 -- 172.21.15.160:0/2491133710 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f166c009320 con 0x7f167c0f6510 2023-11-17T05:34:31.577 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:31.576+0000 7f16824ed700 1 -- 172.21.15.160:0/2491133710 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f167c106e20 con 0x7f167c0f6510 2023-11-17T05:34:31.578 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:31.576+0000 7f1678ff9700 1 -- 172.21.15.160:0/2491133710 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f166c013f00 con 0x7f167c0f6510 2023-11-17T05:34:31.578 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:31.576+0000 7f1678ff9700 1 -- 172.21.15.160:0/2491133710 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f166c014d90 con 0x7f167c0f6510 2023-11-17T05:34:31.578 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:31.576+0000 7f16824ed700 1 -- 172.21.15.160:0/2491133710 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f167c1072b0 con 0x7f167c0f6510 2023-11-17T05:34:31.579 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:31.577+0000 7f16824ed700 1 -- 172.21.15.160:0/2491133710 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f167c045ad0 con 0x7f167c0f6510 2023-11-17T05:34:31.579 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:31.577+0000 7f1678ff9700 1 -- 172.21.15.160:0/2491133710 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f166c0095f0 con 0x7f167c0f6510 2023-11-17T05:34:31.579 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:31.577+0000 7f1678ff9700 1 --2- 172.21.15.160:0/2491133710 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f166402ff60 0x7f1664032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:34:31.579 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:31.577+0000 7f1678ff9700 1 -- 172.21.15.160:0/2491133710 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f166c040fc0 con 0x7f167c0f6510 2023-11-17T05:34:31.579 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:31.577+0000 7f167b7fe700 1 --2- 172.21.15.160:0/2491133710 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f166402ff60 0x7f1664032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:34:31.579 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:31.578+0000 7f167b7fe700 1 --2- 172.21.15.160:0/2491133710 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f166402ff60 0x7f1664032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f16700096a0 tx=0x7f1670006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:34:31.584 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:31.582+0000 7f1678ff9700 1 -- 172.21.15.160:0/2491133710 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f166c014420 con 0x7f167c0f6510 2023-11-17T05:34:31.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:31.783+0000 7f16824ed700 1 -- 172.21.15.160:0/2491133710 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f167c0fbf60 con 0x7f167c0f6510 2023-11-17T05:34:31.784 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:31.784+0000 7f1678ff9700 1 -- 172.21.15.160:0/2491133710 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f166c010230 con 0x7f167c0f6510 2023-11-17T05:34:31.784 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:34:31.785 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:34:31.786 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:31.785+0000 7f16824ed700 1 -- 172.21.15.160:0/2491133710 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f166402ff60 msgr2=0x7f1664032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:31.786 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:31.785+0000 7f16824ed700 1 --2- 172.21.15.160:0/2491133710 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f166402ff60 0x7f1664032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f16700096a0 tx=0x7f1670006b40).stop 2023-11-17T05:34:31.787 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:31.785+0000 7f16824ed700 1 -- 172.21.15.160:0/2491133710 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f167c0f6510 msgr2=0x7f167c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:31.787 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:31.785+0000 7f16824ed700 1 --2- 172.21.15.160:0/2491133710 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f167c0f6510 0x7f167c106730 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f166c013510 tx=0x7f166c00b040).stop 2023-11-17T05:34:31.787 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:31.785+0000 7f16824ed700 1 -- 172.21.15.160:0/2491133710 shutdown_connections 2023-11-17T05:34:31.787 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:31.785+0000 7f16824ed700 1 --2- 172.21.15.160:0/2491133710 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f166402ff60 0x7f1664032420 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:31.787 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:31.785+0000 7f16824ed700 1 --2- 172.21.15.160:0/2491133710 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f167c0f6510 0x7f167c106730 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:31.787 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:31.785+0000 7f16824ed700 1 -- 172.21.15.160:0/2491133710 >> 172.21.15.160:0/2491133710 conn(0x7f167c0f1c70 msgr2=0x7f167c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:34:31.787 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:31.785+0000 7f16824ed700 1 -- 172.21.15.160:0/2491133710 shutdown_connections 2023-11-17T05:34:31.787 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:31.785+0000 7f16824ed700 1 -- 172.21.15.160:0/2491133710 wait complete. 2023-11-17T05:34:31.787 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:34:33.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:32 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:34:31.477178+0000 mgr.smithi003.eddcuc (mgr.14164) 280 : cluster [DBG] pgmap v266: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:33.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:32 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:34:31.784802+0000 mon.smithi003 (mon.0) 249 : audit [DBG] from='client.? 172.21.15.160:0/2491133710' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:34:33.220 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:34:33.221 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:34:34.427 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:34.424+0000 7fdc50827700 1 -- 172.21.15.160:0/639241279 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc4c0f78e0 msgr2=0x7fdc4c0f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:34.427 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:34.424+0000 7fdc50827700 1 --2- 172.21.15.160:0/639241279 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc4c0f78e0 0x7fdc4c0f7d00 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7fdc3c009fb0 tx=0x7fdc3c009b30).stop 2023-11-17T05:34:34.427 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:34.424+0000 7fdc50827700 1 -- 172.21.15.160:0/639241279 shutdown_connections 2023-11-17T05:34:34.427 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:34.424+0000 7fdc50827700 1 --2- 172.21.15.160:0/639241279 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc4c0f78e0 0x7fdc4c0f7d00 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:34.427 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:34.424+0000 7fdc50827700 1 -- 172.21.15.160:0/639241279 >> 172.21.15.160:0/639241279 conn(0x7fdc4c0f0120 msgr2=0x7fdc4c0f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:34:34.427 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:34.424+0000 7fdc50827700 1 -- 172.21.15.160:0/639241279 shutdown_connections 2023-11-17T05:34:34.427 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:34.424+0000 7fdc50827700 1 -- 172.21.15.160:0/639241279 wait complete. 2023-11-17T05:34:34.428 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:34.425+0000 7fdc50827700 1 Processor -- start 2023-11-17T05:34:34.428 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:34.425+0000 7fdc50827700 1 -- start start 2023-11-17T05:34:34.428 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:34.425+0000 7fdc50827700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc4c0f78e0 0x7fdc4c1064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:34:34.428 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:34.425+0000 7fdc50827700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4c1069d0 con 0x7fdc4c0f78e0 2023-11-17T05:34:34.428 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:34.426+0000 7fdc4ad9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc4c0f78e0 0x7fdc4c1064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:34:34.428 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:34.426+0000 7fdc4ad9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc4c0f78e0 0x7fdc4c1064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:36968/0 (socket says 172.21.15.160:36968) 2023-11-17T05:34:34.428 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:34.426+0000 7fdc4ad9d700 1 -- 172.21.15.160:0/1720421140 learned_addr learned my addr 172.21.15.160:0/1720421140 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:34:34.428 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:34.426+0000 7fdc4ad9d700 1 -- 172.21.15.160:0/1720421140 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc3c005040 con 0x7fdc4c0f78e0 2023-11-17T05:34:34.428 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:34.426+0000 7fdc4ad9d700 1 --2- 172.21.15.160:0/1720421140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc4c0f78e0 0x7fdc4c1064e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7fdc3c009830 tx=0x7fdc3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:34:34.428 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:34.426+0000 7fdc337fe700 1 -- 172.21.15.160:0/1720421140 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc3c009320 con 0x7fdc4c0f78e0 2023-11-17T05:34:34.428 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:34.426+0000 7fdc337fe700 1 -- 172.21.15.160:0/1720421140 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdc3c0136c0 con 0x7fdc4c0f78e0 2023-11-17T05:34:34.428 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:34.426+0000 7fdc50827700 1 -- 172.21.15.160:0/1720421140 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc4c106bd0 con 0x7fdc4c0f78e0 2023-11-17T05:34:34.428 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:34.426+0000 7fdc337fe700 1 -- 172.21.15.160:0/1720421140 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc3c01ee90 con 0x7fdc4c0f78e0 2023-11-17T05:34:34.429 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:34.426+0000 7fdc50827700 1 -- 172.21.15.160:0/1720421140 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc4c107060 con 0x7fdc4c0f78e0 2023-11-17T05:34:34.430 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:34.427+0000 7fdc337fe700 1 -- 172.21.15.160:0/1720421140 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdc3c01e420 con 0x7fdc4c0f78e0 2023-11-17T05:34:34.430 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:34.427+0000 7fdc50827700 1 -- 172.21.15.160:0/1720421140 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc4c045ad0 con 0x7fdc4c0f78e0 2023-11-17T05:34:34.430 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:34.427+0000 7fdc337fe700 1 --2- 172.21.15.160:0/1720421140 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fdc3402ff10 0x7fdc340323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:34:34.430 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:34.427+0000 7fdc337fe700 1 -- 172.21.15.160:0/1720421140 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdc3c0410b0 con 0x7fdc4c0f78e0 2023-11-17T05:34:34.430 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:34.427+0000 7fdc4a59c700 1 --2- 172.21.15.160:0/1720421140 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fdc3402ff10 0x7fdc340323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:34:34.430 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:34.428+0000 7fdc4a59c700 1 --2- 172.21.15.160:0/1720421140 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fdc3402ff10 0x7fdc340323d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fdc400096a0 tx=0x7fdc40006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:34:34.433 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:34.432+0000 7fdc337fe700 1 -- 172.21.15.160:0/1720421140 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdc3c0095f0 con 0x7fdc4c0f78e0 2023-11-17T05:34:34.638 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:34.637+0000 7fdc50827700 1 -- 172.21.15.160:0/1720421140 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdc4c107f10 con 0x7fdc4c0f78e0 2023-11-17T05:34:34.639 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:34.638+0000 7fdc337fe700 1 -- 172.21.15.160:0/1720421140 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fdc3c02c2c0 con 0x7fdc4c0f78e0 2023-11-17T05:34:34.639 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:34:34.639 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:34:34.640 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:34.640+0000 7fdc50827700 1 -- 172.21.15.160:0/1720421140 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fdc3402ff10 msgr2=0x7fdc340323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:34.641 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:34.640+0000 7fdc50827700 1 --2- 172.21.15.160:0/1720421140 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fdc3402ff10 0x7fdc340323d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fdc400096a0 tx=0x7fdc40006b40).stop 2023-11-17T05:34:34.641 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:34.640+0000 7fdc50827700 1 -- 172.21.15.160:0/1720421140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc4c0f78e0 msgr2=0x7fdc4c1064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:34.641 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:34.640+0000 7fdc50827700 1 --2- 172.21.15.160:0/1720421140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc4c0f78e0 0x7fdc4c1064e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7fdc3c009830 tx=0x7fdc3c00b040).stop 2023-11-17T05:34:34.641 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:34.640+0000 7fdc50827700 1 -- 172.21.15.160:0/1720421140 shutdown_connections 2023-11-17T05:34:34.641 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:34.640+0000 7fdc50827700 1 --2- 172.21.15.160:0/1720421140 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fdc3402ff10 0x7fdc340323d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:34.641 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:34.640+0000 7fdc50827700 1 --2- 172.21.15.160:0/1720421140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc4c0f78e0 0x7fdc4c1064e0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:34.641 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:34.640+0000 7fdc50827700 1 -- 172.21.15.160:0/1720421140 >> 172.21.15.160:0/1720421140 conn(0x7fdc4c0f0120 msgr2=0x7fdc4c0f0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:34:34.641 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:34.640+0000 7fdc50827700 1 -- 172.21.15.160:0/1720421140 shutdown_connections 2023-11-17T05:34:34.641 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:34.640+0000 7fdc50827700 1 -- 172.21.15.160:0/1720421140 wait complete. 2023-11-17T05:34:34.642 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:34:35.021 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:34 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:34:33.477494+0000 mgr.smithi003.eddcuc (mgr.14164) 281 : cluster [DBG] pgmap v267: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:36.021 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:35 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:34:34.639196+0000 mon.smithi003 (mon.0) 250 : audit [DBG] from='client.? 172.21.15.160:0/1720421140' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:34:36.025 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:34:36.025 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:34:37.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:36 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:34:35.477874+0000 mgr.smithi003.eddcuc (mgr.14164) 282 : cluster [DBG] pgmap v268: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:37.258 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:37.256+0000 7fe217dbd700 1 -- 172.21.15.160:0/1895644206 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2100f5410 msgr2=0x7fe2100f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:37.258 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:37.256+0000 7fe217dbd700 1 --2- 172.21.15.160:0/1895644206 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2100f5410 0x7fe2100f5830 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7fe200009fb0 tx=0x7fe200009b30).stop 2023-11-17T05:34:37.258 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:37.256+0000 7fe217dbd700 1 -- 172.21.15.160:0/1895644206 shutdown_connections 2023-11-17T05:34:37.258 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:37.256+0000 7fe217dbd700 1 --2- 172.21.15.160:0/1895644206 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2100f5410 0x7fe2100f5830 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:37.258 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:37.256+0000 7fe217dbd700 1 -- 172.21.15.160:0/1895644206 >> 172.21.15.160:0/1895644206 conn(0x7fe2100f0b70 msgr2=0x7fe2100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:34:37.258 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:37.256+0000 7fe217dbd700 1 -- 172.21.15.160:0/1895644206 shutdown_connections 2023-11-17T05:34:37.258 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:37.256+0000 7fe217dbd700 1 -- 172.21.15.160:0/1895644206 wait complete. 2023-11-17T05:34:37.258 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:37.257+0000 7fe217dbd700 1 Processor -- start 2023-11-17T05:34:37.258 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:37.257+0000 7fe217dbd700 1 -- start start 2023-11-17T05:34:37.259 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:37.257+0000 7fe217dbd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2100f5410 0x7fe2100fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:34:37.259 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:37.257+0000 7fe217dbd700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2100fb820 con 0x7fe2100f5410 2023-11-17T05:34:37.260 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:37.257+0000 7fe215b59700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2100f5410 0x7fe2100fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:34:37.260 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:37.257+0000 7fe215b59700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2100f5410 0x7fe2100fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:36988/0 (socket says 172.21.15.160:36988) 2023-11-17T05:34:37.260 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:37.257+0000 7fe215b59700 1 -- 172.21.15.160:0/619041569 learned_addr learned my addr 172.21.15.160:0/619041569 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:34:37.260 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:37.258+0000 7fe215b59700 1 -- 172.21.15.160:0/619041569 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe200005040 con 0x7fe2100f5410 2023-11-17T05:34:37.260 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:37.258+0000 7fe215b59700 1 --2- 172.21.15.160:0/619041569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2100f5410 0x7fe2100fd3a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7fe200013510 tx=0x7fe20000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:34:37.260 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:37.258+0000 7fe2067fc700 1 -- 172.21.15.160:0/619041569 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe200009320 con 0x7fe2100f5410 2023-11-17T05:34:37.261 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:37.258+0000 7fe217dbd700 1 -- 172.21.15.160:0/619041569 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2100fb9c0 con 0x7fe2100f5410 2023-11-17T05:34:37.261 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:37.258+0000 7fe2067fc700 1 -- 172.21.15.160:0/619041569 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe200013f00 con 0x7fe2100f5410 2023-11-17T05:34:37.261 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:37.258+0000 7fe2067fc700 1 -- 172.21.15.160:0/619041569 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe200014d90 con 0x7fe2100f5410 2023-11-17T05:34:37.261 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:37.258+0000 7fe217dbd700 1 -- 172.21.15.160:0/619041569 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2100fbe50 con 0x7fe2100f5410 2023-11-17T05:34:37.261 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:37.259+0000 7fe2067fc700 1 -- 172.21.15.160:0/619041569 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe2000095f0 con 0x7fe2100f5410 2023-11-17T05:34:37.261 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:37.259+0000 7fe217dbd700 1 -- 172.21.15.160:0/619041569 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe210059100 con 0x7fe2100f5410 2023-11-17T05:34:37.261 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:37.259+0000 7fe2067fc700 1 --2- 172.21.15.160:0/619041569 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe1fc02ff60 0x7fe1fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:34:37.261 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:37.259+0000 7fe2067fc700 1 -- 172.21.15.160:0/619041569 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe200041000 con 0x7fe2100f5410 2023-11-17T05:34:37.261 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:37.259+0000 7fe215358700 1 --2- 172.21.15.160:0/619041569 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe1fc02ff60 0x7fe1fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:34:37.261 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:37.260+0000 7fe215358700 1 --2- 172.21.15.160:0/619041569 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe1fc02ff60 0x7fe1fc032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fe20c0096a0 tx=0x7fe20c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:34:37.265 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:37.264+0000 7fe2067fc700 1 -- 172.21.15.160:0/619041569 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe200014420 con 0x7fe2100f5410 2023-11-17T05:34:37.467 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:37.466+0000 7fe217dbd700 1 -- 172.21.15.160:0/619041569 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe2100fae60 con 0x7fe2100f5410 2023-11-17T05:34:37.468 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:37.467+0000 7fe2067fc700 1 -- 172.21.15.160:0/619041569 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fe200010230 con 0x7fe2100f5410 2023-11-17T05:34:37.468 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:34:37.468 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:34:37.470 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:37.468+0000 7fe217dbd700 1 -- 172.21.15.160:0/619041569 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe1fc02ff60 msgr2=0x7fe1fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:37.470 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:37.469+0000 7fe217dbd700 1 --2- 172.21.15.160:0/619041569 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe1fc02ff60 0x7fe1fc032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fe20c0096a0 tx=0x7fe20c006b40).stop 2023-11-17T05:34:37.470 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:37.469+0000 7fe217dbd700 1 -- 172.21.15.160:0/619041569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2100f5410 msgr2=0x7fe2100fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:37.470 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:37.469+0000 7fe217dbd700 1 --2- 172.21.15.160:0/619041569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2100f5410 0x7fe2100fd3a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7fe200013510 tx=0x7fe20000b040).stop 2023-11-17T05:34:37.470 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:37.469+0000 7fe217dbd700 1 -- 172.21.15.160:0/619041569 shutdown_connections 2023-11-17T05:34:37.470 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:37.469+0000 7fe217dbd700 1 --2- 172.21.15.160:0/619041569 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe1fc02ff60 0x7fe1fc032420 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:37.470 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:37.469+0000 7fe217dbd700 1 --2- 172.21.15.160:0/619041569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2100f5410 0x7fe2100fd3a0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:37.470 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:37.469+0000 7fe217dbd700 1 -- 172.21.15.160:0/619041569 >> 172.21.15.160:0/619041569 conn(0x7fe2100f0b70 msgr2=0x7fe2100f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:34:37.470 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:37.469+0000 7fe217dbd700 1 -- 172.21.15.160:0/619041569 shutdown_connections 2023-11-17T05:34:37.470 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:37.469+0000 7fe217dbd700 1 -- 172.21.15.160:0/619041569 wait complete. 2023-11-17T05:34:37.471 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:34:38.026 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:37 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:34:37.468205+0000 mon.smithi003 (mon.0) 251 : audit [DBG] from='client.? 172.21.15.160:0/619041569' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:34:38.898 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:34:38.899 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:34:39.021 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:34:37.478271+0000 mgr.smithi003.eddcuc (mgr.14164) 283 : cluster [DBG] pgmap v269: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:40.054 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:40.053+0000 7f2ec8811700 1 -- 172.21.15.160:0/2064864736 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ec40f51a0 msgr2=0x7f2ec40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:40.055 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:40.053+0000 7f2ec8811700 1 --2- 172.21.15.160:0/2064864736 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ec40f51a0 0x7f2ec40f55c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f2eb4009fb0 tx=0x7f2eb4009b30).stop 2023-11-17T05:34:40.055 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:40.053+0000 7f2ec8811700 1 -- 172.21.15.160:0/2064864736 shutdown_connections 2023-11-17T05:34:40.055 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:40.053+0000 7f2ec8811700 1 --2- 172.21.15.160:0/2064864736 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ec40f51a0 0x7f2ec40f55c0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:40.055 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:40.053+0000 7f2ec8811700 1 -- 172.21.15.160:0/2064864736 >> 172.21.15.160:0/2064864736 conn(0x7f2ec40f09a0 msgr2=0x7f2ec40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:34:40.055 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:40.053+0000 7f2ec8811700 1 -- 172.21.15.160:0/2064864736 shutdown_connections 2023-11-17T05:34:40.055 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:40.053+0000 7f2ec8811700 1 -- 172.21.15.160:0/2064864736 wait complete. 2023-11-17T05:34:40.055 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:40.053+0000 7f2ec8811700 1 Processor -- start 2023-11-17T05:34:40.055 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:40.054+0000 7f2ec8811700 1 -- start start 2023-11-17T05:34:40.055 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:40.054+0000 7f2ec8811700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ec40f51a0 0x7f2ec40fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:34:40.055 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:40.054+0000 7f2ec8811700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec40fb5a0 con 0x7f2ec40f51a0 2023-11-17T05:34:40.056 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:40.054+0000 7f2ec37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ec40f51a0 0x7f2ec40fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:34:40.057 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:40.054+0000 7f2ec37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ec40f51a0 0x7f2ec40fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:37006/0 (socket says 172.21.15.160:37006) 2023-11-17T05:34:40.057 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:40.054+0000 7f2ec37fe700 1 -- 172.21.15.160:0/3071387839 learned_addr learned my addr 172.21.15.160:0/3071387839 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:34:40.057 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:40.054+0000 7f2ec37fe700 1 -- 172.21.15.160:0/3071387839 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb4005040 con 0x7f2ec40f51a0 2023-11-17T05:34:40.057 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:40.055+0000 7f2ec37fe700 1 --2- 172.21.15.160:0/3071387839 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ec40f51a0 0x7f2ec40fd120 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f2eb4009830 tx=0x7f2eb400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:34:40.057 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:40.055+0000 7f2eabfff700 1 -- 172.21.15.160:0/3071387839 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2eb4009320 con 0x7f2ec40f51a0 2023-11-17T05:34:40.057 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:40.055+0000 7f2ec8811700 1 -- 172.21.15.160:0/3071387839 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ec40fb740 con 0x7f2ec40f51a0 2023-11-17T05:34:40.057 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:40.055+0000 7f2ec8811700 1 -- 172.21.15.160:0/3071387839 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ec40fbbd0 con 0x7f2ec40f51a0 2023-11-17T05:34:40.057 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:40.055+0000 7f2eabfff700 1 -- 172.21.15.160:0/3071387839 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2eb40136c0 con 0x7f2ec40f51a0 2023-11-17T05:34:40.057 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:40.055+0000 7f2eabfff700 1 -- 172.21.15.160:0/3071387839 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2eb401ee90 con 0x7f2ec40f51a0 2023-11-17T05:34:40.057 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:40.056+0000 7f2eabfff700 1 -- 172.21.15.160:0/3071387839 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2eb4014420 con 0x7f2ec40f51a0 2023-11-17T05:34:40.057 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:40.056+0000 7f2eabfff700 1 --2- 172.21.15.160:0/3071387839 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f2eac02ff10 0x7f2eac0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:34:40.057 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:40.056+0000 7f2ec8811700 1 -- 172.21.15.160:0/3071387839 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ec4059100 con 0x7f2ec40f51a0 2023-11-17T05:34:40.057 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:40.056+0000 7f2eabfff700 1 -- 172.21.15.160:0/3071387839 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2eb4040f60 con 0x7f2ec40f51a0 2023-11-17T05:34:40.058 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:40.056+0000 7f2ec2ffd700 1 --2- 172.21.15.160:0/3071387839 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f2eac02ff10 0x7f2eac0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:34:40.058 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:40.057+0000 7f2ec2ffd700 1 --2- 172.21.15.160:0/3071387839 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f2eac02ff10 0x7f2eac0323d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f2eb80096a0 tx=0x7f2eb8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:34:40.062 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:40.061+0000 7f2eabfff700 1 -- 172.21.15.160:0/3071387839 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2eb4027770 con 0x7f2ec40f51a0 2023-11-17T05:34:40.268 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:40.267+0000 7f2ec8811700 1 -- 172.21.15.160:0/3071387839 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2ec40fabe0 con 0x7f2ec40f51a0 2023-11-17T05:34:40.269 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:40.268+0000 7f2eabfff700 1 -- 172.21.15.160:0/3071387839 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f2eb401a230 con 0x7f2ec40f51a0 2023-11-17T05:34:40.269 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:34:40.269 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:34:40.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:39 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:34:39.478646+0000 mgr.smithi003.eddcuc (mgr.14164) 284 : cluster [DBG] pgmap v270: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:40.271 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:40.270+0000 7f2ec8811700 1 -- 172.21.15.160:0/3071387839 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f2eac02ff10 msgr2=0x7f2eac0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:40.272 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:40.270+0000 7f2ec8811700 1 --2- 172.21.15.160:0/3071387839 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f2eac02ff10 0x7f2eac0323d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f2eb80096a0 tx=0x7f2eb8006b40).stop 2023-11-17T05:34:40.272 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:40.270+0000 7f2ec8811700 1 -- 172.21.15.160:0/3071387839 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ec40f51a0 msgr2=0x7f2ec40fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:40.272 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:40.270+0000 7f2ec8811700 1 --2- 172.21.15.160:0/3071387839 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ec40f51a0 0x7f2ec40fd120 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f2eb4009830 tx=0x7f2eb400b040).stop 2023-11-17T05:34:40.272 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:40.270+0000 7f2ec8811700 1 -- 172.21.15.160:0/3071387839 shutdown_connections 2023-11-17T05:34:40.272 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:40.270+0000 7f2ec8811700 1 --2- 172.21.15.160:0/3071387839 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f2eac02ff10 0x7f2eac0323d0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:40.272 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:40.270+0000 7f2ec8811700 1 --2- 172.21.15.160:0/3071387839 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ec40f51a0 0x7f2ec40fd120 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:40.272 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:40.270+0000 7f2ec8811700 1 -- 172.21.15.160:0/3071387839 >> 172.21.15.160:0/3071387839 conn(0x7f2ec40f09a0 msgr2=0x7f2ec405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:34:40.272 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:40.270+0000 7f2ec8811700 1 -- 172.21.15.160:0/3071387839 shutdown_connections 2023-11-17T05:34:40.272 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:40.270+0000 7f2ec8811700 1 -- 172.21.15.160:0/3071387839 wait complete. 2023-11-17T05:34:40.272 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:34:41.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:40 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:34:40.269398+0000 mon.smithi003 (mon.0) 252 : audit [DBG] from='client.? 172.21.15.160:0/3071387839' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:34:41.674 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:34:41.674 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:34:42.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:41 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:34:41.479007+0000 mgr.smithi003.eddcuc (mgr.14164) 285 : cluster [DBG] pgmap v271: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:42.898 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:42.897+0000 7f7904e01700 1 -- 172.21.15.160:0/1917284567 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f79000f51a0 msgr2=0x7f79000f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:42.898 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:42.897+0000 7f7904e01700 1 --2- 172.21.15.160:0/1917284567 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f79000f51a0 0x7f79000f55c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f78f0009fb0 tx=0x7f78f0009b30).stop 2023-11-17T05:34:42.898 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:42.897+0000 7f7904e01700 1 -- 172.21.15.160:0/1917284567 shutdown_connections 2023-11-17T05:34:42.899 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:42.897+0000 7f7904e01700 1 --2- 172.21.15.160:0/1917284567 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f79000f51a0 0x7f79000f55c0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:42.899 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:42.897+0000 7f7904e01700 1 -- 172.21.15.160:0/1917284567 >> 172.21.15.160:0/1917284567 conn(0x7f79000f09a0 msgr2=0x7f79000f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:34:42.899 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:42.897+0000 7f7904e01700 1 -- 172.21.15.160:0/1917284567 shutdown_connections 2023-11-17T05:34:42.899 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:42.897+0000 7f7904e01700 1 -- 172.21.15.160:0/1917284567 wait complete. 2023-11-17T05:34:42.899 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:42.897+0000 7f7904e01700 1 Processor -- start 2023-11-17T05:34:42.899 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:42.898+0000 7f7904e01700 1 -- start start 2023-11-17T05:34:42.900 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:42.898+0000 7f7904e01700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7900060200 0x7f7900109340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:34:42.901 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:42.898+0000 7f7904e01700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7900109830 con 0x7f7900060200 2023-11-17T05:34:42.901 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:42.898+0000 7f78ff7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7900060200 0x7f7900109340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:34:42.901 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:42.898+0000 7f78ff7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7900060200 0x7f7900109340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:53140/0 (socket says 172.21.15.160:53140) 2023-11-17T05:34:42.901 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:42.898+0000 7f78ff7fe700 1 -- 172.21.15.160:0/1499722096 learned_addr learned my addr 172.21.15.160:0/1499722096 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:34:42.901 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:42.898+0000 7f78ff7fe700 1 -- 172.21.15.160:0/1499722096 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f0005040 con 0x7f7900060200 2023-11-17T05:34:42.901 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:42.899+0000 7f78ff7fe700 1 --2- 172.21.15.160:0/1499722096 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7900060200 0x7f7900109340 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f78f0009f80 tx=0x7f78f0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:34:42.901 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:42.899+0000 7f78f7fff700 1 -- 172.21.15.160:0/1499722096 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78f00136c0 con 0x7f7900060200 2023-11-17T05:34:42.901 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:42.899+0000 7f7904e01700 1 -- 172.21.15.160:0/1499722096 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7900109970 con 0x7f7900060200 2023-11-17T05:34:42.901 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:42.899+0000 7f78f7fff700 1 -- 172.21.15.160:0/1499722096 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f78f00095f0 con 0x7f7900060200 2023-11-17T05:34:42.901 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:42.899+0000 7f78f7fff700 1 -- 172.21.15.160:0/1499722096 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78f0014420 con 0x7f7900060200 2023-11-17T05:34:42.901 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:42.899+0000 7f7904e01700 1 -- 172.21.15.160:0/1499722096 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7900109da0 con 0x7f7900060200 2023-11-17T05:34:42.901 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:42.900+0000 7f78f7fff700 1 -- 172.21.15.160:0/1499722096 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f78f0025060 con 0x7f7900060200 2023-11-17T05:34:42.901 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:42.900+0000 7f78f7fff700 1 --2- 172.21.15.160:0/1499722096 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f78ec02ff60 0x7f78ec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:34:42.902 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:42.900+0000 7f78f7fff700 1 -- 172.21.15.160:0/1499722096 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f78f0040eb0 con 0x7f7900060200 2023-11-17T05:34:42.902 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:42.900+0000 7f7904e01700 1 -- 172.21.15.160:0/1499722096 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7900045ad0 con 0x7f7900060200 2023-11-17T05:34:42.902 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:42.900+0000 7f78feffd700 1 --2- 172.21.15.160:0/1499722096 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f78ec02ff60 0x7f78ec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:34:42.902 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:42.901+0000 7f78feffd700 1 --2- 172.21.15.160:0/1499722096 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f78ec02ff60 0x7f78ec032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f78e80096a0 tx=0x7f78e8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:34:42.905 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:42.906+0000 7f78f7fff700 1 -- 172.21.15.160:0/1499722096 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f78f0028440 con 0x7f7900060200 2023-11-17T05:34:43.109 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:43.108+0000 7f7904e01700 1 -- 172.21.15.160:0/1499722096 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f790010a500 con 0x7f7900060200 2023-11-17T05:34:43.109 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:43.109+0000 7f78f7fff700 1 -- 172.21.15.160:0/1499722096 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f78f0026340 con 0x7f7900060200 2023-11-17T05:34:43.109 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:34:43.110 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:34:43.111 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:43.110+0000 7f7904e01700 1 -- 172.21.15.160:0/1499722096 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f78ec02ff60 msgr2=0x7f78ec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:43.111 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:43.110+0000 7f7904e01700 1 --2- 172.21.15.160:0/1499722096 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f78ec02ff60 0x7f78ec032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f78e80096a0 tx=0x7f78e8006b40).stop 2023-11-17T05:34:43.111 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:43.110+0000 7f7904e01700 1 -- 172.21.15.160:0/1499722096 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7900060200 msgr2=0x7f7900109340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:43.111 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:43.110+0000 7f7904e01700 1 --2- 172.21.15.160:0/1499722096 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7900060200 0x7f7900109340 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f78f0009f80 tx=0x7f78f0009b30).stop 2023-11-17T05:34:43.111 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:43.110+0000 7f7904e01700 1 -- 172.21.15.160:0/1499722096 shutdown_connections 2023-11-17T05:34:43.111 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:43.110+0000 7f7904e01700 1 --2- 172.21.15.160:0/1499722096 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f78ec02ff60 0x7f78ec032420 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:43.112 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:43.110+0000 7f7904e01700 1 --2- 172.21.15.160:0/1499722096 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7900060200 0x7f7900109340 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:43.112 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:43.110+0000 7f7904e01700 1 -- 172.21.15.160:0/1499722096 >> 172.21.15.160:0/1499722096 conn(0x7f79000f09a0 msgr2=0x7f79000f1500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:34:43.112 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:43.110+0000 7f7904e01700 1 -- 172.21.15.160:0/1499722096 shutdown_connections 2023-11-17T05:34:43.112 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:43.110+0000 7f7904e01700 1 -- 172.21.15.160:0/1499722096 wait complete. 2023-11-17T05:34:43.112 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:34:43.520 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:43 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:34:43.109852+0000 mon.smithi003 (mon.0) 253 : audit [DBG] from='client.? 172.21.15.160:0/1499722096' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:34:44.520 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:44 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:34:43.479359+0000 mgr.smithi003.eddcuc (mgr.14164) 286 : cluster [DBG] pgmap v272: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:44.528 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:34:44.528 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:34:45.690 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:45.688+0000 7f79e4933700 1 -- 172.21.15.160:0/748816164 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f79e00f5410 msgr2=0x7f79e00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:45.690 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:45.688+0000 7f79e4933700 1 --2- 172.21.15.160:0/748816164 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f79e00f5410 0x7f79e00f5830 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f79c8009fb0 tx=0x7f79c8009b30).stop 2023-11-17T05:34:45.690 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:45.688+0000 7f79e4933700 1 -- 172.21.15.160:0/748816164 shutdown_connections 2023-11-17T05:34:45.690 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:45.688+0000 7f79e4933700 1 --2- 172.21.15.160:0/748816164 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f79e00f5410 0x7f79e00f5830 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:45.691 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:45.688+0000 7f79e4933700 1 -- 172.21.15.160:0/748816164 >> 172.21.15.160:0/748816164 conn(0x7f79e00f0b70 msgr2=0x7f79e00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:34:45.691 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:45.688+0000 7f79e4933700 1 -- 172.21.15.160:0/748816164 shutdown_connections 2023-11-17T05:34:45.691 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:45.688+0000 7f79e4933700 1 -- 172.21.15.160:0/748816164 wait complete. 2023-11-17T05:34:45.691 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:45.688+0000 7f79e4933700 1 Processor -- start 2023-11-17T05:34:45.691 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:45.688+0000 7f79e4933700 1 -- start start 2023-11-17T05:34:45.691 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:45.688+0000 7f79e4933700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f79e00f5410 0x7f79e00fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:34:45.691 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:45.688+0000 7f79e4933700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e00fb820 con 0x7f79e00f5410 2023-11-17T05:34:45.691 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:45.689+0000 7f79de59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f79e00f5410 0x7f79e00fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:34:45.691 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:45.689+0000 7f79de59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f79e00f5410 0x7f79e00fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:53152/0 (socket says 172.21.15.160:53152) 2023-11-17T05:34:45.691 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:45.689+0000 7f79de59c700 1 -- 172.21.15.160:0/2905240870 learned_addr learned my addr 172.21.15.160:0/2905240870 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:34:45.691 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:45.689+0000 7f79de59c700 1 -- 172.21.15.160:0/2905240870 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79c8005040 con 0x7f79e00f5410 2023-11-17T05:34:45.691 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:45.689+0000 7f79de59c700 1 --2- 172.21.15.160:0/2905240870 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f79e00f5410 0x7f79e00fd3a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f79c8013510 tx=0x7f79c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:34:45.691 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:45.689+0000 7f79d6ffd700 1 -- 172.21.15.160:0/2905240870 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f79c8009320 con 0x7f79e00f5410 2023-11-17T05:34:45.692 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:45.689+0000 7f79e4933700 1 -- 172.21.15.160:0/2905240870 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79e00fb9c0 con 0x7f79e00f5410 2023-11-17T05:34:45.692 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:45.689+0000 7f79d6ffd700 1 -- 172.21.15.160:0/2905240870 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f79c8013f00 con 0x7f79e00f5410 2023-11-17T05:34:45.692 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:45.689+0000 7f79d6ffd700 1 -- 172.21.15.160:0/2905240870 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f79c8014d90 con 0x7f79e00f5410 2023-11-17T05:34:45.692 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:45.689+0000 7f79e4933700 1 -- 172.21.15.160:0/2905240870 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79e00fbe50 con 0x7f79e00f5410 2023-11-17T05:34:45.693 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:45.690+0000 7f79d6ffd700 1 -- 172.21.15.160:0/2905240870 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f79c80263a0 con 0x7f79e00f5410 2023-11-17T05:34:45.693 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:45.690+0000 7f79d6ffd700 1 --2- 172.21.15.160:0/2905240870 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f79cc02ff60 0x7f79cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:34:45.693 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:45.690+0000 7f79d6ffd700 1 -- 172.21.15.160:0/2905240870 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f79c80412d0 con 0x7f79e00f5410 2023-11-17T05:34:45.693 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:45.690+0000 7f79e4933700 1 -- 172.21.15.160:0/2905240870 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79e0059100 con 0x7f79e00f5410 2023-11-17T05:34:45.693 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:45.691+0000 7f79ddd9b700 1 --2- 172.21.15.160:0/2905240870 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f79cc02ff60 0x7f79cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:34:45.693 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:45.691+0000 7f79ddd9b700 1 --2- 172.21.15.160:0/2905240870 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f79cc02ff60 0x7f79cc032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f79d00096a0 tx=0x7f79d0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:34:45.696 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:45.696+0000 7f79d6ffd700 1 -- 172.21.15.160:0/2905240870 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f79c8012330 con 0x7f79e00f5410 2023-11-17T05:34:45.901 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:45.900+0000 7f79e4933700 1 -- 172.21.15.160:0/2905240870 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f79e00fae60 con 0x7f79e00f5410 2023-11-17T05:34:45.902 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:45.901+0000 7f79d6ffd700 1 -- 172.21.15.160:0/2905240870 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f79c802b030 con 0x7f79e00f5410 2023-11-17T05:34:45.902 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:34:45.902 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:34:45.903 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:45.902+0000 7f79e4933700 1 -- 172.21.15.160:0/2905240870 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f79cc02ff60 msgr2=0x7f79cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:45.904 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:45.902+0000 7f79e4933700 1 --2- 172.21.15.160:0/2905240870 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f79cc02ff60 0x7f79cc032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f79d00096a0 tx=0x7f79d0006b40).stop 2023-11-17T05:34:45.904 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:45.902+0000 7f79e4933700 1 -- 172.21.15.160:0/2905240870 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f79e00f5410 msgr2=0x7f79e00fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:45.904 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:45.903+0000 7f79e4933700 1 --2- 172.21.15.160:0/2905240870 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f79e00f5410 0x7f79e00fd3a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f79c8013510 tx=0x7f79c800b040).stop 2023-11-17T05:34:45.904 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:45.903+0000 7f79e4933700 1 -- 172.21.15.160:0/2905240870 shutdown_connections 2023-11-17T05:34:45.904 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:45.903+0000 7f79e4933700 1 --2- 172.21.15.160:0/2905240870 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f79cc02ff60 0x7f79cc032420 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:45.904 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:45.903+0000 7f79e4933700 1 --2- 172.21.15.160:0/2905240870 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f79e00f5410 0x7f79e00fd3a0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:45.904 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:45.903+0000 7f79e4933700 1 -- 172.21.15.160:0/2905240870 >> 172.21.15.160:0/2905240870 conn(0x7f79e00f0b70 msgr2=0x7f79e005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:34:45.904 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:45.903+0000 7f79e4933700 1 -- 172.21.15.160:0/2905240870 shutdown_connections 2023-11-17T05:34:45.904 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:45.903+0000 7f79e4933700 1 -- 172.21.15.160:0/2905240870 wait complete. 2023-11-17T05:34:45.905 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:34:47.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:46 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:34:45.479713+0000 mgr.smithi003.eddcuc (mgr.14164) 287 : cluster [DBG] pgmap v273: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:47.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:46 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:34:45.902107+0000 mon.smithi003 (mon.0) 254 : audit [DBG] from='client.? 172.21.15.160:0/2905240870' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:34:47.292 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:34:47.292 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:34:48.602 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:48.600+0000 7f2d72291700 1 -- 172.21.15.160:0/788872738 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d6c0f51a0 msgr2=0x7f2d6c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:48.602 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:48.600+0000 7f2d72291700 1 --2- 172.21.15.160:0/788872738 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d6c0f51a0 0x7f2d6c0f55c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f2d68009fb0 tx=0x7f2d68009b30).stop 2023-11-17T05:34:48.602 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:48.600+0000 7f2d72291700 1 -- 172.21.15.160:0/788872738 shutdown_connections 2023-11-17T05:34:48.602 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:48.600+0000 7f2d72291700 1 --2- 172.21.15.160:0/788872738 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d6c0f51a0 0x7f2d6c0f55c0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:48.602 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:48.600+0000 7f2d72291700 1 -- 172.21.15.160:0/788872738 >> 172.21.15.160:0/788872738 conn(0x7f2d6c0f09a0 msgr2=0x7f2d6c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:34:48.602 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:48.600+0000 7f2d72291700 1 -- 172.21.15.160:0/788872738 shutdown_connections 2023-11-17T05:34:48.603 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:48.600+0000 7f2d72291700 1 -- 172.21.15.160:0/788872738 wait complete. 2023-11-17T05:34:48.603 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:48.600+0000 7f2d72291700 1 Processor -- start 2023-11-17T05:34:48.603 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:48.600+0000 7f2d72291700 1 -- start start 2023-11-17T05:34:48.603 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:48.601+0000 7f2d72291700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d6c0f51a0 0x7f2d6c1053e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:34:48.603 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:48.601+0000 7f2d72291700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6c1058d0 con 0x7f2d6c0f51a0 2023-11-17T05:34:48.603 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:48.601+0000 7f2d7128f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d6c0f51a0 0x7f2d6c1053e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:34:48.603 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:48.601+0000 7f2d7128f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d6c0f51a0 0x7f2d6c1053e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:53174/0 (socket says 172.21.15.160:53174) 2023-11-17T05:34:48.603 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:48.601+0000 7f2d7128f700 1 -- 172.21.15.160:0/490698926 learned_addr learned my addr 172.21.15.160:0/490698926 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:34:48.603 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:48.601+0000 7f2d7128f700 1 -- 172.21.15.160:0/490698926 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d68005040 con 0x7f2d6c0f51a0 2023-11-17T05:34:48.603 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:48.601+0000 7f2d7128f700 1 --2- 172.21.15.160:0/490698926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d6c0f51a0 0x7f2d6c1053e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f2d68013510 tx=0x7f2d6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:34:48.604 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:48.601+0000 7f2d61ffb700 1 -- 172.21.15.160:0/490698926 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d68009320 con 0x7f2d6c0f51a0 2023-11-17T05:34:48.604 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:48.601+0000 7f2d72291700 1 -- 172.21.15.160:0/490698926 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d6c105ad0 con 0x7f2d6c0f51a0 2023-11-17T05:34:48.604 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:48.601+0000 7f2d61ffb700 1 -- 172.21.15.160:0/490698926 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2d68013f00 con 0x7f2d6c0f51a0 2023-11-17T05:34:48.604 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:48.601+0000 7f2d61ffb700 1 -- 172.21.15.160:0/490698926 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d68014dc0 con 0x7f2d6c0f51a0 2023-11-17T05:34:48.604 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:48.602+0000 7f2d72291700 1 -- 172.21.15.160:0/490698926 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d6c105f60 con 0x7f2d6c0f51a0 2023-11-17T05:34:48.605 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:48.603+0000 7f2d72291700 1 -- 172.21.15.160:0/490698926 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d6c045ad0 con 0x7f2d6c0f51a0 2023-11-17T05:34:48.605 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:48.603+0000 7f2d61ffb700 1 -- 172.21.15.160:0/490698926 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2d680095f0 con 0x7f2d6c0f51a0 2023-11-17T05:34:48.605 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:48.603+0000 7f2d61ffb700 1 --2- 172.21.15.160:0/490698926 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f2d5802ff60 0x7f2d58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:34:48.605 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:48.603+0000 7f2d61ffb700 1 -- 172.21.15.160:0/490698926 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2d68041020 con 0x7f2d6c0f51a0 2023-11-17T05:34:48.607 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:48.603+0000 7f2d70a8e700 1 --2- 172.21.15.160:0/490698926 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f2d5802ff60 0x7f2d58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:34:48.607 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:48.603+0000 7f2d70a8e700 1 --2- 172.21.15.160:0/490698926 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f2d5802ff60 0x7f2d58032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f2d5c0096a0 tx=0x7f2d5c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:34:48.607 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:48.608+0000 7f2d61ffb700 1 -- 172.21.15.160:0/490698926 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2d68014420 con 0x7f2d6c0f51a0 2023-11-17T05:34:48.811 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:48.811+0000 7f2d72291700 1 -- 172.21.15.160:0/490698926 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2d6c106db0 con 0x7f2d6c0f51a0 2023-11-17T05:34:48.812 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:48.812+0000 7f2d61ffb700 1 -- 172.21.15.160:0/490698926 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f2d6801c920 con 0x7f2d6c0f51a0 2023-11-17T05:34:48.812 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:34:48.812 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:34:48.814 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:48.814+0000 7f2d72291700 1 -- 172.21.15.160:0/490698926 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f2d5802ff60 msgr2=0x7f2d58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:48.814 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:48.814+0000 7f2d72291700 1 --2- 172.21.15.160:0/490698926 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f2d5802ff60 0x7f2d58032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f2d5c0096a0 tx=0x7f2d5c006b40).stop 2023-11-17T05:34:48.814 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:48.814+0000 7f2d72291700 1 -- 172.21.15.160:0/490698926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d6c0f51a0 msgr2=0x7f2d6c1053e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:48.814 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:48.814+0000 7f2d72291700 1 --2- 172.21.15.160:0/490698926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d6c0f51a0 0x7f2d6c1053e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f2d68013510 tx=0x7f2d6800b040).stop 2023-11-17T05:34:48.814 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:48.814+0000 7f2d72291700 1 -- 172.21.15.160:0/490698926 shutdown_connections 2023-11-17T05:34:48.814 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:48.814+0000 7f2d72291700 1 --2- 172.21.15.160:0/490698926 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f2d5802ff60 0x7f2d58032420 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:48.815 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:48.814+0000 7f2d72291700 1 --2- 172.21.15.160:0/490698926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d6c0f51a0 0x7f2d6c1053e0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:48.815 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:48.814+0000 7f2d72291700 1 -- 172.21.15.160:0/490698926 >> 172.21.15.160:0/490698926 conn(0x7f2d6c0f09a0 msgr2=0x7f2d6c0f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:34:48.815 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:48.814+0000 7f2d72291700 1 -- 172.21.15.160:0/490698926 shutdown_connections 2023-11-17T05:34:48.815 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:48.814+0000 7f2d72291700 1 -- 172.21.15.160:0/490698926 wait complete. 2023-11-17T05:34:48.815 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:34:49.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:48 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:34:47.480051+0000 mgr.smithi003.eddcuc (mgr.14164) 288 : cluster [DBG] pgmap v274: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:50.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:49 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:34:48.812318+0000 mon.smithi003 (mon.0) 255 : audit [DBG] from='client.? 172.21.15.160:0/490698926' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:34:50.194 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:34:50.194 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:34:51.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:50 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:34:49.480398+0000 mgr.smithi003.eddcuc (mgr.14164) 289 : cluster [DBG] pgmap v275: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:34:51.399 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:51.395+0000 7f4a0ffc4700 1 -- 172.21.15.160:0/2833007743 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a080f5410 msgr2=0x7f4a080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:34:51.399 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:51.395+0000 7f4a0ffc4700 1 --2- 172.21.15.160:0/2833007743 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a080f5410 0x7f4a080f5830 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f49f8009fb0 tx=0x7f49f8009b30).stop 2023-11-17T05:34:51.400 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:51.395+0000 7f4a0ffc4700 1 -- 172.21.15.160:0/2833007743 shutdown_connections 2023-11-17T05:34:51.400 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:51.395+0000 7f4a0ffc4700 1 --2- 172.21.15.160:0/2833007743 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a080f5410 0x7f4a080f5830 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:34:51.400 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:51.395+0000 7f4a0ffc4700 1 -- 172.21.15.160:0/2833007743 >> 172.21.15.160:0/2833007743 conn(0x7f4a080f0b70 msgr2=0x7f4a080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:34:51.400 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:51.396+0000 7f4a0ffc4700 1 -- 172.21.15.160:0/2833007743 shutdown_connections 2023-11-17T05:34:51.400 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:51.396+0000 7f4a0ffc4700 1 -- 172.21.15.160:0/2833007743 wait complete. 2023-11-17T05:34:51.421 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:51.396+0000 7f4a0ffc4700 1 Processor -- start 2023-11-17T05:34:51.421 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:51.396+0000 7f4a0ffc4700 1 -- start start 2023-11-17T05:34:51.421 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:51.396+0000 7f4a0ffc4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a080f5410 0x7f4a08105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:34:51.421 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:51.396+0000 7f4a0ffc4700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a08105b20 con 0x7f4a080f5410 2023-11-17T05:34:51.421 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:51.397+0000 7f4a0dd60700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a080f5410 0x7f4a08105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:34:51.421 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:51.397+0000 7f4a0dd60700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a080f5410 0x7f4a08105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:42294/0 (socket says 172.21.15.160:42294) 2023-11-17T05:34:51.421 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:51.397+0000 7f4a0dd60700 1 -- 172.21.15.160:0/1829794321 learned_addr learned my addr 172.21.15.160:0/1829794321 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:34:51.421 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:51.397+0000 7f4a0dd60700 1 -- 172.21.15.160:0/1829794321 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49f8005040 con 0x7f4a080f5410 2023-11-17T05:34:51.421 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:51.397+0000 7f4a0dd60700 1 --2- 172.21.15.160:0/1829794321 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a080f5410 0x7f4a08105630 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f49f8013510 tx=0x7f49f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:34:51.421 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:51.397+0000 7f49fe7fc700 1 -- 172.21.15.160:0/1829794321 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f49f8009320 con 0x7f4a080f5410 2023-11-17T05:34:51.421 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:51.397+0000 7f49fe7fc700 1 -- 172.21.15.160:0/1829794321 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f49f8013f00 con 0x7f4a080f5410 2023-11-17T05:34:51.422 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:51.397+0000 7f4a0ffc4700 1 -- 172.21.15.160:0/1829794321 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a08105d20 con 0x7f4a080f5410 2023-11-17T05:34:51.422 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:51.397+0000 7f49fe7fc700 1 -- 172.21.15.160:0/1829794321 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f49f8014d90 con 0x7f4a080f5410 2023-11-17T05:34:51.422 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:51.397+0000 7f4a0ffc4700 1 -- 172.21.15.160:0/1829794321 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a081061b0 con 0x7f4a080f5410 2023-11-17T05:34:51.422 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:51.398+0000 7f49fe7fc700 1 -- 172.21.15.160:0/1829794321 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f49f80095f0 con 0x7f4a080f5410 2023-11-17T05:34:51.422 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:51.398+0000 7f49fe7fc700 1 --2- 172.21.15.160:0/1829794321 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f49f402ff60 0x7f49f4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:34:51.422 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:51.398+0000 7f49fe7fc700 1 -- 172.21.15.160:0/1829794321 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f49f8042d90 con 0x7f4a080f5410 2023-11-17T05:34:51.422 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:51.398+0000 7f4a0ffc4700 1 -- 172.21.15.160:0/1829794321 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a08045ad0 con 0x7f4a080f5410 2023-11-17T05:34:51.422 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:51.399+0000 7f4a0d55f700 1 --2- 172.21.15.160:0/1829794321 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f49f402ff60 0x7f49f4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:34:51.423 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:51.399+0000 7f4a0d55f700 1 --2- 172.21.15.160:0/1829794321 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f49f402ff60 0x7f49f4032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f4a040096a0 tx=0x7f4a04006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:34:51.423 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:51.403+0000 7f49fe7fc700 1 -- 172.21.15.160:0/1829794321 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f49f801f7f0 con 0x7f4a080f5410 2023-11-17T05:34:51.608 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:51.607+0000 7f4a0ffc4700 1 -- 172.21.15.160:0/1829794321 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4a080faf50 con 0x7f4a080f5410 2023-11-17T05:35:05.929 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:51.608+0000 7f49fe7fc700 1 -- 172.21.15.160:0/1829794321 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f49f8011290 con 0x7f4a080f5410 2023-11-17T05:35:05.929 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:35:05.930 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:35:05.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:52 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:34:51.480803+0000 mgr.smithi003.eddcuc (mgr.14164) 290 : cluster [DBG] pgmap v276: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:05.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:52 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:34:51.608643+0000 mon.smithi003 (mon.0) 256 : audit [DBG] from='client.? 172.21.15.160:0/1829794321' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:35:05.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:54 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:34:53.481131+0000 mgr.smithi003.eddcuc (mgr.14164) 291 : cluster [DBG] pgmap v277: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:05.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:56 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:34:55.481454+0000 mgr.smithi003.eddcuc (mgr.14164) 292 : cluster [DBG] pgmap v278: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:05.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:58 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:34:57.481860+0000 mgr.smithi003.eddcuc (mgr.14164) 293 : cluster [DBG] pgmap v279: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:05.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:34:59 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:34:59.482176+0000 mgr.smithi003.eddcuc (mgr.14164) 294 : cluster [DBG] pgmap v280: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:05.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:02 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:35:01.482536+0000 mgr.smithi003.eddcuc (mgr.14164) 295 : cluster [DBG] pgmap v281: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:05.931 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:04 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:35:03.482857+0000 mgr.smithi003.eddcuc (mgr.14164) 296 : cluster [DBG] pgmap v282: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:05.932 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:51.609+0000 7f4a0ffc4700 1 -- 172.21.15.160:0/1829794321 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f49f402ff60 msgr2=0x7f49f4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:35:05.932 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:51.609+0000 7f4a0ffc4700 1 --2- 172.21.15.160:0/1829794321 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f49f402ff60 0x7f49f4032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f4a040096a0 tx=0x7f4a04006b40).stop 2023-11-17T05:35:05.932 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:51.609+0000 7f4a0ffc4700 1 -- 172.21.15.160:0/1829794321 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a080f5410 msgr2=0x7f4a08105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:35:05.933 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:51.609+0000 7f4a0ffc4700 1 --2- 172.21.15.160:0/1829794321 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a080f5410 0x7f4a08105630 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f49f8013510 tx=0x7f49f800b040).stop 2023-11-17T05:35:05.933 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:51.609+0000 7f4a0ffc4700 1 -- 172.21.15.160:0/1829794321 shutdown_connections 2023-11-17T05:35:05.933 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:51.609+0000 7f4a0ffc4700 1 --2- 172.21.15.160:0/1829794321 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f49f402ff60 0x7f49f4032420 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:35:05.933 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:51.609+0000 7f4a0ffc4700 1 --2- 172.21.15.160:0/1829794321 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a080f5410 0x7f4a08105630 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:35:05.933 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:51.609+0000 7f4a0ffc4700 1 -- 172.21.15.160:0/1829794321 >> 172.21.15.160:0/1829794321 conn(0x7f4a080f0b70 msgr2=0x7f4a080f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:35:05.933 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:51.609+0000 7f4a0ffc4700 1 -- 172.21.15.160:0/1829794321 shutdown_connections 2023-11-17T05:35:05.933 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:34:51.609+0000 7f4a0ffc4700 1 -- 172.21.15.160:0/1829794321 wait complete. 2023-11-17T05:35:05.933 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:35:06.934 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:35:06.935 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:35:07.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:06 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:35:05.483213+0000 mgr.smithi003.eddcuc (mgr.14164) 297 : cluster [DBG] pgmap v283: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:08.133 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:08.131+0000 7fc78f4aa700 1 -- 172.21.15.160:0/2178233506 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7880f51a0 msgr2=0x7fc7880f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:35:08.134 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:08.131+0000 7fc78f4aa700 1 --2- 172.21.15.160:0/2178233506 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7880f51a0 0x7fc7880f55c0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7fc778009fb0 tx=0x7fc778009b30).stop 2023-11-17T05:35:08.134 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:08.131+0000 7fc78f4aa700 1 -- 172.21.15.160:0/2178233506 shutdown_connections 2023-11-17T05:35:08.134 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:08.131+0000 7fc78f4aa700 1 --2- 172.21.15.160:0/2178233506 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7880f51a0 0x7fc7880f55c0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:35:08.134 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:08.131+0000 7fc78f4aa700 1 -- 172.21.15.160:0/2178233506 >> 172.21.15.160:0/2178233506 conn(0x7fc7880f09a0 msgr2=0x7fc7880f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:35:08.134 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:08.131+0000 7fc78f4aa700 1 -- 172.21.15.160:0/2178233506 shutdown_connections 2023-11-17T05:35:08.134 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:08.131+0000 7fc78f4aa700 1 -- 172.21.15.160:0/2178233506 wait complete. 2023-11-17T05:35:08.134 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:08.131+0000 7fc78f4aa700 1 Processor -- start 2023-11-17T05:35:08.134 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:08.131+0000 7fc78f4aa700 1 -- start start 2023-11-17T05:35:08.134 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:08.131+0000 7fc78f4aa700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7880f51a0 0x7fc7881091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:35:08.134 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:08.131+0000 7fc78f4aa700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc788060220 con 0x7fc7880f51a0 2023-11-17T05:35:08.134 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:08.132+0000 7fc78e4a8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7880f51a0 0x7fc7881091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:35:08.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:08.132+0000 7fc78e4a8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7880f51a0 0x7fc7881091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:53582/0 (socket says 172.21.15.160:53582) 2023-11-17T05:35:08.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:08.132+0000 7fc78e4a8700 1 -- 172.21.15.160:0/3352423036 learned_addr learned my addr 172.21.15.160:0/3352423036 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:35:08.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:08.132+0000 7fc78e4a8700 1 -- 172.21.15.160:0/3352423036 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc778005040 con 0x7fc7880f51a0 2023-11-17T05:35:08.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:08.132+0000 7fc78e4a8700 1 --2- 172.21.15.160:0/3352423036 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7880f51a0 0x7fc7881091d0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7fc778013510 tx=0x7fc77800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:35:08.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:08.132+0000 7fc77effd700 1 -- 172.21.15.160:0/3352423036 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc778009320 con 0x7fc7880f51a0 2023-11-17T05:35:08.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:08.132+0000 7fc78f4aa700 1 -- 172.21.15.160:0/3352423036 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc788060420 con 0x7fc7880f51a0 2023-11-17T05:35:08.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:08.132+0000 7fc77effd700 1 -- 172.21.15.160:0/3352423036 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc778013f00 con 0x7fc7880f51a0 2023-11-17T05:35:08.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:08.132+0000 7fc77effd700 1 -- 172.21.15.160:0/3352423036 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc778014dc0 con 0x7fc7880f51a0 2023-11-17T05:35:08.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:08.132+0000 7fc78f4aa700 1 -- 172.21.15.160:0/3352423036 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc788109880 con 0x7fc7880f51a0 2023-11-17T05:35:08.136 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:08.133+0000 7fc77effd700 1 -- 172.21.15.160:0/3352423036 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc7780095f0 con 0x7fc7880f51a0 2023-11-17T05:35:08.136 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:08.133+0000 7fc78f4aa700 1 -- 172.21.15.160:0/3352423036 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc788059100 con 0x7fc7880f51a0 2023-11-17T05:35:08.136 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:08.133+0000 7fc77effd700 1 --2- 172.21.15.160:0/3352423036 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc77402ff60 0x7fc774032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:35:08.136 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:08.133+0000 7fc77effd700 1 -- 172.21.15.160:0/3352423036 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc778041030 con 0x7fc7880f51a0 2023-11-17T05:35:08.137 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:08.133+0000 7fc78dca7700 1 --2- 172.21.15.160:0/3352423036 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc77402ff60 0x7fc774032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:35:08.137 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:08.134+0000 7fc78dca7700 1 --2- 172.21.15.160:0/3352423036 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc77402ff60 0x7fc774032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fc7840096a0 tx=0x7fc784006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:35:08.139 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:08.138+0000 7fc77effd700 1 -- 172.21.15.160:0/3352423036 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc778014420 con 0x7fc7880f51a0 2023-11-17T05:35:08.347 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:08.346+0000 7fc78f4aa700 1 -- 172.21.15.160:0/3352423036 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc788045ad0 con 0x7fc7880f51a0 2023-11-17T05:35:08.347 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:08.347+0000 7fc77effd700 1 -- 172.21.15.160:0/3352423036 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fc778010230 con 0x7fc7880f51a0 2023-11-17T05:35:08.347 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:35:08.348 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:35:08.349 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:08.348+0000 7fc78f4aa700 1 -- 172.21.15.160:0/3352423036 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc77402ff60 msgr2=0x7fc774032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:35:08.350 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:08.348+0000 7fc78f4aa700 1 --2- 172.21.15.160:0/3352423036 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc77402ff60 0x7fc774032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fc7840096a0 tx=0x7fc784006b40).stop 2023-11-17T05:35:08.350 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:08.348+0000 7fc78f4aa700 1 -- 172.21.15.160:0/3352423036 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7880f51a0 msgr2=0x7fc7881091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:35:08.350 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:08.348+0000 7fc78f4aa700 1 --2- 172.21.15.160:0/3352423036 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7880f51a0 0x7fc7881091d0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7fc778013510 tx=0x7fc77800b040).stop 2023-11-17T05:35:08.350 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:08.348+0000 7fc78f4aa700 1 -- 172.21.15.160:0/3352423036 shutdown_connections 2023-11-17T05:35:08.350 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:08.348+0000 7fc78f4aa700 1 --2- 172.21.15.160:0/3352423036 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc77402ff60 0x7fc774032420 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:35:08.350 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:08.348+0000 7fc78f4aa700 1 --2- 172.21.15.160:0/3352423036 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc7880f51a0 0x7fc7881091d0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:35:08.350 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:08.348+0000 7fc78f4aa700 1 -- 172.21.15.160:0/3352423036 >> 172.21.15.160:0/3352423036 conn(0x7fc7880f09a0 msgr2=0x7fc7880f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:35:08.350 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:08.348+0000 7fc78f4aa700 1 -- 172.21.15.160:0/3352423036 shutdown_connections 2023-11-17T05:35:08.350 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:08.349+0000 7fc78f4aa700 1 -- 172.21.15.160:0/3352423036 wait complete. 2023-11-17T05:35:08.351 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:35:09.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:08 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:35:07.483636+0000 mgr.smithi003.eddcuc (mgr.14164) 298 : cluster [DBG] pgmap v284: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:09.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:08 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:35:08.347717+0000 mon.smithi003 (mon.0) 257 : audit [DBG] from='client.? 172.21.15.160:0/3352423036' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:35:09.725 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:35:09.725 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:35:10.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:09 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:35:09.483944+0000 mgr.smithi003.eddcuc (mgr.14164) 299 : cluster [DBG] pgmap v285: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:10.888 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:10.881+0000 7f450f0c1700 1 -- 172.21.15.160:0/3601204259 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4508060280 msgr2=0x7f45080f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:35:10.888 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:10.881+0000 7f450f0c1700 1 --2- 172.21.15.160:0/3601204259 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4508060280 0x7f45080f90f0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f44fc009fb0 tx=0x7f44fc009b30).stop 2023-11-17T05:35:10.888 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:10.881+0000 7f450f0c1700 1 -- 172.21.15.160:0/3601204259 shutdown_connections 2023-11-17T05:35:10.888 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:10.881+0000 7f450f0c1700 1 --2- 172.21.15.160:0/3601204259 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4508060280 0x7f45080f90f0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:35:10.888 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:10.881+0000 7f450f0c1700 1 -- 172.21.15.160:0/3601204259 >> 172.21.15.160:0/3601204259 conn(0x7f45080f0b70 msgr2=0x7f45080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:35:10.888 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:10.881+0000 7f450f0c1700 1 -- 172.21.15.160:0/3601204259 shutdown_connections 2023-11-17T05:35:10.888 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:10.881+0000 7f450f0c1700 1 -- 172.21.15.160:0/3601204259 wait complete. 2023-11-17T05:35:10.889 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:10.882+0000 7f450f0c1700 1 Processor -- start 2023-11-17T05:35:10.889 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:10.882+0000 7f450f0c1700 1 -- start start 2023-11-17T05:35:10.889 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:10.882+0000 7f450f0c1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4508060280 0x7f45080f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:35:10.889 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:10.882+0000 7f450f0c1700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45080f4ff0 con 0x7f4508060280 2023-11-17T05:35:10.889 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:10.882+0000 7f450ce5d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4508060280 0x7f45080f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:35:10.889 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:10.882+0000 7f450ce5d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4508060280 0x7f45080f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:46502/0 (socket says 172.21.15.160:46502) 2023-11-17T05:35:10.889 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:10.882+0000 7f450ce5d700 1 -- 172.21.15.160:0/215097006 learned_addr learned my addr 172.21.15.160:0/215097006 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:35:10.889 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:10.883+0000 7f450ce5d700 1 -- 172.21.15.160:0/215097006 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44fc005040 con 0x7f4508060280 2023-11-17T05:35:10.889 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:10.883+0000 7f450ce5d700 1 --2- 172.21.15.160:0/215097006 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4508060280 0x7f45080f6b70 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f44fc013510 tx=0x7f44fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:35:10.889 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:10.883+0000 7f45057fa700 1 -- 172.21.15.160:0/215097006 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44fc009320 con 0x7f4508060280 2023-11-17T05:35:10.889 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:10.883+0000 7f45057fa700 1 -- 172.21.15.160:0/215097006 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f44fc013f00 con 0x7f4508060280 2023-11-17T05:35:10.889 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:10.883+0000 7f450f0c1700 1 -- 172.21.15.160:0/215097006 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45080f5190 con 0x7f4508060280 2023-11-17T05:35:10.889 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:10.883+0000 7f45057fa700 1 -- 172.21.15.160:0/215097006 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44fc014dc0 con 0x7f4508060280 2023-11-17T05:35:10.890 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:10.883+0000 7f450f0c1700 1 -- 172.21.15.160:0/215097006 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45080f5620 con 0x7f4508060280 2023-11-17T05:35:10.890 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:10.884+0000 7f45057fa700 1 -- 172.21.15.160:0/215097006 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f44fc0095f0 con 0x7f4508060280 2023-11-17T05:35:10.890 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:10.884+0000 7f450f0c1700 1 -- 172.21.15.160:0/215097006 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4508059100 con 0x7f4508060280 2023-11-17T05:35:10.890 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:10.884+0000 7f45057fa700 1 --2- 172.21.15.160:0/215097006 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f44f802ff60 0x7f44f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:35:10.890 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:10.884+0000 7f45057fa700 1 -- 172.21.15.160:0/215097006 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f44fc041100 con 0x7f4508060280 2023-11-17T05:35:10.890 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:10.884+0000 7f4507fff700 1 --2- 172.21.15.160:0/215097006 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f44f802ff60 0x7f44f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:35:10.890 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:10.884+0000 7f4507fff700 1 --2- 172.21.15.160:0/215097006 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f44f802ff60 0x7f44f8032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f44f40096a0 tx=0x7f44f4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:35:10.890 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:10.889+0000 7f45057fa700 1 -- 172.21.15.160:0/215097006 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f44fc014420 con 0x7f4508060280 2023-11-17T05:35:11.097 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:11.096+0000 7f450f0c1700 1 -- 172.21.15.160:0/215097006 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f45080fbf50 con 0x7f4508060280 2023-11-17T05:35:11.097 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:11.097+0000 7f45057fa700 1 -- 172.21.15.160:0/215097006 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f44fc010230 con 0x7f4508060280 2023-11-17T05:35:11.098 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:35:11.098 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:35:11.099 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:11.098+0000 7f450f0c1700 1 -- 172.21.15.160:0/215097006 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f44f802ff60 msgr2=0x7f44f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:35:11.099 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:11.098+0000 7f450f0c1700 1 --2- 172.21.15.160:0/215097006 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f44f802ff60 0x7f44f8032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f44f40096a0 tx=0x7f44f4006b40).stop 2023-11-17T05:35:11.099 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:11.098+0000 7f450f0c1700 1 -- 172.21.15.160:0/215097006 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4508060280 msgr2=0x7f45080f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:35:11.099 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:11.098+0000 7f450f0c1700 1 --2- 172.21.15.160:0/215097006 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4508060280 0x7f45080f6b70 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f44fc013510 tx=0x7f44fc00b040).stop 2023-11-17T05:35:11.099 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:11.098+0000 7f450f0c1700 1 -- 172.21.15.160:0/215097006 shutdown_connections 2023-11-17T05:35:11.100 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:11.098+0000 7f450f0c1700 1 --2- 172.21.15.160:0/215097006 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f44f802ff60 0x7f44f8032420 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:35:11.100 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:11.098+0000 7f450f0c1700 1 --2- 172.21.15.160:0/215097006 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4508060280 0x7f45080f6b70 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:35:11.100 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:11.098+0000 7f450f0c1700 1 -- 172.21.15.160:0/215097006 >> 172.21.15.160:0/215097006 conn(0x7f45080f0b70 msgr2=0x7f45080f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:35:11.100 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:11.098+0000 7f450f0c1700 1 -- 172.21.15.160:0/215097006 shutdown_connections 2023-11-17T05:35:11.100 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:11.099+0000 7f450f0c1700 1 -- 172.21.15.160:0/215097006 wait complete. 2023-11-17T05:35:11.100 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:35:11.522 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:11 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:35:11.098065+0000 mon.smithi003 (mon.0) 258 : audit [DBG] from='client.? 172.21.15.160:0/215097006' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:35:12.503 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:35:12.503 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:35:12.520 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:12 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:35:11.484326+0000 mgr.smithi003.eddcuc (mgr.14164) 300 : cluster [DBG] pgmap v286: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:13.824 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:13.823+0000 7f1923fff700 1 -- 172.21.15.160:0/643710553 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1924060140 msgr2=0x7f19240f8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:35:13.824 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:13.823+0000 7f1923fff700 1 --2- 172.21.15.160:0/643710553 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1924060140 0x7f19240f8fc0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f1914009fb0 tx=0x7f1914009b30).stop 2023-11-17T05:35:13.824 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:13.823+0000 7f1923fff700 1 -- 172.21.15.160:0/643710553 shutdown_connections 2023-11-17T05:35:13.824 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:13.823+0000 7f1923fff700 1 --2- 172.21.15.160:0/643710553 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1924060140 0x7f19240f8fc0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:35:13.824 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:13.823+0000 7f1923fff700 1 -- 172.21.15.160:0/643710553 >> 172.21.15.160:0/643710553 conn(0x7f19240f09a0 msgr2=0x7f19240f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:35:13.824 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:13.823+0000 7f1923fff700 1 -- 172.21.15.160:0/643710553 shutdown_connections 2023-11-17T05:35:13.824 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:13.823+0000 7f1923fff700 1 -- 172.21.15.160:0/643710553 wait complete. 2023-11-17T05:35:13.826 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:13.823+0000 7f1923fff700 1 Processor -- start 2023-11-17T05:35:13.826 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:13.823+0000 7f1923fff700 1 -- start start 2023-11-17T05:35:13.826 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:13.824+0000 7f1923fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1924060140 0x7f1924106520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:35:13.826 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:13.824+0000 7f1923fff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1924106a10 con 0x7f1924060140 2023-11-17T05:35:13.827 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:13.824+0000 7f1922ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1924060140 0x7f1924106520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:35:13.827 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:13.824+0000 7f1922ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1924060140 0x7f1924106520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:46520/0 (socket says 172.21.15.160:46520) 2023-11-17T05:35:13.827 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:13.824+0000 7f1922ffd700 1 -- 172.21.15.160:0/3244140753 learned_addr learned my addr 172.21.15.160:0/3244140753 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:35:13.827 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:13.824+0000 7f1922ffd700 1 -- 172.21.15.160:0/3244140753 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1914005040 con 0x7f1924060140 2023-11-17T05:35:13.827 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:13.824+0000 7f1922ffd700 1 --2- 172.21.15.160:0/3244140753 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1924060140 0x7f1924106520 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f1914013510 tx=0x7f191400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:35:13.827 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:13.824+0000 7f190b7fe700 1 -- 172.21.15.160:0/3244140753 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1914009320 con 0x7f1924060140 2023-11-17T05:35:13.828 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:13.824+0000 7f1923fff700 1 -- 172.21.15.160:0/3244140753 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1924106c10 con 0x7f1924060140 2023-11-17T05:35:13.828 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:13.824+0000 7f190b7fe700 1 -- 172.21.15.160:0/3244140753 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1914013f00 con 0x7f1924060140 2023-11-17T05:35:13.828 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:13.824+0000 7f190b7fe700 1 -- 172.21.15.160:0/3244140753 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1914014d90 con 0x7f1924060140 2023-11-17T05:35:13.828 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:13.825+0000 7f1923fff700 1 -- 172.21.15.160:0/3244140753 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19241070a0 con 0x7f1924060140 2023-11-17T05:35:13.828 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:13.825+0000 7f190b7fe700 1 -- 172.21.15.160:0/3244140753 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f19140263a0 con 0x7f1924060140 2023-11-17T05:35:13.828 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:13.825+0000 7f190b7fe700 1 --2- 172.21.15.160:0/3244140753 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f190c02ff60 0x7f190c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:35:13.829 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:13.825+0000 7f1923fff700 1 -- 172.21.15.160:0/3244140753 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1924045ad0 con 0x7f1924060140 2023-11-17T05:35:13.829 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:13.826+0000 7f190b7fe700 1 -- 172.21.15.160:0/3244140753 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f19140413c0 con 0x7f1924060140 2023-11-17T05:35:13.829 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:13.826+0000 7f19227fc700 1 --2- 172.21.15.160:0/3244140753 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f190c02ff60 0x7f190c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:35:13.830 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:13.826+0000 7f19227fc700 1 --2- 172.21.15.160:0/3244140753 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f190c02ff60 0x7f190c032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f19180096a0 tx=0x7f1918006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:35:13.831 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:13.830+0000 7f190b7fe700 1 -- 172.21.15.160:0/3244140753 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f191401cb10 con 0x7f1924060140 2023-11-17T05:35:14.034 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:14.034+0000 7f1923fff700 1 -- 172.21.15.160:0/3244140753 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1924107f50 con 0x7f1924060140 2023-11-17T05:35:14.035 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:14.034+0000 7f190b7fe700 1 -- 172.21.15.160:0/3244140753 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f191402b030 con 0x7f1924060140 2023-11-17T05:35:14.035 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:35:14.035 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:35:14.036 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:14.036+0000 7f1923fff700 1 -- 172.21.15.160:0/3244140753 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f190c02ff60 msgr2=0x7f190c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:35:14.037 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:14.036+0000 7f1923fff700 1 --2- 172.21.15.160:0/3244140753 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f190c02ff60 0x7f190c032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f19180096a0 tx=0x7f1918006b40).stop 2023-11-17T05:35:14.037 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:14.036+0000 7f1923fff700 1 -- 172.21.15.160:0/3244140753 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1924060140 msgr2=0x7f1924106520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:35:14.037 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:14.036+0000 7f1923fff700 1 --2- 172.21.15.160:0/3244140753 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1924060140 0x7f1924106520 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f1914013510 tx=0x7f191400b040).stop 2023-11-17T05:35:14.037 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:14.036+0000 7f1923fff700 1 -- 172.21.15.160:0/3244140753 shutdown_connections 2023-11-17T05:35:14.037 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:14.036+0000 7f1923fff700 1 --2- 172.21.15.160:0/3244140753 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f190c02ff60 0x7f190c032420 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:35:14.037 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:14.036+0000 7f1923fff700 1 --2- 172.21.15.160:0/3244140753 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1924060140 0x7f1924106520 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:35:14.037 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:14.036+0000 7f1923fff700 1 -- 172.21.15.160:0/3244140753 >> 172.21.15.160:0/3244140753 conn(0x7f19240f09a0 msgr2=0x7f19240f14d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:35:14.037 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:14.036+0000 7f1923fff700 1 -- 172.21.15.160:0/3244140753 shutdown_connections 2023-11-17T05:35:14.037 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:14.036+0000 7f1923fff700 1 -- 172.21.15.160:0/3244140753 wait complete. 2023-11-17T05:35:14.038 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:35:15.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:14 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:35:13.484768+0000 mgr.smithi003.eddcuc (mgr.14164) 301 : cluster [DBG] pgmap v287: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:15.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:14 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:35:14.035430+0000 mon.smithi003 (mon.0) 259 : audit [DBG] from='client.? 172.21.15.160:0/3244140753' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:35:15.424 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:35:15.424 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:35:16.633 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:16.633+0000 7f2ff9b9b700 1 -- 172.21.15.160:0/3234588357 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ff40f62a0 msgr2=0x7f2ff40f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:35:16.634 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:16.633+0000 7f2ff9b9b700 1 --2- 172.21.15.160:0/3234588357 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ff40f62a0 0x7f2ff40f66c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f2fe8009fb0 tx=0x7f2fe8009b30).stop 2023-11-17T05:35:16.634 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:16.633+0000 7f2ff9b9b700 1 -- 172.21.15.160:0/3234588357 shutdown_connections 2023-11-17T05:35:16.634 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:16.633+0000 7f2ff9b9b700 1 --2- 172.21.15.160:0/3234588357 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ff40f62a0 0x7f2ff40f66c0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:35:16.634 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:16.633+0000 7f2ff9b9b700 1 -- 172.21.15.160:0/3234588357 >> 172.21.15.160:0/3234588357 conn(0x7f2ff40f1aa0 msgr2=0x7f2ff40f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:35:16.634 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:16.633+0000 7f2ff9b9b700 1 -- 172.21.15.160:0/3234588357 shutdown_connections 2023-11-17T05:35:16.634 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:16.633+0000 7f2ff9b9b700 1 -- 172.21.15.160:0/3234588357 wait complete. 2023-11-17T05:35:16.635 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:16.633+0000 7f2ff9b9b700 1 Processor -- start 2023-11-17T05:35:16.635 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:16.633+0000 7f2ff9b9b700 1 -- start start 2023-11-17T05:35:16.636 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:16.634+0000 7f2ff9b9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ff40f62a0 0x7f2ff41064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:35:16.636 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:16.634+0000 7f2ff9b9b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff41069a0 con 0x7f2ff40f62a0 2023-11-17T05:35:16.636 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:16.634+0000 7f2ff8b99700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ff40f62a0 0x7f2ff41064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:35:16.636 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:16.634+0000 7f2ff8b99700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ff40f62a0 0x7f2ff41064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:46546/0 (socket says 172.21.15.160:46546) 2023-11-17T05:35:16.636 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:16.634+0000 7f2ff8b99700 1 -- 172.21.15.160:0/842827623 learned_addr learned my addr 172.21.15.160:0/842827623 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:35:16.636 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:16.634+0000 7f2ff8b99700 1 -- 172.21.15.160:0/842827623 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe8005040 con 0x7f2ff40f62a0 2023-11-17T05:35:16.636 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:16.634+0000 7f2ff8b99700 1 --2- 172.21.15.160:0/842827623 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ff40f62a0 0x7f2ff41064b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f2fe8009830 tx=0x7f2fe800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:35:16.636 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:16.634+0000 7f2ff17fa700 1 -- 172.21.15.160:0/842827623 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fe8009320 con 0x7f2ff40f62a0 2023-11-17T05:35:16.636 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:16.634+0000 7f2ff17fa700 1 -- 172.21.15.160:0/842827623 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2fe80136c0 con 0x7f2ff40f62a0 2023-11-17T05:35:16.636 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:16.634+0000 7f2ff9b9b700 1 -- 172.21.15.160:0/842827623 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ff4106ba0 con 0x7f2ff40f62a0 2023-11-17T05:35:16.636 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:16.634+0000 7f2ff17fa700 1 -- 172.21.15.160:0/842827623 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fe801ee90 con 0x7f2ff40f62a0 2023-11-17T05:35:16.636 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:16.635+0000 7f2ff9b9b700 1 -- 172.21.15.160:0/842827623 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ff4107030 con 0x7f2ff40f62a0 2023-11-17T05:35:16.637 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:16.635+0000 7f2ff17fa700 1 -- 172.21.15.160:0/842827623 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2fe801e420 con 0x7f2ff40f62a0 2023-11-17T05:35:16.637 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:16.635+0000 7f2ff9b9b700 1 -- 172.21.15.160:0/842827623 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ff4045ad0 con 0x7f2ff40f62a0 2023-11-17T05:35:16.637 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:16.635+0000 7f2ff17fa700 1 --2- 172.21.15.160:0/842827623 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f2fe402ff10 0x7f2fe40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:35:16.637 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:16.635+0000 7f2ff17fa700 1 -- 172.21.15.160:0/842827623 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2fe8040f40 con 0x7f2ff40f62a0 2023-11-17T05:35:16.638 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:16.636+0000 7f2ff3fff700 1 --2- 172.21.15.160:0/842827623 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f2fe402ff10 0x7f2fe40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:35:16.638 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:16.636+0000 7f2ff3fff700 1 --2- 172.21.15.160:0/842827623 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f2fe402ff10 0x7f2fe40323d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f2fe00096a0 tx=0x7f2fe0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:35:16.641 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:16.641+0000 7f2ff17fa700 1 -- 172.21.15.160:0/842827623 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2fe806d2b0 con 0x7f2ff40f62a0 2023-11-17T05:35:16.845 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:16.845+0000 7f2ff9b9b700 1 -- 172.21.15.160:0/842827623 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2ff4107ee0 con 0x7f2ff40f62a0 2023-11-17T05:35:16.846 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:16.845+0000 7f2ff17fa700 1 -- 172.21.15.160:0/842827623 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f2fe80095f0 con 0x7f2ff40f62a0 2023-11-17T05:35:16.846 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:35:16.846 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:35:16.848 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:16.847+0000 7f2ff9b9b700 1 -- 172.21.15.160:0/842827623 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f2fe402ff10 msgr2=0x7f2fe40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:35:16.848 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:16.847+0000 7f2ff9b9b700 1 --2- 172.21.15.160:0/842827623 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f2fe402ff10 0x7f2fe40323d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f2fe00096a0 tx=0x7f2fe0006b40).stop 2023-11-17T05:35:16.848 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:16.847+0000 7f2ff9b9b700 1 -- 172.21.15.160:0/842827623 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ff40f62a0 msgr2=0x7f2ff41064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:35:16.848 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:16.847+0000 7f2ff9b9b700 1 --2- 172.21.15.160:0/842827623 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ff40f62a0 0x7f2ff41064b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f2fe8009830 tx=0x7f2fe800b040).stop 2023-11-17T05:35:16.848 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:16.847+0000 7f2ff9b9b700 1 -- 172.21.15.160:0/842827623 shutdown_connections 2023-11-17T05:35:16.848 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:16.847+0000 7f2ff9b9b700 1 --2- 172.21.15.160:0/842827623 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f2fe402ff10 0x7f2fe40323d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:35:16.848 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:16.847+0000 7f2ff9b9b700 1 --2- 172.21.15.160:0/842827623 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ff40f62a0 0x7f2ff41064b0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:35:16.849 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:16.847+0000 7f2ff9b9b700 1 -- 172.21.15.160:0/842827623 >> 172.21.15.160:0/842827623 conn(0x7f2ff40f1aa0 msgr2=0x7f2ff40f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:35:16.849 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:16.847+0000 7f2ff9b9b700 1 -- 172.21.15.160:0/842827623 shutdown_connections 2023-11-17T05:35:16.849 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:16.847+0000 7f2ff9b9b700 1 -- 172.21.15.160:0/842827623 wait complete. 2023-11-17T05:35:16.849 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:35:16.906 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:16 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:35:15.485186+0000 mgr.smithi003.eddcuc (mgr.14164) 302 : cluster [DBG] pgmap v288: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:18.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:17 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:35:16.846480+0000 mon.smithi003 (mon.0) 260 : audit [DBG] from='client.? 172.21.15.160:0/842827623' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:35:18.258 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:35:18.259 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:35:19.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:18 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:35:17.485591+0000 mgr.smithi003.eddcuc (mgr.14164) 303 : cluster [DBG] pgmap v289: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:19.462 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:19.460+0000 7f3069d25700 1 -- 172.21.15.160:0/1522542463 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30640f6510 msgr2=0x7f30640f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:35:19.462 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:19.460+0000 7f3069d25700 1 --2- 172.21.15.160:0/1522542463 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30640f6510 0x7f30640f6930 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f3058009fb0 tx=0x7f3058009b30).stop 2023-11-17T05:35:19.462 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:19.461+0000 7f3069d25700 1 -- 172.21.15.160:0/1522542463 shutdown_connections 2023-11-17T05:35:19.462 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:19.461+0000 7f3069d25700 1 --2- 172.21.15.160:0/1522542463 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30640f6510 0x7f30640f6930 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:35:19.462 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:19.461+0000 7f3069d25700 1 -- 172.21.15.160:0/1522542463 >> 172.21.15.160:0/1522542463 conn(0x7f30640f1c70 msgr2=0x7f30640f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:35:19.462 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:19.461+0000 7f3069d25700 1 -- 172.21.15.160:0/1522542463 shutdown_connections 2023-11-17T05:35:19.467 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:19.461+0000 7f3069d25700 1 -- 172.21.15.160:0/1522542463 wait complete. 2023-11-17T05:35:19.469 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:19.461+0000 7f3069d25700 1 Processor -- start 2023-11-17T05:35:19.469 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:19.462+0000 7f3069d25700 1 -- start start 2023-11-17T05:35:19.469 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:19.462+0000 7f3069d25700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30640f6510 0x7f3064106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:35:19.469 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:19.462+0000 7f3069d25700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3064106c20 con 0x7f30640f6510 2023-11-17T05:35:19.469 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:19.462+0000 7f30637fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30640f6510 0x7f3064106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:35:19.469 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:19.462+0000 7f30637fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30640f6510 0x7f3064106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:46564/0 (socket says 172.21.15.160:46564) 2023-11-17T05:35:19.469 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:19.462+0000 7f30637fe700 1 -- 172.21.15.160:0/1995561608 learned_addr learned my addr 172.21.15.160:0/1995561608 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:35:19.469 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:19.462+0000 7f30637fe700 1 -- 172.21.15.160:0/1995561608 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3058005040 con 0x7f30640f6510 2023-11-17T05:35:19.469 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:19.463+0000 7f30637fe700 1 --2- 172.21.15.160:0/1995561608 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30640f6510 0x7f3064106730 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f3058013510 tx=0x7f305800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:35:19.469 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:19.463+0000 7f304bfff700 1 -- 172.21.15.160:0/1995561608 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3058009320 con 0x7f30640f6510 2023-11-17T05:35:19.470 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:19.463+0000 7f3069d25700 1 -- 172.21.15.160:0/1995561608 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3064106e20 con 0x7f30640f6510 2023-11-17T05:35:19.470 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:19.463+0000 7f304bfff700 1 -- 172.21.15.160:0/1995561608 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3058013f00 con 0x7f30640f6510 2023-11-17T05:35:19.470 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:19.463+0000 7f304bfff700 1 -- 172.21.15.160:0/1995561608 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3058014d90 con 0x7f30640f6510 2023-11-17T05:35:19.470 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:19.463+0000 7f3069d25700 1 -- 172.21.15.160:0/1995561608 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30641072b0 con 0x7f30640f6510 2023-11-17T05:35:19.470 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:19.464+0000 7f304bfff700 1 -- 172.21.15.160:0/1995561608 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f30580263a0 con 0x7f30640f6510 2023-11-17T05:35:19.470 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:19.464+0000 7f3069d25700 1 -- 172.21.15.160:0/1995561608 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3064045ad0 con 0x7f30640f6510 2023-11-17T05:35:19.470 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:19.464+0000 7f304bfff700 1 --2- 172.21.15.160:0/1995561608 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f304c02ff60 0x7f304c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:35:19.470 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:19.464+0000 7f304bfff700 1 -- 172.21.15.160:0/1995561608 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f30580413a0 con 0x7f30640f6510 2023-11-17T05:35:19.470 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:19.464+0000 7f3062ffd700 1 --2- 172.21.15.160:0/1995561608 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f304c02ff60 0x7f304c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:35:19.470 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:19.465+0000 7f3062ffd700 1 --2- 172.21.15.160:0/1995561608 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f304c02ff60 0x7f304c032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f30540096a0 tx=0x7f3054006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:35:19.471 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:19.469+0000 7f304bfff700 1 -- 172.21.15.160:0/1995561608 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f305801cac0 con 0x7f30640f6510 2023-11-17T05:35:19.674 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:19.673+0000 7f3069d25700 1 -- 172.21.15.160:0/1995561608 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f30640fbf60 con 0x7f30640f6510 2023-11-17T05:35:19.675 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:19.673+0000 7f304bfff700 1 -- 172.21.15.160:0/1995561608 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f305801cac0 con 0x7f30640f6510 2023-11-17T05:35:19.675 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:35:19.675 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:35:19.679 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:19.675+0000 7f3069d25700 1 -- 172.21.15.160:0/1995561608 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f304c02ff60 msgr2=0x7f304c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:35:19.679 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:19.675+0000 7f3069d25700 1 --2- 172.21.15.160:0/1995561608 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f304c02ff60 0x7f304c032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f30540096a0 tx=0x7f3054006b40).stop 2023-11-17T05:35:19.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:19.675+0000 7f3069d25700 1 -- 172.21.15.160:0/1995561608 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30640f6510 msgr2=0x7f3064106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:35:19.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:19.675+0000 7f3069d25700 1 --2- 172.21.15.160:0/1995561608 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30640f6510 0x7f3064106730 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f3058013510 tx=0x7f305800b040).stop 2023-11-17T05:35:19.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:19.675+0000 7f3069d25700 1 -- 172.21.15.160:0/1995561608 shutdown_connections 2023-11-17T05:35:19.681 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:19.675+0000 7f3069d25700 1 --2- 172.21.15.160:0/1995561608 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f304c02ff60 0x7f304c032420 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:35:19.681 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:19.675+0000 7f3069d25700 1 --2- 172.21.15.160:0/1995561608 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f30640f6510 0x7f3064106730 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:35:19.681 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:19.675+0000 7f3069d25700 1 -- 172.21.15.160:0/1995561608 >> 172.21.15.160:0/1995561608 conn(0x7f30640f1c70 msgr2=0x7f30640f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:35:19.682 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:19.675+0000 7f3069d25700 1 -- 172.21.15.160:0/1995561608 shutdown_connections 2023-11-17T05:35:19.682 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:19.675+0000 7f3069d25700 1 -- 172.21.15.160:0/1995561608 wait complete. 2023-11-17T05:35:19.682 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:35:20.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:20 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:35:19.485885+0000 mgr.smithi003.eddcuc (mgr.14164) 304 : cluster [DBG] pgmap v290: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:20.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:20 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:35:19.674689+0000 mon.smithi003 (mon.0) 261 : audit [DBG] from='client.? 172.21.15.160:0/1995561608' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:35:21.071 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:35:21.071 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:35:22.258 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:22.256+0000 7fda5cc45700 1 -- 172.21.15.160:0/558286908 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda580f5410 msgr2=0x7fda580f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:35:22.258 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:22.256+0000 7fda5cc45700 1 --2- 172.21.15.160:0/558286908 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda580f5410 0x7fda580f5830 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7fda4c009fb0 tx=0x7fda4c009b30).stop 2023-11-17T05:35:22.258 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:22.256+0000 7fda5cc45700 1 -- 172.21.15.160:0/558286908 shutdown_connections 2023-11-17T05:35:22.258 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:22.256+0000 7fda5cc45700 1 --2- 172.21.15.160:0/558286908 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda580f5410 0x7fda580f5830 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:35:22.258 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:22.256+0000 7fda5cc45700 1 -- 172.21.15.160:0/558286908 >> 172.21.15.160:0/558286908 conn(0x7fda580f0b70 msgr2=0x7fda580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:35:22.258 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:22.257+0000 7fda5cc45700 1 -- 172.21.15.160:0/558286908 shutdown_connections 2023-11-17T05:35:22.258 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:22.257+0000 7fda5cc45700 1 -- 172.21.15.160:0/558286908 wait complete. 2023-11-17T05:35:22.259 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:22.257+0000 7fda5cc45700 1 Processor -- start 2023-11-17T05:35:22.259 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:22.257+0000 7fda5cc45700 1 -- start start 2023-11-17T05:35:22.259 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:22.257+0000 7fda5cc45700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda580f5410 0x7fda58109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:35:22.259 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:22.257+0000 7fda5cc45700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda58060240 con 0x7fda580f5410 2023-11-17T05:35:22.260 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:22.258+0000 7fda5659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda580f5410 0x7fda58109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:35:22.260 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:22.258+0000 7fda5659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda580f5410 0x7fda58109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:57190/0 (socket says 172.21.15.160:57190) 2023-11-17T05:35:22.260 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:22.258+0000 7fda5659c700 1 -- 172.21.15.160:0/421252488 learned_addr learned my addr 172.21.15.160:0/421252488 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:35:22.260 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:22.258+0000 7fda5659c700 1 -- 172.21.15.160:0/421252488 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda4c005040 con 0x7fda580f5410 2023-11-17T05:35:22.261 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:22.258+0000 7fda5659c700 1 --2- 172.21.15.160:0/421252488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda580f5410 0x7fda58109430 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7fda4c013510 tx=0x7fda4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:35:22.261 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:22.258+0000 7fda46ffd700 1 -- 172.21.15.160:0/421252488 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda4c009320 con 0x7fda580f5410 2023-11-17T05:35:22.261 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:22.258+0000 7fda46ffd700 1 -- 172.21.15.160:0/421252488 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fda4c013f00 con 0x7fda580f5410 2023-11-17T05:35:22.261 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:22.258+0000 7fda5cc45700 1 -- 172.21.15.160:0/421252488 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda58060440 con 0x7fda580f5410 2023-11-17T05:35:22.261 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:22.258+0000 7fda46ffd700 1 -- 172.21.15.160:0/421252488 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda4c014d90 con 0x7fda580f5410 2023-11-17T05:35:22.261 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:22.258+0000 7fda5cc45700 1 -- 172.21.15.160:0/421252488 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda58109ae0 con 0x7fda580f5410 2023-11-17T05:35:22.261 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:22.259+0000 7fda5cc45700 1 -- 172.21.15.160:0/421252488 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda58045ad0 con 0x7fda580f5410 2023-11-17T05:35:22.261 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:22.259+0000 7fda46ffd700 1 -- 172.21.15.160:0/421252488 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fda4c0095f0 con 0x7fda580f5410 2023-11-17T05:35:22.261 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:22.259+0000 7fda46ffd700 1 --2- 172.21.15.160:0/421252488 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fda4002ff60 0x7fda40032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:35:22.261 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:22.259+0000 7fda46ffd700 1 -- 172.21.15.160:0/421252488 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fda4c0410c0 con 0x7fda580f5410 2023-11-17T05:35:22.261 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:22.260+0000 7fda55d9b700 1 --2- 172.21.15.160:0/421252488 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fda4002ff60 0x7fda40032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:35:22.262 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:22.260+0000 7fda55d9b700 1 --2- 172.21.15.160:0/421252488 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fda4002ff60 0x7fda40032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fda480096a0 tx=0x7fda48006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:35:22.266 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:22.264+0000 7fda46ffd700 1 -- 172.21.15.160:0/421252488 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fda4c01d7f0 con 0x7fda580f5410 2023-11-17T05:35:22.471 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:22.470+0000 7fda5cc45700 1 -- 172.21.15.160:0/421252488 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fda580fae60 con 0x7fda580f5410 2023-11-17T05:35:22.472 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:22.471+0000 7fda46ffd700 1 -- 172.21.15.160:0/421252488 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fda4c010230 con 0x7fda580f5410 2023-11-17T05:35:22.472 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:35:22.472 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:35:22.474 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:22.473+0000 7fda5cc45700 1 -- 172.21.15.160:0/421252488 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fda4002ff60 msgr2=0x7fda40032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:35:22.474 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:22.473+0000 7fda5cc45700 1 --2- 172.21.15.160:0/421252488 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fda4002ff60 0x7fda40032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fda480096a0 tx=0x7fda48006b40).stop 2023-11-17T05:35:22.474 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:22.473+0000 7fda5cc45700 1 -- 172.21.15.160:0/421252488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda580f5410 msgr2=0x7fda58109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:35:22.474 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:22.473+0000 7fda5cc45700 1 --2- 172.21.15.160:0/421252488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda580f5410 0x7fda58109430 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7fda4c013510 tx=0x7fda4c00b040).stop 2023-11-17T05:35:22.474 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:22.473+0000 7fda5cc45700 1 -- 172.21.15.160:0/421252488 shutdown_connections 2023-11-17T05:35:22.475 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:22.473+0000 7fda5cc45700 1 --2- 172.21.15.160:0/421252488 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fda4002ff60 0x7fda40032420 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:35:22.475 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:22.473+0000 7fda5cc45700 1 --2- 172.21.15.160:0/421252488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda580f5410 0x7fda58109430 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:35:22.475 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:22.473+0000 7fda5cc45700 1 -- 172.21.15.160:0/421252488 >> 172.21.15.160:0/421252488 conn(0x7fda580f0b70 msgr2=0x7fda580f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:35:22.475 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:22.473+0000 7fda5cc45700 1 -- 172.21.15.160:0/421252488 shutdown_connections 2023-11-17T05:35:22.475 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:22.473+0000 7fda5cc45700 1 -- 172.21.15.160:0/421252488 wait complete. 2023-11-17T05:35:22.475 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:35:22.958 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:22 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:35:21.486142+0000 mgr.smithi003.eddcuc (mgr.14164) 305 : cluster [DBG] pgmap v291: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:22.959 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:22 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:35:22.472132+0000 mon.smithi003 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.160:0/421252488' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:35:23.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:23 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:35:22.957465+0000 mon.smithi003 (mon.0) 263 : audit [DBG] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-17T05:35:23.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:23 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:35:23.184270+0000 mon.smithi003 (mon.0) 264 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' 2023-11-17T05:35:23.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:23 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:35:23.423303+0000 mon.smithi003 (mon.0) 265 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' 2023-11-17T05:35:23.858 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:35:23.858 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:35:25.021 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cephadm 2023-11-17T05:35:22.956884+0000 mgr.smithi003.eddcuc (mgr.14164) 306 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi003 ... 2023-11-17T05:35:25.021 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2023-11-17T05:35:25.021 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2023-11-17T05:35:25.021 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: stat: stderr time="2023-11-17T05:33:30Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-17T05:35:25.021 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: stat: stderr time="2023-11-17T05:34:08Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-17T05:35:25.021 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: stat: stderr time="2023-11-17T05:34:45Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-17T05:35:25.021 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-17T05:35:25.021 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: Traceback (most recent call last): 2023-11-17T05:35:25.021 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: File "/var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-17T05:35:25.021 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: main() 2023-11-17T05:35:25.022 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: File "/var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-17T05:35:25.022 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: r = ctx.func(ctx) 2023-11-17T05:35:25.022 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: File "/var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-17T05:35:25.022 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: return func(ctx) 2023-11-17T05:35:25.022 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: File "/var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-11-17T05:35:25.022 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-11-17T05:35:25.022 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: File "/var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2023-11-17T05:35:25.022 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2023-11-17T05:35:25.022 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: File "/var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-11-17T05:35:25.022 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: raise RuntimeError('uid/gid not found') 2023-11-17T05:35:25.022 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: RuntimeError: uid/gid not found 2023-11-17T05:35:25.022 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: Traceback (most recent call last): 2023-11-17T05:35:25.023 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-11-17T05:35:25.023 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: yield (conn, connr) 2023-11-17T05:35:25.023 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-11-17T05:35:25.023 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: code, '\n'.join(err))) 2023-11-17T05:35:25.023 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi003 ... 2023-11-17T05:35:25.023 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi003 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2023-11-17T05:35:25.023 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2023-11-17T05:35:25.023 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: stat: stderr time="2023-11-17T05:33:30Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-17T05:35:25.023 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: stat: stderr time="2023-11-17T05:34:08Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-17T05:35:25.023 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: stat: stderr time="2023-11-17T05:34:45Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-17T05:35:25.023 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-17T05:35:25.023 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: Traceback (most recent call last): 2023-11-17T05:35:25.024 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: File "/var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-17T05:35:25.024 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: main() 2023-11-17T05:35:25.024 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: File "/var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-17T05:35:25.024 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: r = ctx.func(ctx) 2023-11-17T05:35:25.024 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: File "/var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-17T05:35:25.024 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: return func(ctx) 2023-11-17T05:35:25.024 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: File "/var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-11-17T05:35:25.024 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-11-17T05:35:25.024 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: File "/var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2023-11-17T05:35:25.024 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2023-11-17T05:35:25.024 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: File "/var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-11-17T05:35:25.024 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: raise RuntimeError('uid/gid not found') 2023-11-17T05:35:25.024 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: RuntimeError: uid/gid not found 2023-11-17T05:35:25.025 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:35:23.486400+0000 mgr.smithi003.eddcuc (mgr.14164) 307 : cluster [DBG] pgmap v292: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:25.025 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:35:24.263456+0000 mon.smithi003 (mon.0) 266 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' 2023-11-17T05:35:25.329 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:25.327+0000 7f25c208b700 1 -- 172.21.15.160:0/1977408718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25bc060220 msgr2=0x7f25bc0fb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:35:25.330 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:25.327+0000 7f25c208b700 1 --2- 172.21.15.160:0/1977408718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25bc060220 0x7f25bc0fb550 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f25ac009fb0 tx=0x7f25ac009b30).stop 2023-11-17T05:35:25.330 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:25.327+0000 7f25c208b700 1 -- 172.21.15.160:0/1977408718 shutdown_connections 2023-11-17T05:35:25.330 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:25.327+0000 7f25c208b700 1 --2- 172.21.15.160:0/1977408718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25bc060220 0x7f25bc0fb550 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:35:25.330 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:25.327+0000 7f25c208b700 1 -- 172.21.15.160:0/1977408718 >> 172.21.15.160:0/1977408718 conn(0x7f25bc0f0b70 msgr2=0x7f25bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:35:25.330 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:25.327+0000 7f25c208b700 1 -- 172.21.15.160:0/1977408718 shutdown_connections 2023-11-17T05:35:25.330 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:25.327+0000 7f25c208b700 1 -- 172.21.15.160:0/1977408718 wait complete. 2023-11-17T05:35:25.330 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:25.327+0000 7f25c208b700 1 Processor -- start 2023-11-17T05:35:25.330 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:25.328+0000 7f25c208b700 1 -- start start 2023-11-17T05:35:25.331 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:25.328+0000 7f25c208b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25bc060220 0x7f25bc1088b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:35:25.331 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:25.328+0000 7f25c208b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25bc108da0 con 0x7f25bc060220 2023-11-17T05:35:25.331 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:25.328+0000 7f25bb7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25bc060220 0x7f25bc1088b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:35:25.331 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:25.328+0000 7f25bb7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25bc060220 0x7f25bc1088b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:57214/0 (socket says 172.21.15.160:57214) 2023-11-17T05:35:25.331 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:25.328+0000 7f25bb7fe700 1 -- 172.21.15.160:0/352185292 learned_addr learned my addr 172.21.15.160:0/352185292 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:35:25.331 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:25.328+0000 7f25bb7fe700 1 -- 172.21.15.160:0/352185292 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ac005040 con 0x7f25bc060220 2023-11-17T05:35:25.333 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:25.328+0000 7f25bb7fe700 1 --2- 172.21.15.160:0/352185292 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25bc060220 0x7f25bc1088b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f25ac013510 tx=0x7f25ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:35:25.333 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:25.329+0000 7f25a3fff700 1 -- 172.21.15.160:0/352185292 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25ac009320 con 0x7f25bc060220 2023-11-17T05:35:25.333 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:25.329+0000 7f25c208b700 1 -- 172.21.15.160:0/352185292 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25bc108fa0 con 0x7f25bc060220 2023-11-17T05:35:25.333 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:25.329+0000 7f25a3fff700 1 -- 172.21.15.160:0/352185292 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f25ac013f00 con 0x7f25bc060220 2023-11-17T05:35:25.333 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:25.329+0000 7f25a3fff700 1 -- 172.21.15.160:0/352185292 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25ac014dc0 con 0x7f25bc060220 2023-11-17T05:35:25.333 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:25.329+0000 7f25c208b700 1 -- 172.21.15.160:0/352185292 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25bc109430 con 0x7f25bc060220 2023-11-17T05:35:25.335 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:25.329+0000 7f25c208b700 1 -- 172.21.15.160:0/352185292 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25bc045ad0 con 0x7f25bc060220 2023-11-17T05:35:25.335 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:25.329+0000 7f25a3fff700 1 -- 172.21.15.160:0/352185292 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f25ac0095f0 con 0x7f25bc060220 2023-11-17T05:35:25.335 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:25.330+0000 7f25a3fff700 1 --2- 172.21.15.160:0/352185292 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f25a402ff60 0x7f25a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:35:25.335 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:25.330+0000 7f25a3fff700 1 -- 172.21.15.160:0/352185292 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f25ac01b920 con 0x7f25bc060220 2023-11-17T05:35:25.335 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:25.330+0000 7f25baffd700 1 --2- 172.21.15.160:0/352185292 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f25a402ff60 0x7f25a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:35:25.335 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:25.330+0000 7f25baffd700 1 --2- 172.21.15.160:0/352185292 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f25a402ff60 0x7f25a4032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f25b00096a0 tx=0x7f25b0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:35:25.336 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:25.334+0000 7f25a3fff700 1 -- 172.21.15.160:0/352185292 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f25ac014420 con 0x7f25bc060220 2023-11-17T05:35:25.551 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:25.550+0000 7f25c208b700 1 -- 172.21.15.160:0/352185292 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f25bc0fe150 con 0x7f25bc060220 2023-11-17T05:35:25.552 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:25.551+0000 7f25a3fff700 1 -- 172.21.15.160:0/352185292 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f25ac01b740 con 0x7f25bc060220 2023-11-17T05:35:25.553 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:35:25.553 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:35:25.554 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:25.553+0000 7f25c208b700 1 -- 172.21.15.160:0/352185292 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f25a402ff60 msgr2=0x7f25a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:35:25.554 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:25.553+0000 7f25c208b700 1 --2- 172.21.15.160:0/352185292 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f25a402ff60 0x7f25a4032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f25b00096a0 tx=0x7f25b0006b40).stop 2023-11-17T05:35:25.554 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:25.553+0000 7f25c208b700 1 -- 172.21.15.160:0/352185292 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25bc060220 msgr2=0x7f25bc1088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:35:25.555 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:25.553+0000 7f25c208b700 1 --2- 172.21.15.160:0/352185292 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25bc060220 0x7f25bc1088b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f25ac013510 tx=0x7f25ac00b040).stop 2023-11-17T05:35:25.555 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:25.553+0000 7f25c208b700 1 -- 172.21.15.160:0/352185292 shutdown_connections 2023-11-17T05:35:25.555 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:25.553+0000 7f25c208b700 1 --2- 172.21.15.160:0/352185292 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f25a402ff60 0x7f25a4032420 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:35:25.555 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:25.553+0000 7f25c208b700 1 --2- 172.21.15.160:0/352185292 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f25bc060220 0x7f25bc1088b0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:35:25.555 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:25.553+0000 7f25c208b700 1 -- 172.21.15.160:0/352185292 >> 172.21.15.160:0/352185292 conn(0x7f25bc0f0b70 msgr2=0x7f25bc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:35:25.555 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:25.553+0000 7f25c208b700 1 -- 172.21.15.160:0/352185292 shutdown_connections 2023-11-17T05:35:25.555 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:25.553+0000 7f25c208b700 1 -- 172.21.15.160:0/352185292 wait complete. 2023-11-17T05:35:25.555 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:35:27.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:26 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:35:25.486832+0000 mgr.smithi003.eddcuc (mgr.14164) 308 : cluster [DBG] pgmap v293: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:27.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:26 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:35:25.552298+0000 mon.smithi003 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.160:0/352185292' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:35:27.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:26 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:35:26.064651+0000 mon.smithi003 (mon.0) 268 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' 2023-11-17T05:35:27.021 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:26 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:35:26.353795+0000 mon.smithi003 (mon.0) 269 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' 2023-11-17T05:35:27.021 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:26 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:35:26.356610+0000 mon.smithi003 (mon.0) 270 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' 2023-11-17T05:35:27.080 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:35:27.081 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:35:28.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:27 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:35:26.354475+0000 mgr.smithi003.eddcuc (mgr.14164) 309 : cluster [DBG] pgmap v294: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:28.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:27 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cephadm 2023-11-17T05:35:26.358924+0000 mgr.smithi003.eddcuc (mgr.14164) 310 : cephadm [INF] Deploying daemon alertmanager.smithi160 on smithi160 2023-11-17T05:35:28.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:27 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:35:27.352464+0000 mon.smithi003 (mon.0) 271 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-11-17T05:35:28.314 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:28.312+0000 7fc6c0075700 1 -- 172.21.15.160:0/4257986794 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6b80f5410 msgr2=0x7fc6b80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:35:28.314 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:28.312+0000 7fc6c0075700 1 --2- 172.21.15.160:0/4257986794 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6b80f5410 0x7fc6b80f5830 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7fc6a8009fb0 tx=0x7fc6a8009b30).stop 2023-11-17T05:35:28.315 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:28.312+0000 7fc6c0075700 1 -- 172.21.15.160:0/4257986794 shutdown_connections 2023-11-17T05:35:28.315 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:28.312+0000 7fc6c0075700 1 --2- 172.21.15.160:0/4257986794 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6b80f5410 0x7fc6b80f5830 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:35:28.315 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:28.312+0000 7fc6c0075700 1 -- 172.21.15.160:0/4257986794 >> 172.21.15.160:0/4257986794 conn(0x7fc6b80f0b70 msgr2=0x7fc6b80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:35:28.315 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:28.313+0000 7fc6c0075700 1 -- 172.21.15.160:0/4257986794 shutdown_connections 2023-11-17T05:35:28.315 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:28.313+0000 7fc6c0075700 1 -- 172.21.15.160:0/4257986794 wait complete. 2023-11-17T05:35:28.315 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:28.313+0000 7fc6c0075700 1 Processor -- start 2023-11-17T05:35:28.315 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:28.313+0000 7fc6c0075700 1 -- start start 2023-11-17T05:35:28.315 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:28.313+0000 7fc6c0075700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6b8106810 0x7fc6b8106c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:35:28.315 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:28.313+0000 7fc6c0075700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b8107120 con 0x7fc6b8106810 2023-11-17T05:35:28.317 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:28.314+0000 7fc6bde11700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6b8106810 0x7fc6b8106c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:35:28.317 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:28.314+0000 7fc6bde11700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6b8106810 0x7fc6b8106c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:57230/0 (socket says 172.21.15.160:57230) 2023-11-17T05:35:28.317 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:28.314+0000 7fc6bde11700 1 -- 172.21.15.160:0/2245860588 learned_addr learned my addr 172.21.15.160:0/2245860588 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:35:28.317 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:28.314+0000 7fc6bde11700 1 -- 172.21.15.160:0/2245860588 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6a8005040 con 0x7fc6b8106810 2023-11-17T05:35:28.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:28.314+0000 7fc6bde11700 1 --2- 172.21.15.160:0/2245860588 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6b8106810 0x7fc6b8106c30 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7fc6a8009f80 tx=0x7fc6a8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:35:28.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:28.314+0000 7fc6ae7fc700 1 -- 172.21.15.160:0/2245860588 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc6a80136c0 con 0x7fc6b8106810 2023-11-17T05:35:28.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:28.314+0000 7fc6c0075700 1 -- 172.21.15.160:0/2245860588 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6b8107320 con 0x7fc6b8106810 2023-11-17T05:35:28.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:28.314+0000 7fc6ae7fc700 1 -- 172.21.15.160:0/2245860588 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc6a80095f0 con 0x7fc6b8106810 2023-11-17T05:35:28.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:28.314+0000 7fc6ae7fc700 1 -- 172.21.15.160:0/2245860588 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc6a8014420 con 0x7fc6b8106810 2023-11-17T05:35:28.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:28.315+0000 7fc6c0075700 1 -- 172.21.15.160:0/2245860588 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6b8107e70 con 0x7fc6b8106810 2023-11-17T05:35:28.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:28.315+0000 7fc6c0075700 1 -- 172.21.15.160:0/2245860588 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6b8059100 con 0x7fc6b8106810 2023-11-17T05:35:28.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:28.315+0000 7fc6ae7fc700 1 -- 172.21.15.160:0/2245860588 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc6a8025060 con 0x7fc6b8106810 2023-11-17T05:35:28.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:28.315+0000 7fc6ae7fc700 1 --2- 172.21.15.160:0/2245860588 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc6a402ff60 0x7fc6a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:35:28.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:28.316+0000 7fc6ae7fc700 1 -- 172.21.15.160:0/2245860588 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc6a8040d80 con 0x7fc6b8106810 2023-11-17T05:35:28.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:28.316+0000 7fc6bd610700 1 --2- 172.21.15.160:0/2245860588 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc6a402ff60 0x7fc6a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:35:28.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:28.316+0000 7fc6bd610700 1 --2- 172.21.15.160:0/2245860588 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc6a402ff60 0x7fc6a4032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fc6b40096a0 tx=0x7fc6b4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:35:28.321 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:28.320+0000 7fc6ae7fc700 1 -- 172.21.15.160:0/2245860588 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc6a801cb60 con 0x7fc6b8106810 2023-11-17T05:35:28.526 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:28.525+0000 7fc6c0075700 1 -- 172.21.15.160:0/2245860588 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc6b8108630 con 0x7fc6b8106810 2023-11-17T05:35:28.527 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:28.526+0000 7fc6ae7fc700 1 -- 172.21.15.160:0/2245860588 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fc6a8042070 con 0x7fc6b8106810 2023-11-17T05:35:28.528 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:35:28.528 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:35:28.530 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:28.528+0000 7fc6c0075700 1 -- 172.21.15.160:0/2245860588 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc6a402ff60 msgr2=0x7fc6a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:35:28.530 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:28.528+0000 7fc6c0075700 1 --2- 172.21.15.160:0/2245860588 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc6a402ff60 0x7fc6a4032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fc6b40096a0 tx=0x7fc6b4006b40).stop 2023-11-17T05:35:28.530 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:28.528+0000 7fc6c0075700 1 -- 172.21.15.160:0/2245860588 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6b8106810 msgr2=0x7fc6b8106c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:35:28.530 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:28.528+0000 7fc6c0075700 1 --2- 172.21.15.160:0/2245860588 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6b8106810 0x7fc6b8106c30 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7fc6a8009f80 tx=0x7fc6a8009b30).stop 2023-11-17T05:35:28.530 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:28.528+0000 7fc6c0075700 1 -- 172.21.15.160:0/2245860588 shutdown_connections 2023-11-17T05:35:28.530 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:28.528+0000 7fc6c0075700 1 --2- 172.21.15.160:0/2245860588 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc6a402ff60 0x7fc6a4032420 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:35:28.530 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:28.528+0000 7fc6c0075700 1 --2- 172.21.15.160:0/2245860588 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6b8106810 0x7fc6b8106c30 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:35:28.530 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:28.528+0000 7fc6c0075700 1 -- 172.21.15.160:0/2245860588 >> 172.21.15.160:0/2245860588 conn(0x7fc6b80f0b70 msgr2=0x7fc6b805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:35:28.531 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:28.528+0000 7fc6c0075700 1 -- 172.21.15.160:0/2245860588 shutdown_connections 2023-11-17T05:35:28.531 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:28.528+0000 7fc6c0075700 1 -- 172.21.15.160:0/2245860588 wait complete. 2023-11-17T05:35:28.531 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:35:29.021 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:35:27.556202+0000 mon.smithi003 (mon.0) 272 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.eddcuc/mirror_snapshot_schedule"}]: dispatch 2023-11-17T05:35:29.021 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:35:27.559853+0000 mon.smithi003 (mon.0) 273 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.eddcuc/trash_purge_schedule"}]: dispatch 2023-11-17T05:35:29.022 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:35:28.527174+0000 mon.smithi003 (mon.0) 274 : audit [DBG] from='client.? 172.21.15.160:0/2245860588' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:35:29.950 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:35:29.950 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:35:30.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:29 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:35:28.354859+0000 mgr.smithi003.eddcuc (mgr.14164) 311 : cluster [DBG] pgmap v295: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:31.192 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:31.190+0000 7fa03c68e700 1 -- 172.21.15.160:0/1391441379 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0340f5410 msgr2=0x7fa0340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:35:31.192 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:31.190+0000 7fa03c68e700 1 --2- 172.21.15.160:0/1391441379 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0340f5410 0x7fa0340f5830 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7fa028009fb0 tx=0x7fa028009b30).stop 2023-11-17T05:35:31.193 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:31.191+0000 7fa03c68e700 1 -- 172.21.15.160:0/1391441379 shutdown_connections 2023-11-17T05:35:31.193 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:31.191+0000 7fa03c68e700 1 --2- 172.21.15.160:0/1391441379 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0340f5410 0x7fa0340f5830 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:35:31.193 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:31.191+0000 7fa03c68e700 1 -- 172.21.15.160:0/1391441379 >> 172.21.15.160:0/1391441379 conn(0x7fa0340f0b70 msgr2=0x7fa0340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:35:31.193 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:31.191+0000 7fa03c68e700 1 -- 172.21.15.160:0/1391441379 shutdown_connections 2023-11-17T05:35:31.193 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:31.191+0000 7fa03c68e700 1 -- 172.21.15.160:0/1391441379 wait complete. 2023-11-17T05:35:31.193 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:31.191+0000 7fa03c68e700 1 Processor -- start 2023-11-17T05:35:31.193 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:31.191+0000 7fa03c68e700 1 -- start start 2023-11-17T05:35:31.193 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:31.191+0000 7fa03c68e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0340f5410 0x7fa034109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:35:31.193 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:31.191+0000 7fa03c68e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa034060240 con 0x7fa0340f5410 2023-11-17T05:35:31.193 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:31.192+0000 7fa03a42a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0340f5410 0x7fa034109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:35:31.193 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:31.192+0000 7fa03a42a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0340f5410 0x7fa034109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:46760/0 (socket says 172.21.15.160:46760) 2023-11-17T05:35:31.193 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:31.192+0000 7fa03a42a700 1 -- 172.21.15.160:0/4163419146 learned_addr learned my addr 172.21.15.160:0/4163419146 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:35:31.195 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:31.192+0000 7fa03a42a700 1 -- 172.21.15.160:0/4163419146 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa028005040 con 0x7fa0340f5410 2023-11-17T05:35:31.195 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:31.192+0000 7fa03a42a700 1 --2- 172.21.15.160:0/4163419146 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0340f5410 0x7fa034109430 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7fa028013510 tx=0x7fa02800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:35:31.195 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:31.192+0000 7fa026ffd700 1 -- 172.21.15.160:0/4163419146 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa028009320 con 0x7fa0340f5410 2023-11-17T05:35:31.195 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:31.192+0000 7fa03c68e700 1 -- 172.21.15.160:0/4163419146 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa034060440 con 0x7fa0340f5410 2023-11-17T05:35:31.195 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:31.192+0000 7fa03c68e700 1 -- 172.21.15.160:0/4163419146 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa034109ae0 con 0x7fa0340f5410 2023-11-17T05:35:31.195 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:31.192+0000 7fa026ffd700 1 -- 172.21.15.160:0/4163419146 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa028013f00 con 0x7fa0340f5410 2023-11-17T05:35:31.195 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:31.192+0000 7fa026ffd700 1 -- 172.21.15.160:0/4163419146 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa028014d60 con 0x7fa0340f5410 2023-11-17T05:35:31.195 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:31.193+0000 7fa026ffd700 1 -- 172.21.15.160:0/4163419146 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa0280263a0 con 0x7fa0340f5410 2023-11-17T05:35:31.196 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:31.193+0000 7fa026ffd700 1 --2- 172.21.15.160:0/4163419146 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa02002ff60 0x7fa020032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:35:31.196 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:31.193+0000 7fa026ffd700 1 -- 172.21.15.160:0/4163419146 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa028040fe0 con 0x7fa0340f5410 2023-11-17T05:35:31.196 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:31.193+0000 7fa03c68e700 1 -- 172.21.15.160:0/4163419146 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa01c004fa0 con 0x7fa0340f5410 2023-11-17T05:35:31.196 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:31.194+0000 7fa039c29700 1 --2- 172.21.15.160:0/4163419146 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa02002ff60 0x7fa020032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:35:31.196 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:31.194+0000 7fa039c29700 1 --2- 172.21.15.160:0/4163419146 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa02002ff60 0x7fa020032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fa0300096a0 tx=0x7fa030006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:35:31.199 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:31.199+0000 7fa026ffd700 1 -- 172.21.15.160:0/4163419146 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa02801d380 con 0x7fa0340f5410 2023-11-17T05:35:31.408 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:31.407+0000 7fa03c68e700 1 -- 172.21.15.160:0/4163419146 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa01c005670 con 0x7fa0340f5410 2023-11-17T05:35:31.409 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:31.408+0000 7fa026ffd700 1 -- 172.21.15.160:0/4163419146 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fa02802b020 con 0x7fa0340f5410 2023-11-17T05:35:31.409 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:35:31.409 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:35:31.411 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:31.410+0000 7fa03c68e700 1 -- 172.21.15.160:0/4163419146 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa02002ff60 msgr2=0x7fa020032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:35:31.411 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:31.410+0000 7fa03c68e700 1 --2- 172.21.15.160:0/4163419146 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa02002ff60 0x7fa020032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fa0300096a0 tx=0x7fa030006b40).stop 2023-11-17T05:35:31.411 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:31.410+0000 7fa03c68e700 1 -- 172.21.15.160:0/4163419146 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0340f5410 msgr2=0x7fa034109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:35:31.411 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:31.410+0000 7fa03c68e700 1 --2- 172.21.15.160:0/4163419146 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0340f5410 0x7fa034109430 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7fa028013510 tx=0x7fa02800b040).stop 2023-11-17T05:35:31.411 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:31.410+0000 7fa03c68e700 1 -- 172.21.15.160:0/4163419146 shutdown_connections 2023-11-17T05:35:31.411 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:31.410+0000 7fa03c68e700 1 --2- 172.21.15.160:0/4163419146 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa02002ff60 0x7fa020032420 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:35:31.411 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:31.410+0000 7fa03c68e700 1 --2- 172.21.15.160:0/4163419146 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0340f5410 0x7fa034109430 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:35:31.411 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:31.410+0000 7fa03c68e700 1 -- 172.21.15.160:0/4163419146 >> 172.21.15.160:0/4163419146 conn(0x7fa0340f0b70 msgr2=0x7fa0340f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:35:31.411 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:31.410+0000 7fa03c68e700 1 -- 172.21.15.160:0/4163419146 shutdown_connections 2023-11-17T05:35:31.411 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:31.410+0000 7fa03c68e700 1 -- 172.21.15.160:0/4163419146 wait complete. 2023-11-17T05:35:31.412 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:35:32.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:31 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:35:30.355141+0000 mgr.smithi003.eddcuc (mgr.14164) 312 : cluster [DBG] pgmap v296: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:32.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:31 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:35:31.409245+0000 mon.smithi003 (mon.0) 275 : audit [DBG] from='client.? 172.21.15.160:0/4163419146' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:35:32.851 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:35:32.852 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:35:34.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:35:32.355385+0000 mgr.smithi003.eddcuc (mgr.14164) 313 : cluster [DBG] pgmap v297: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:34.054 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:34.052+0000 7fc07a27a700 1 -- 172.21.15.160:0/290134775 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0740f6510 msgr2=0x7fc0740f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:35:34.054 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:34.052+0000 7fc07a27a700 1 --2- 172.21.15.160:0/290134775 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0740f6510 0x7fc0740f6930 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7fc064009fb0 tx=0x7fc064009b30).stop 2023-11-17T05:35:34.054 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:34.052+0000 7fc07a27a700 1 -- 172.21.15.160:0/290134775 shutdown_connections 2023-11-17T05:35:34.054 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:34.052+0000 7fc07a27a700 1 --2- 172.21.15.160:0/290134775 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0740f6510 0x7fc0740f6930 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:35:34.054 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:34.052+0000 7fc07a27a700 1 -- 172.21.15.160:0/290134775 >> 172.21.15.160:0/290134775 conn(0x7fc0740f1c70 msgr2=0x7fc0740f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:35:34.054 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:34.052+0000 7fc07a27a700 1 -- 172.21.15.160:0/290134775 shutdown_connections 2023-11-17T05:35:34.054 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:34.052+0000 7fc07a27a700 1 -- 172.21.15.160:0/290134775 wait complete. 2023-11-17T05:35:34.054 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:34.052+0000 7fc07a27a700 1 Processor -- start 2023-11-17T05:35:34.054 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:34.052+0000 7fc07a27a700 1 -- start start 2023-11-17T05:35:34.054 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:34.053+0000 7fc07a27a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0740f6510 0x7fc074106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:35:34.055 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:34.053+0000 7fc07a27a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc074106c20 con 0x7fc0740f6510 2023-11-17T05:35:34.055 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:34.053+0000 7fc073fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0740f6510 0x7fc074106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:35:34.055 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:34.053+0000 7fc073fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0740f6510 0x7fc074106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:46778/0 (socket says 172.21.15.160:46778) 2023-11-17T05:35:34.055 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:34.053+0000 7fc073fff700 1 -- 172.21.15.160:0/3506084526 learned_addr learned my addr 172.21.15.160:0/3506084526 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:35:34.056 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:34.053+0000 7fc073fff700 1 -- 172.21.15.160:0/3506084526 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc064005040 con 0x7fc0740f6510 2023-11-17T05:35:34.056 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:34.053+0000 7fc073fff700 1 --2- 172.21.15.160:0/3506084526 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0740f6510 0x7fc074106730 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7fc064013510 tx=0x7fc06400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:35:34.056 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:34.053+0000 7fc070ff9700 1 -- 172.21.15.160:0/3506084526 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc064009320 con 0x7fc0740f6510 2023-11-17T05:35:34.056 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:34.054+0000 7fc070ff9700 1 -- 172.21.15.160:0/3506084526 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc064013f00 con 0x7fc0740f6510 2023-11-17T05:35:34.057 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:34.054+0000 7fc07a27a700 1 -- 172.21.15.160:0/3506084526 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc074106e20 con 0x7fc0740f6510 2023-11-17T05:35:34.057 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:34.054+0000 7fc070ff9700 1 -- 172.21.15.160:0/3506084526 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc064014d90 con 0x7fc0740f6510 2023-11-17T05:35:34.057 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:34.054+0000 7fc07a27a700 1 -- 172.21.15.160:0/3506084526 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0741072b0 con 0x7fc0740f6510 2023-11-17T05:35:34.057 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:34.055+0000 7fc070ff9700 1 -- 172.21.15.160:0/3506084526 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc0640263a0 con 0x7fc0740f6510 2023-11-17T05:35:34.057 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:34.055+0000 7fc070ff9700 1 --2- 172.21.15.160:0/3506084526 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc05c02ff60 0x7fc05c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:35:34.057 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:34.055+0000 7fc070ff9700 1 -- 172.21.15.160:0/3506084526 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc064041020 con 0x7fc0740f6510 2023-11-17T05:35:34.057 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:34.055+0000 7fc07a27a700 1 -- 172.21.15.160:0/3506084526 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc074045ad0 con 0x7fc0740f6510 2023-11-17T05:35:34.057 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:34.055+0000 7fc0737fe700 1 --2- 172.21.15.160:0/3506084526 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc05c02ff60 0x7fc05c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:35:34.057 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:34.056+0000 7fc0737fe700 1 --2- 172.21.15.160:0/3506084526 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc05c02ff60 0x7fc05c032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fc0680096a0 tx=0x7fc068006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:35:34.061 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:34.060+0000 7fc070ff9700 1 -- 172.21.15.160:0/3506084526 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc064012920 con 0x7fc0740f6510 2023-11-17T05:35:34.266 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:34.265+0000 7fc07a27a700 1 -- 172.21.15.160:0/3506084526 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc0740fbf60 con 0x7fc0740f6510 2023-11-17T05:35:34.267 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:34.266+0000 7fc070ff9700 1 -- 172.21.15.160:0/3506084526 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fc06402b020 con 0x7fc0740f6510 2023-11-17T05:35:34.267 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:35:34.267 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:35:34.269 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:34.267+0000 7fc07a27a700 1 -- 172.21.15.160:0/3506084526 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc05c02ff60 msgr2=0x7fc05c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:35:34.269 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:34.267+0000 7fc07a27a700 1 --2- 172.21.15.160:0/3506084526 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc05c02ff60 0x7fc05c032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fc0680096a0 tx=0x7fc068006b40).stop 2023-11-17T05:35:34.269 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:34.267+0000 7fc07a27a700 1 -- 172.21.15.160:0/3506084526 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0740f6510 msgr2=0x7fc074106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:35:34.270 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:34.267+0000 7fc07a27a700 1 --2- 172.21.15.160:0/3506084526 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0740f6510 0x7fc074106730 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7fc064013510 tx=0x7fc06400b040).stop 2023-11-17T05:35:34.270 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:34.267+0000 7fc07a27a700 1 -- 172.21.15.160:0/3506084526 shutdown_connections 2023-11-17T05:35:34.270 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:34.267+0000 7fc07a27a700 1 --2- 172.21.15.160:0/3506084526 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc05c02ff60 0x7fc05c032420 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:35:34.270 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:34.267+0000 7fc07a27a700 1 --2- 172.21.15.160:0/3506084526 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0740f6510 0x7fc074106730 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:35:34.270 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:34.267+0000 7fc07a27a700 1 -- 172.21.15.160:0/3506084526 >> 172.21.15.160:0/3506084526 conn(0x7fc0740f1c70 msgr2=0x7fc0740f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:35:34.271 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:34.267+0000 7fc07a27a700 1 -- 172.21.15.160:0/3506084526 shutdown_connections 2023-11-17T05:35:34.271 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:34.267+0000 7fc07a27a700 1 -- 172.21.15.160:0/3506084526 wait complete. 2023-11-17T05:35:34.271 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:35:35.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:35 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:35:34.266726+0000 mon.smithi003 (mon.0) 276 : audit [DBG] from='client.? 172.21.15.160:0/3506084526' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:35:35.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:35 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:35:34.355709+0000 mgr.smithi003.eddcuc (mgr.14164) 314 : cluster [DBG] pgmap v298: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:35:35.644 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:35:35.645 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:35:36.831 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:36.830+0000 7fe39921d700 1 -- 172.21.15.160:0/1768824232 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3940f6510 msgr2=0x7fe3940f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:35:36.832 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:36.830+0000 7fe39921d700 1 --2- 172.21.15.160:0/1768824232 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3940f6510 0x7fe3940f6930 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7fe388009fb0 tx=0x7fe388009b30).stop 2023-11-17T05:35:36.832 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:36.830+0000 7fe39921d700 1 -- 172.21.15.160:0/1768824232 shutdown_connections 2023-11-17T05:35:36.832 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:36.831+0000 7fe39921d700 1 --2- 172.21.15.160:0/1768824232 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3940f6510 0x7fe3940f6930 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:35:36.832 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:36.831+0000 7fe39921d700 1 -- 172.21.15.160:0/1768824232 >> 172.21.15.160:0/1768824232 conn(0x7fe3940f1c70 msgr2=0x7fe3940f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:35:36.832 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:36.831+0000 7fe39921d700 1 -- 172.21.15.160:0/1768824232 shutdown_connections 2023-11-17T05:35:36.832 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:36.831+0000 7fe39921d700 1 -- 172.21.15.160:0/1768824232 wait complete. 2023-11-17T05:35:36.834 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:36.831+0000 7fe39921d700 1 Processor -- start 2023-11-17T05:35:36.834 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:36.831+0000 7fe39921d700 1 -- start start 2023-11-17T05:35:36.834 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:36.831+0000 7fe39921d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3940f6510 0x7fe394106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:35:36.834 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:36.831+0000 7fe39921d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe394106c20 con 0x7fe3940f6510 2023-11-17T05:35:36.834 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:36.832+0000 7fe392d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3940f6510 0x7fe394106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:35:36.834 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:36.832+0000 7fe392d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3940f6510 0x7fe394106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:46790/0 (socket says 172.21.15.160:46790) 2023-11-17T05:35:36.834 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:36.832+0000 7fe392d9d700 1 -- 172.21.15.160:0/3347569091 learned_addr learned my addr 172.21.15.160:0/3347569091 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:35:36.834 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:36.832+0000 7fe392d9d700 1 -- 172.21.15.160:0/3347569091 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe388005040 con 0x7fe3940f6510 2023-11-17T05:35:36.834 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:36.832+0000 7fe392d9d700 1 --2- 172.21.15.160:0/3347569091 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3940f6510 0x7fe394106730 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7fe388013510 tx=0x7fe38800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:35:36.834 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:36.832+0000 7fe37b7fe700 1 -- 172.21.15.160:0/3347569091 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe388009320 con 0x7fe3940f6510 2023-11-17T05:35:36.834 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:36.832+0000 7fe39921d700 1 -- 172.21.15.160:0/3347569091 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe394106e20 con 0x7fe3940f6510 2023-11-17T05:35:36.834 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:36.832+0000 7fe37b7fe700 1 -- 172.21.15.160:0/3347569091 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe388013f00 con 0x7fe3940f6510 2023-11-17T05:35:36.835 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:36.832+0000 7fe37b7fe700 1 -- 172.21.15.160:0/3347569091 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe388014d90 con 0x7fe3940f6510 2023-11-17T05:35:36.835 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:36.833+0000 7fe39921d700 1 -- 172.21.15.160:0/3347569091 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3941072b0 con 0x7fe3940f6510 2023-11-17T05:35:36.836 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:36.833+0000 7fe37b7fe700 1 -- 172.21.15.160:0/3347569091 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe388014420 con 0x7fe3940f6510 2023-11-17T05:35:36.836 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:36.833+0000 7fe39921d700 1 -- 172.21.15.160:0/3347569091 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe394045ad0 con 0x7fe3940f6510 2023-11-17T05:35:36.836 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:36.833+0000 7fe37b7fe700 1 --2- 172.21.15.160:0/3347569091 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe37c02ff60 0x7fe37c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:35:36.836 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:36.833+0000 7fe37b7fe700 1 -- 172.21.15.160:0/3347569091 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe388041000 con 0x7fe3940f6510 2023-11-17T05:35:36.836 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:36.834+0000 7fe39259c700 1 --2- 172.21.15.160:0/3347569091 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe37c02ff60 0x7fe37c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:35:36.836 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:36.834+0000 7fe39259c700 1 --2- 172.21.15.160:0/3347569091 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe37c02ff60 0x7fe37c032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fe3840096a0 tx=0x7fe384006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:35:36.839 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:36.838+0000 7fe37b7fe700 1 -- 172.21.15.160:0/3347569091 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe3880095f0 con 0x7fe3940f6510 2023-11-17T05:35:37.047 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:37.046+0000 7fe39921d700 1 -- 172.21.15.160:0/3347569091 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe3940fbf60 con 0x7fe3940f6510 2023-11-17T05:35:37.048 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:37.047+0000 7fe37b7fe700 1 -- 172.21.15.160:0/3347569091 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fe38801db70 con 0x7fe3940f6510 2023-11-17T05:35:37.048 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:35:37.048 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:35:37.049 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:37.049+0000 7fe39921d700 1 -- 172.21.15.160:0/3347569091 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe37c02ff60 msgr2=0x7fe37c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:35:37.050 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:37.049+0000 7fe39921d700 1 --2- 172.21.15.160:0/3347569091 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe37c02ff60 0x7fe37c032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fe3840096a0 tx=0x7fe384006b40).stop 2023-11-17T05:35:37.050 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:37.049+0000 7fe39921d700 1 -- 172.21.15.160:0/3347569091 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3940f6510 msgr2=0x7fe394106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:35:37.050 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:37.049+0000 7fe39921d700 1 --2- 172.21.15.160:0/3347569091 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3940f6510 0x7fe394106730 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7fe388013510 tx=0x7fe38800b040).stop 2023-11-17T05:35:37.050 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:37.049+0000 7fe39921d700 1 -- 172.21.15.160:0/3347569091 shutdown_connections 2023-11-17T05:35:37.050 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:37.049+0000 7fe39921d700 1 --2- 172.21.15.160:0/3347569091 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe37c02ff60 0x7fe37c032420 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:35:37.050 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:37.049+0000 7fe39921d700 1 --2- 172.21.15.160:0/3347569091 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3940f6510 0x7fe394106730 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:35:37.050 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:37.049+0000 7fe39921d700 1 -- 172.21.15.160:0/3347569091 >> 172.21.15.160:0/3347569091 conn(0x7fe3940f1c70 msgr2=0x7fe3940f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:35:37.050 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:37.049+0000 7fe39921d700 1 -- 172.21.15.160:0/3347569091 shutdown_connections 2023-11-17T05:35:37.050 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:35:37.049+0000 7fe39921d700 1 -- 172.21.15.160:0/3347569091 wait complete. 2023-11-17T05:35:37.051 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:35:37.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:37 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:35:36.355968+0000 mgr.smithi003.eddcuc (mgr.14164) 315 : cluster [DBG] pgmap v299: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:25.004 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:37 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:35:37.048366+0000 mon.smithi003 (mon.0) 277 : audit [DBG] from='client.? 172.21.15.160:0/3347569091' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:36:25.008 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:36:25.009 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:36:25.081 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:39 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:35:38.356353+0000 mgr.smithi003.eddcuc (mgr.14164) 316 : cluster [DBG] pgmap v300: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:25.081 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:41 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:35:40.356767+0000 mgr.smithi003.eddcuc (mgr.14164) 317 : cluster [DBG] pgmap v301: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:25.081 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:43 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:35:42.357040+0000 mgr.smithi003.eddcuc (mgr.14164) 318 : cluster [DBG] pgmap v302: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:25.081 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:45 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:35:44.357270+0000 mgr.smithi003.eddcuc (mgr.14164) 319 : cluster [DBG] pgmap v303: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:25.081 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:47 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:35:46.357555+0000 mgr.smithi003.eddcuc (mgr.14164) 320 : cluster [DBG] pgmap v304: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:25.081 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:49 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:35:48.357772+0000 mgr.smithi003.eddcuc (mgr.14164) 321 : cluster [DBG] pgmap v305: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:25.081 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:51 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:35:50.358039+0000 mgr.smithi003.eddcuc (mgr.14164) 322 : cluster [DBG] pgmap v306: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:25.081 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:53 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:35:52.358287+0000 mgr.smithi003.eddcuc (mgr.14164) 323 : cluster [DBG] pgmap v307: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:25.081 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:55 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:35:54.358681+0000 mgr.smithi003.eddcuc (mgr.14164) 324 : cluster [DBG] pgmap v308: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:25.081 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:35:56.359004+0000 mgr.smithi003.eddcuc (mgr.14164) 325 : cluster [DBG] pgmap v309: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:25.082 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:35:59 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:35:58.359365+0000 mgr.smithi003.eddcuc (mgr.14164) 326 : cluster [DBG] pgmap v310: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:25.082 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:36:01 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:36:00.359763+0000 mgr.smithi003.eddcuc (mgr.14164) 327 : cluster [DBG] pgmap v311: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:25.082 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:36:03 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:36:02.360054+0000 mgr.smithi003.eddcuc (mgr.14164) 328 : cluster [DBG] pgmap v312: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:25.082 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:36:05 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:36:04.360430+0000 mgr.smithi003.eddcuc (mgr.14164) 329 : cluster [DBG] pgmap v313: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:25.154 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:36:07 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:36:06.360816+0000 mgr.smithi003.eddcuc (mgr.14164) 330 : cluster [DBG] pgmap v314: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:25.154 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:36:09 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:36:08.361232+0000 mgr.smithi003.eddcuc (mgr.14164) 331 : cluster [DBG] pgmap v315: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:25.154 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:36:11 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:36:10.361587+0000 mgr.smithi003.eddcuc (mgr.14164) 332 : cluster [DBG] pgmap v316: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:25.154 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:36:13 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:36:12.361944+0000 mgr.smithi003.eddcuc (mgr.14164) 333 : cluster [DBG] pgmap v317: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:25.154 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:36:15 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:36:14.362350+0000 mgr.smithi003.eddcuc (mgr.14164) 334 : cluster [DBG] pgmap v318: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:25.154 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:36:17 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:36:16.362631+0000 mgr.smithi003.eddcuc (mgr.14164) 335 : cluster [DBG] pgmap v319: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:25.155 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:36:19 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:36:18.362931+0000 mgr.smithi003.eddcuc (mgr.14164) 336 : cluster [DBG] pgmap v320: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:25.155 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:36:21 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:36:20.363294+0000 mgr.smithi003.eddcuc (mgr.14164) 337 : cluster [DBG] pgmap v321: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:25.155 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:36:23 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:36:22.363682+0000 mgr.smithi003.eddcuc (mgr.14164) 338 : cluster [DBG] pgmap v322: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:25.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:36:25 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:36:24.363965+0000 mgr.smithi003.eddcuc (mgr.14164) 339 : cluster [DBG] pgmap v323: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:26.374 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:26.372+0000 7fbaa2f4b700 1 -- 172.21.15.160:0/3299912839 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba9c0f6990 msgr2=0x7fba9c0f6db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:36:26.374 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:26.372+0000 7fbaa2f4b700 1 --2- 172.21.15.160:0/3299912839 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba9c0f6990 0x7fba9c0f6db0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7fba90009fb0 tx=0x7fba90009b30).stop 2023-11-17T05:36:26.374 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:26.372+0000 7fbaa2f4b700 1 -- 172.21.15.160:0/3299912839 shutdown_connections 2023-11-17T05:36:26.374 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:26.372+0000 7fbaa2f4b700 1 --2- 172.21.15.160:0/3299912839 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba9c0f6990 0x7fba9c0f6db0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:36:26.375 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:26.372+0000 7fbaa2f4b700 1 -- 172.21.15.160:0/3299912839 >> 172.21.15.160:0/3299912839 conn(0x7fba9c0f0260 msgr2=0x7fba9c0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:36:26.375 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:26.372+0000 7fbaa2f4b700 1 -- 172.21.15.160:0/3299912839 shutdown_connections 2023-11-17T05:36:26.375 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:26.372+0000 7fbaa2f4b700 1 -- 172.21.15.160:0/3299912839 wait complete. 2023-11-17T05:36:26.375 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:26.373+0000 7fbaa2f4b700 1 Processor -- start 2023-11-17T05:36:26.375 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:26.373+0000 7fbaa2f4b700 1 -- start start 2023-11-17T05:36:26.375 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:26.374+0000 7fbaa2f4b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba9c0f6990 0x7fba9c109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:36:26.375 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:26.374+0000 7fbaa2f4b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba9c0f52b0 con 0x7fba9c0f6990 2023-11-17T05:36:26.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:26.374+0000 7fbaa0ce7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba9c0f6990 0x7fba9c109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:36:26.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:26.374+0000 7fbaa0ce7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba9c0f6990 0x7fba9c109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:36210/0 (socket says 172.21.15.160:36210) 2023-11-17T05:36:26.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:26.374+0000 7fbaa0ce7700 1 -- 172.21.15.160:0/2898345511 learned_addr learned my addr 172.21.15.160:0/2898345511 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:36:26.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:26.374+0000 7fbaa0ce7700 1 -- 172.21.15.160:0/2898345511 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba90005040 con 0x7fba9c0f6990 2023-11-17T05:36:26.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:26.374+0000 7fbaa0ce7700 1 --2- 172.21.15.160:0/2898345511 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba9c0f6990 0x7fba9c109450 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fba90013510 tx=0x7fba9000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:36:26.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:26.375+0000 7fba997fa700 1 -- 172.21.15.160:0/2898345511 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba90009320 con 0x7fba9c0f6990 2023-11-17T05:36:26.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:26.375+0000 7fbaa2f4b700 1 -- 172.21.15.160:0/2898345511 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba9c0f54b0 con 0x7fba9c0f6990 2023-11-17T05:36:26.377 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:26.375+0000 7fba997fa700 1 -- 172.21.15.160:0/2898345511 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fba90013f00 con 0x7fba9c0f6990 2023-11-17T05:36:26.377 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:26.375+0000 7fba997fa700 1 -- 172.21.15.160:0/2898345511 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba90014d90 con 0x7fba9c0f6990 2023-11-17T05:36:26.377 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:26.375+0000 7fbaa2f4b700 1 -- 172.21.15.160:0/2898345511 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba9c109b00 con 0x7fba9c0f6990 2023-11-17T05:36:26.377 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:26.376+0000 7fba997fa700 1 -- 172.21.15.160:0/2898345511 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fba900095f0 con 0x7fba9c0f6990 2023-11-17T05:36:26.378 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:26.376+0000 7fbaa2f4b700 1 -- 172.21.15.160:0/2898345511 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba9c059100 con 0x7fba9c0f6990 2023-11-17T05:36:26.378 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:26.376+0000 7fba997fa700 1 --2- 172.21.15.160:0/2898345511 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fba8c02ff60 0x7fba8c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:36:26.378 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:26.376+0000 7fba997fa700 1 -- 172.21.15.160:0/2898345511 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fba90040fd0 con 0x7fba9c0f6990 2023-11-17T05:36:26.378 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:26.376+0000 7fba9bfff700 1 --2- 172.21.15.160:0/2898345511 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fba8c02ff60 0x7fba8c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:36:26.378 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:26.376+0000 7fba9bfff700 1 --2- 172.21.15.160:0/2898345511 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fba8c02ff60 0x7fba8c032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fba880096a0 tx=0x7fba88006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:36:26.381 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:26.381+0000 7fba997fa700 1 -- 172.21.15.160:0/2898345511 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fba90014420 con 0x7fba9c0f6990 2023-11-17T05:36:26.590 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:26.589+0000 7fbaa2f4b700 1 -- 172.21.15.160:0/2898345511 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fba9c045ad0 con 0x7fba9c0f6990 2023-11-17T05:36:26.591 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:26.590+0000 7fba997fa700 1 -- 172.21.15.160:0/2898345511 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fba9001d7f0 con 0x7fba9c0f6990 2023-11-17T05:36:26.591 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:36:26.591 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:36:26.592 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:26.591+0000 7fbaa2f4b700 1 -- 172.21.15.160:0/2898345511 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fba8c02ff60 msgr2=0x7fba8c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:36:26.592 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:26.591+0000 7fbaa2f4b700 1 --2- 172.21.15.160:0/2898345511 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fba8c02ff60 0x7fba8c032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fba880096a0 tx=0x7fba88006b40).stop 2023-11-17T05:36:26.592 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:26.591+0000 7fbaa2f4b700 1 -- 172.21.15.160:0/2898345511 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba9c0f6990 msgr2=0x7fba9c109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:36:26.593 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:26.591+0000 7fbaa2f4b700 1 --2- 172.21.15.160:0/2898345511 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba9c0f6990 0x7fba9c109450 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fba90013510 tx=0x7fba9000b040).stop 2023-11-17T05:36:26.593 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:26.592+0000 7fbaa2f4b700 1 -- 172.21.15.160:0/2898345511 shutdown_connections 2023-11-17T05:36:26.593 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:26.592+0000 7fbaa2f4b700 1 --2- 172.21.15.160:0/2898345511 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fba8c02ff60 0x7fba8c032420 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:36:26.593 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:26.592+0000 7fbaa2f4b700 1 --2- 172.21.15.160:0/2898345511 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba9c0f6990 0x7fba9c109450 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:36:26.593 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:26.592+0000 7fbaa2f4b700 1 -- 172.21.15.160:0/2898345511 >> 172.21.15.160:0/2898345511 conn(0x7fba9c0f0260 msgr2=0x7fba9c0f0d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:36:26.593 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:26.592+0000 7fbaa2f4b700 1 -- 172.21.15.160:0/2898345511 shutdown_connections 2023-11-17T05:36:26.593 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:26.592+0000 7fbaa2f4b700 1 -- 172.21.15.160:0/2898345511 wait complete. 2023-11-17T05:36:26.593 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:36:27.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:36:27 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:36:26.364295+0000 mgr.smithi003.eddcuc (mgr.14164) 340 : cluster [DBG] pgmap v324: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:27.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:36:27 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:36:26.591041+0000 mon.smithi003 (mon.0) 278 : audit [DBG] from='client.? 172.21.15.160:0/2898345511' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:36:28.027 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:36:28.028 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:36:28.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:36:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:36:27.557277+0000 mon.smithi003 (mon.0) 279 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.eddcuc/mirror_snapshot_schedule"}]: dispatch 2023-11-17T05:36:28.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:36:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:36:27.560810+0000 mon.smithi003 (mon.0) 280 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.eddcuc/trash_purge_schedule"}]: dispatch 2023-11-17T05:36:29.216 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:29.214+0000 7fe90e5bb700 1 -- 172.21.15.160:0/3242837792 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe9080f5410 msgr2=0x7fe9080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:36:29.216 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:29.214+0000 7fe90e5bb700 1 --2- 172.21.15.160:0/3242837792 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe9080f5410 0x7fe9080f5830 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7fe8fc009fb0 tx=0x7fe8fc009b30).stop 2023-11-17T05:36:29.216 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:29.214+0000 7fe90e5bb700 1 -- 172.21.15.160:0/3242837792 shutdown_connections 2023-11-17T05:36:29.216 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:29.214+0000 7fe90e5bb700 1 --2- 172.21.15.160:0/3242837792 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe9080f5410 0x7fe9080f5830 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:36:29.216 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:29.214+0000 7fe90e5bb700 1 -- 172.21.15.160:0/3242837792 >> 172.21.15.160:0/3242837792 conn(0x7fe9080f0b70 msgr2=0x7fe9080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:36:29.217 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:29.214+0000 7fe90e5bb700 1 -- 172.21.15.160:0/3242837792 shutdown_connections 2023-11-17T05:36:29.217 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:29.214+0000 7fe90e5bb700 1 -- 172.21.15.160:0/3242837792 wait complete. 2023-11-17T05:36:29.217 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:29.214+0000 7fe90e5bb700 1 Processor -- start 2023-11-17T05:36:29.217 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:29.215+0000 7fe90e5bb700 1 -- start start 2023-11-17T05:36:29.217 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:29.215+0000 7fe90e5bb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe9080f5410 0x7fe90810b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:36:29.217 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:29.215+0000 7fe90e5bb700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe908107240 con 0x7fe9080f5410 2023-11-17T05:36:29.217 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:29.215+0000 7fe907fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe9080f5410 0x7fe90810b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:36:29.217 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:29.215+0000 7fe907fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe9080f5410 0x7fe90810b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:36226/0 (socket says 172.21.15.160:36226) 2023-11-17T05:36:29.217 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:29.215+0000 7fe907fff700 1 -- 172.21.15.160:0/775569924 learned_addr learned my addr 172.21.15.160:0/775569924 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:36:29.217 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:29.215+0000 7fe907fff700 1 -- 172.21.15.160:0/775569924 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8fc005040 con 0x7fe9080f5410 2023-11-17T05:36:29.217 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:29.215+0000 7fe907fff700 1 --2- 172.21.15.160:0/775569924 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe9080f5410 0x7fe90810b850 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7fe8fc013510 tx=0x7fe8fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:36:29.219 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:29.216+0000 7fe904ff9700 1 -- 172.21.15.160:0/775569924 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8fc009320 con 0x7fe9080f5410 2023-11-17T05:36:29.219 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:29.216+0000 7fe90e5bb700 1 -- 172.21.15.160:0/775569924 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe908060220 con 0x7fe9080f5410 2023-11-17T05:36:29.219 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:29.216+0000 7fe904ff9700 1 -- 172.21.15.160:0/775569924 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe8fc013f00 con 0x7fe9080f5410 2023-11-17T05:36:29.219 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:29.216+0000 7fe904ff9700 1 -- 172.21.15.160:0/775569924 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8fc014d90 con 0x7fe9080f5410 2023-11-17T05:36:29.219 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:29.216+0000 7fe90e5bb700 1 -- 172.21.15.160:0/775569924 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe90810bd40 con 0x7fe9080f5410 2023-11-17T05:36:29.219 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:29.217+0000 7fe904ff9700 1 -- 172.21.15.160:0/775569924 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe8fc0095f0 con 0x7fe9080f5410 2023-11-17T05:36:29.219 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:29.217+0000 7fe90e5bb700 1 -- 172.21.15.160:0/775569924 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe908045ad0 con 0x7fe9080f5410 2023-11-17T05:36:29.219 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:29.217+0000 7fe904ff9700 1 --2- 172.21.15.160:0/775569924 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe8f002ff60 0x7fe8f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:36:29.219 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:29.217+0000 7fe904ff9700 1 -- 172.21.15.160:0/775569924 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe8fc040fe0 con 0x7fe9080f5410 2023-11-17T05:36:29.219 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:29.217+0000 7fe9077fe700 1 --2- 172.21.15.160:0/775569924 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe8f002ff60 0x7fe8f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:36:29.219 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:29.217+0000 7fe9077fe700 1 --2- 172.21.15.160:0/775569924 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe8f002ff60 0x7fe8f0032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fe8f80096a0 tx=0x7fe8f8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:36:29.222 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:29.222+0000 7fe904ff9700 1 -- 172.21.15.160:0/775569924 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe8fc01d7f0 con 0x7fe9080f5410 2023-11-17T05:36:29.430 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:29.430+0000 7fe90e5bb700 1 -- 172.21.15.160:0/775569924 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe9080fae60 con 0x7fe9080f5410 2023-11-17T05:36:29.431 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:29.430+0000 7fe904ff9700 1 -- 172.21.15.160:0/775569924 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fe8fc010230 con 0x7fe9080f5410 2023-11-17T05:36:29.431 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:36:29.431 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:36:29.433 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:29.432+0000 7fe90e5bb700 1 -- 172.21.15.160:0/775569924 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe8f002ff60 msgr2=0x7fe8f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:36:29.433 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:29.432+0000 7fe90e5bb700 1 --2- 172.21.15.160:0/775569924 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe8f002ff60 0x7fe8f0032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fe8f80096a0 tx=0x7fe8f8006b40).stop 2023-11-17T05:36:29.433 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:29.432+0000 7fe90e5bb700 1 -- 172.21.15.160:0/775569924 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe9080f5410 msgr2=0x7fe90810b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:36:29.433 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:29.432+0000 7fe90e5bb700 1 --2- 172.21.15.160:0/775569924 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe9080f5410 0x7fe90810b850 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7fe8fc013510 tx=0x7fe8fc00b040).stop 2023-11-17T05:36:29.433 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:29.432+0000 7fe90e5bb700 1 -- 172.21.15.160:0/775569924 shutdown_connections 2023-11-17T05:36:29.433 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:29.432+0000 7fe90e5bb700 1 --2- 172.21.15.160:0/775569924 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe8f002ff60 0x7fe8f0032420 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:36:29.433 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:29.432+0000 7fe90e5bb700 1 --2- 172.21.15.160:0/775569924 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe9080f5410 0x7fe90810b850 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:36:29.433 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:29.432+0000 7fe90e5bb700 1 -- 172.21.15.160:0/775569924 >> 172.21.15.160:0/775569924 conn(0x7fe9080f0b70 msgr2=0x7fe9080f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:36:29.434 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:29.432+0000 7fe90e5bb700 1 -- 172.21.15.160:0/775569924 shutdown_connections 2023-11-17T05:36:29.434 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:29.432+0000 7fe90e5bb700 1 -- 172.21.15.160:0/775569924 wait complete. 2023-11-17T05:36:29.434 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:36:29.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:36:29 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:36:28.364595+0000 mgr.smithi003.eddcuc (mgr.14164) 341 : cluster [DBG] pgmap v325: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:30.699 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:36:30 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:36:29.431532+0000 mon.smithi003 (mon.0) 281 : audit [DBG] from='client.? 172.21.15.160:0/775569924' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:36:30.819 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:36:30.820 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:36:31.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:36:31 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:36:30.364864+0000 mgr.smithi003.eddcuc (mgr.14164) 342 : cluster [DBG] pgmap v326: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:32.065 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:32.064+0000 7fc9c124e700 1 -- 172.21.15.160:0/2760724163 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc9bc0f5410 msgr2=0x7fc9bc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:36:32.065 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:32.064+0000 7fc9c124e700 1 --2- 172.21.15.160:0/2760724163 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc9bc0f5410 0x7fc9bc0f5830 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7fc9a4009fb0 tx=0x7fc9a4009b30).stop 2023-11-17T05:36:32.066 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:32.064+0000 7fc9c124e700 1 -- 172.21.15.160:0/2760724163 shutdown_connections 2023-11-17T05:36:32.066 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:32.064+0000 7fc9c124e700 1 --2- 172.21.15.160:0/2760724163 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc9bc0f5410 0x7fc9bc0f5830 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:36:32.066 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:32.064+0000 7fc9c124e700 1 -- 172.21.15.160:0/2760724163 >> 172.21.15.160:0/2760724163 conn(0x7fc9bc0f0b70 msgr2=0x7fc9bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:36:32.066 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:32.064+0000 7fc9c124e700 1 -- 172.21.15.160:0/2760724163 shutdown_connections 2023-11-17T05:36:32.066 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:32.064+0000 7fc9c124e700 1 -- 172.21.15.160:0/2760724163 wait complete. 2023-11-17T05:36:32.067 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:32.065+0000 7fc9c124e700 1 Processor -- start 2023-11-17T05:36:32.067 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:32.065+0000 7fc9c124e700 1 -- start start 2023-11-17T05:36:32.068 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:32.065+0000 7fc9c124e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc9bc0f5410 0x7fc9bc105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:36:32.068 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:32.065+0000 7fc9c124e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9bc105b20 con 0x7fc9bc0f5410 2023-11-17T05:36:32.068 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:32.066+0000 7fc9bad9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc9bc0f5410 0x7fc9bc105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:36:32.068 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:32.066+0000 7fc9bad9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc9bc0f5410 0x7fc9bc105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:40506/0 (socket says 172.21.15.160:40506) 2023-11-17T05:36:32.068 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:32.066+0000 7fc9bad9d700 1 -- 172.21.15.160:0/2404734658 learned_addr learned my addr 172.21.15.160:0/2404734658 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:36:32.068 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:32.066+0000 7fc9bad9d700 1 -- 172.21.15.160:0/2404734658 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a4005040 con 0x7fc9bc0f5410 2023-11-17T05:36:32.068 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:32.066+0000 7fc9bad9d700 1 --2- 172.21.15.160:0/2404734658 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc9bc0f5410 0x7fc9bc105630 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7fc9a4013510 tx=0x7fc9a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:36:32.068 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:32.066+0000 7fc9b37fe700 1 -- 172.21.15.160:0/2404734658 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc9a4009320 con 0x7fc9bc0f5410 2023-11-17T05:36:32.069 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:32.066+0000 7fc9c124e700 1 -- 172.21.15.160:0/2404734658 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9bc105d20 con 0x7fc9bc0f5410 2023-11-17T05:36:32.069 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:32.066+0000 7fc9b37fe700 1 -- 172.21.15.160:0/2404734658 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc9a4013f00 con 0x7fc9bc0f5410 2023-11-17T05:36:32.069 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:32.066+0000 7fc9b37fe700 1 -- 172.21.15.160:0/2404734658 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc9a4014d90 con 0x7fc9bc0f5410 2023-11-17T05:36:32.069 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:32.066+0000 7fc9c124e700 1 -- 172.21.15.160:0/2404734658 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9bc1061b0 con 0x7fc9bc0f5410 2023-11-17T05:36:32.070 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:32.067+0000 7fc9b37fe700 1 -- 172.21.15.160:0/2404734658 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc9a40095f0 con 0x7fc9bc0f5410 2023-11-17T05:36:32.070 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:32.067+0000 7fc9c124e700 1 -- 172.21.15.160:0/2404734658 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9bc045ad0 con 0x7fc9bc0f5410 2023-11-17T05:36:32.070 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:32.067+0000 7fc9b37fe700 1 --2- 172.21.15.160:0/2404734658 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc9a802ff60 0x7fc9a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:36:32.070 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:32.067+0000 7fc9b37fe700 1 -- 172.21.15.160:0/2404734658 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc9a4042e70 con 0x7fc9bc0f5410 2023-11-17T05:36:32.070 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:32.067+0000 7fc9ba59c700 1 --2- 172.21.15.160:0/2404734658 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc9a802ff60 0x7fc9a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:36:32.070 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:32.068+0000 7fc9ba59c700 1 --2- 172.21.15.160:0/2404734658 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc9a802ff60 0x7fc9a8032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fc9ac0096a0 tx=0x7fc9ac006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:36:32.073 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:32.072+0000 7fc9b37fe700 1 -- 172.21.15.160:0/2404734658 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc9a4014420 con 0x7fc9bc0f5410 2023-11-17T05:36:32.276 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:32.275+0000 7fc9c124e700 1 -- 172.21.15.160:0/2404734658 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc9bc0fae60 con 0x7fc9bc0f5410 2023-11-17T05:36:32.278 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:32.276+0000 7fc9b37fe700 1 -- 172.21.15.160:0/2404734658 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fc9a4011290 con 0x7fc9bc0f5410 2023-11-17T05:36:32.278 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:36:32.279 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:36:32.279 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:32.278+0000 7fc9c124e700 1 -- 172.21.15.160:0/2404734658 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc9a802ff60 msgr2=0x7fc9a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:36:32.279 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:32.278+0000 7fc9c124e700 1 --2- 172.21.15.160:0/2404734658 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc9a802ff60 0x7fc9a8032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fc9ac0096a0 tx=0x7fc9ac006b40).stop 2023-11-17T05:36:32.279 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:32.278+0000 7fc9c124e700 1 -- 172.21.15.160:0/2404734658 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc9bc0f5410 msgr2=0x7fc9bc105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:36:32.279 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:32.278+0000 7fc9c124e700 1 --2- 172.21.15.160:0/2404734658 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc9bc0f5410 0x7fc9bc105630 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7fc9a4013510 tx=0x7fc9a400b040).stop 2023-11-17T05:36:32.279 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:32.278+0000 7fc9c124e700 1 -- 172.21.15.160:0/2404734658 shutdown_connections 2023-11-17T05:36:32.280 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:32.278+0000 7fc9c124e700 1 --2- 172.21.15.160:0/2404734658 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc9a802ff60 0x7fc9a8032420 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:36:32.280 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:32.278+0000 7fc9c124e700 1 --2- 172.21.15.160:0/2404734658 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc9bc0f5410 0x7fc9bc105630 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:36:32.280 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:32.278+0000 7fc9c124e700 1 -- 172.21.15.160:0/2404734658 >> 172.21.15.160:0/2404734658 conn(0x7fc9bc0f0b70 msgr2=0x7fc9bc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:36:32.280 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:32.278+0000 7fc9c124e700 1 -- 172.21.15.160:0/2404734658 shutdown_connections 2023-11-17T05:36:32.280 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:32.278+0000 7fc9c124e700 1 -- 172.21.15.160:0/2404734658 wait complete. 2023-11-17T05:36:32.280 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:36:32.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:36:32 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:36:32.277276+0000 mon.smithi003 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.160:0/2404734658' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:36:33.680 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:36:33.681 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:36:33.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:36:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:36:32.365188+0000 mgr.smithi003.eddcuc (mgr.14164) 343 : cluster [DBG] pgmap v327: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:34.906 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:34.904+0000 7fe8ab8e4700 1 -- 172.21.15.160:0/3465679914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8a40f6510 msgr2=0x7fe8a40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:36:34.906 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:34.904+0000 7fe8ab8e4700 1 --2- 172.21.15.160:0/3465679914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8a40f6510 0x7fe8a40f6930 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7fe894009fb0 tx=0x7fe894009b30).stop 2023-11-17T05:36:34.907 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:34.904+0000 7fe8ab8e4700 1 -- 172.21.15.160:0/3465679914 shutdown_connections 2023-11-17T05:36:34.907 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:34.904+0000 7fe8ab8e4700 1 --2- 172.21.15.160:0/3465679914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8a40f6510 0x7fe8a40f6930 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:36:34.907 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:34.904+0000 7fe8ab8e4700 1 -- 172.21.15.160:0/3465679914 >> 172.21.15.160:0/3465679914 conn(0x7fe8a40f1c70 msgr2=0x7fe8a40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:36:34.907 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:34.904+0000 7fe8ab8e4700 1 -- 172.21.15.160:0/3465679914 shutdown_connections 2023-11-17T05:36:34.907 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:34.904+0000 7fe8ab8e4700 1 -- 172.21.15.160:0/3465679914 wait complete. 2023-11-17T05:36:34.907 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:34.905+0000 7fe8ab8e4700 1 Processor -- start 2023-11-17T05:36:34.907 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:34.905+0000 7fe8ab8e4700 1 -- start start 2023-11-17T05:36:34.907 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:34.905+0000 7fe8ab8e4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8a40f6510 0x7fe8a4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:36:34.907 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:34.905+0000 7fe8ab8e4700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a4106c20 con 0x7fe8a40f6510 2023-11-17T05:36:34.907 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:34.905+0000 7fe8a9680700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8a40f6510 0x7fe8a4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:36:34.907 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:34.905+0000 7fe8a9680700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8a40f6510 0x7fe8a4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:40522/0 (socket says 172.21.15.160:40522) 2023-11-17T05:36:34.907 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:34.905+0000 7fe8a9680700 1 -- 172.21.15.160:0/3289116883 learned_addr learned my addr 172.21.15.160:0/3289116883 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:36:34.907 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:34.906+0000 7fe8a9680700 1 -- 172.21.15.160:0/3289116883 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe894005040 con 0x7fe8a40f6510 2023-11-17T05:36:34.908 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:34.906+0000 7fe8a9680700 1 --2- 172.21.15.160:0/3289116883 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8a40f6510 0x7fe8a4106730 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7fe894013510 tx=0x7fe89400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:36:34.909 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:34.906+0000 7fe899ffb700 1 -- 172.21.15.160:0/3289116883 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe894009320 con 0x7fe8a40f6510 2023-11-17T05:36:34.909 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:34.906+0000 7fe8ab8e4700 1 -- 172.21.15.160:0/3289116883 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8a4106e20 con 0x7fe8a40f6510 2023-11-17T05:36:34.910 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:34.906+0000 7fe899ffb700 1 -- 172.21.15.160:0/3289116883 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe894013f00 con 0x7fe8a40f6510 2023-11-17T05:36:34.910 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:34.906+0000 7fe899ffb700 1 -- 172.21.15.160:0/3289116883 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe894014d90 con 0x7fe8a40f6510 2023-11-17T05:36:34.910 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:34.906+0000 7fe8ab8e4700 1 -- 172.21.15.160:0/3289116883 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8a41072b0 con 0x7fe8a40f6510 2023-11-17T05:36:34.910 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:34.907+0000 7fe899ffb700 1 -- 172.21.15.160:0/3289116883 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe8940095f0 con 0x7fe8a40f6510 2023-11-17T05:36:34.910 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:34.907+0000 7fe8ab8e4700 1 -- 172.21.15.160:0/3289116883 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8a4045ad0 con 0x7fe8a40f6510 2023-11-17T05:36:34.910 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:34.907+0000 7fe899ffb700 1 --2- 172.21.15.160:0/3289116883 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe89002ff60 0x7fe890032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:36:34.910 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:34.907+0000 7fe899ffb700 1 -- 172.21.15.160:0/3289116883 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe894041000 con 0x7fe8a40f6510 2023-11-17T05:36:34.910 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:34.907+0000 7fe8a8e7f700 1 --2- 172.21.15.160:0/3289116883 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe89002ff60 0x7fe890032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:36:34.910 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:34.908+0000 7fe8a8e7f700 1 --2- 172.21.15.160:0/3289116883 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe89002ff60 0x7fe890032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fe8a00096a0 tx=0x7fe8a0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:36:34.913 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:34.912+0000 7fe899ffb700 1 -- 172.21.15.160:0/3289116883 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe89401d7f0 con 0x7fe8a40f6510 2023-11-17T05:36:35.121 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:35.120+0000 7fe8ab8e4700 1 -- 172.21.15.160:0/3289116883 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe8a40fbf60 con 0x7fe8a40f6510 2023-11-17T05:36:35.122 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:35.122+0000 7fe899ffb700 1 -- 172.21.15.160:0/3289116883 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fe894010230 con 0x7fe8a40f6510 2023-11-17T05:36:35.122 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:36:35.122 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:36:35.123 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:35.123+0000 7fe8ab8e4700 1 -- 172.21.15.160:0/3289116883 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe89002ff60 msgr2=0x7fe890032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:36:35.123 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:35.123+0000 7fe8ab8e4700 1 --2- 172.21.15.160:0/3289116883 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe89002ff60 0x7fe890032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fe8a00096a0 tx=0x7fe8a0006b40).stop 2023-11-17T05:36:35.123 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:35.123+0000 7fe8ab8e4700 1 -- 172.21.15.160:0/3289116883 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8a40f6510 msgr2=0x7fe8a4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:36:35.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:35.123+0000 7fe8ab8e4700 1 --2- 172.21.15.160:0/3289116883 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8a40f6510 0x7fe8a4106730 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7fe894013510 tx=0x7fe89400b040).stop 2023-11-17T05:36:35.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:35.123+0000 7fe8ab8e4700 1 -- 172.21.15.160:0/3289116883 shutdown_connections 2023-11-17T05:36:35.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:35.123+0000 7fe8ab8e4700 1 --2- 172.21.15.160:0/3289116883 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe89002ff60 0x7fe890032420 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:36:35.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:35.123+0000 7fe8ab8e4700 1 --2- 172.21.15.160:0/3289116883 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8a40f6510 0x7fe8a4106730 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:36:35.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:35.123+0000 7fe8ab8e4700 1 -- 172.21.15.160:0/3289116883 >> 172.21.15.160:0/3289116883 conn(0x7fe8a40f1c70 msgr2=0x7fe8a40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:36:35.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:35.123+0000 7fe8ab8e4700 1 -- 172.21.15.160:0/3289116883 shutdown_connections 2023-11-17T05:36:35.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:35.123+0000 7fe8ab8e4700 1 -- 172.21.15.160:0/3289116883 wait complete. 2023-11-17T05:36:35.124 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:36:35.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:36:35 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:36:34.365592+0000 mgr.smithi003.eddcuc (mgr.14164) 344 : cluster [DBG] pgmap v328: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:36.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:36:36 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:36:35.122201+0000 mon.smithi003 (mon.0) 283 : audit [DBG] from='client.? 172.21.15.160:0/3289116883' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:36:36.498 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:36:36.499 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:36:37.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:36:37 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:36:36.365941+0000 mgr.smithi003.eddcuc (mgr.14164) 345 : cluster [DBG] pgmap v329: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:37.779 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:37.777+0000 7f0116d84700 1 -- 172.21.15.160:0/2880018933 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01100f5410 msgr2=0x7f01100f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:36:37.779 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:37.777+0000 7f0116d84700 1 --2- 172.21.15.160:0/2880018933 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01100f5410 0x7f01100f5830 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f0104009fb0 tx=0x7f0104009b30).stop 2023-11-17T05:36:37.779 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:37.777+0000 7f0116d84700 1 -- 172.21.15.160:0/2880018933 shutdown_connections 2023-11-17T05:36:37.779 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:37.777+0000 7f0116d84700 1 --2- 172.21.15.160:0/2880018933 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01100f5410 0x7f01100f5830 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:36:37.779 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:37.777+0000 7f0116d84700 1 -- 172.21.15.160:0/2880018933 >> 172.21.15.160:0/2880018933 conn(0x7f01100f0b70 msgr2=0x7f01100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:36:37.779 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:37.777+0000 7f0116d84700 1 -- 172.21.15.160:0/2880018933 shutdown_connections 2023-11-17T05:36:37.780 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:37.777+0000 7f0116d84700 1 -- 172.21.15.160:0/2880018933 wait complete. 2023-11-17T05:36:37.780 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:37.778+0000 7f0116d84700 1 Processor -- start 2023-11-17T05:36:37.780 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:37.778+0000 7f0116d84700 1 -- start start 2023-11-17T05:36:37.780 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:37.778+0000 7f0116d84700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01100f5410 0x7f0110109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:36:37.780 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:37.778+0000 7f0116d84700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0110060220 con 0x7f01100f5410 2023-11-17T05:36:37.781 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:37.779+0000 7f0114b20700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01100f5410 0x7f0110109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:36:37.781 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:37.779+0000 7f0114b20700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01100f5410 0x7f0110109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:40536/0 (socket says 172.21.15.160:40536) 2023-11-17T05:36:37.781 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:37.779+0000 7f0114b20700 1 -- 172.21.15.160:0/1139121639 learned_addr learned my addr 172.21.15.160:0/1139121639 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:36:37.781 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:37.779+0000 7f0114b20700 1 -- 172.21.15.160:0/1139121639 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0104005040 con 0x7f01100f5410 2023-11-17T05:36:37.782 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:37.779+0000 7f0114b20700 1 --2- 172.21.15.160:0/1139121639 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01100f5410 0x7f0110109450 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f0104013510 tx=0x7f010400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:36:37.782 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:37.779+0000 7f010d7fa700 1 -- 172.21.15.160:0/1139121639 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0104009320 con 0x7f01100f5410 2023-11-17T05:36:37.782 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:37.779+0000 7f0116d84700 1 -- 172.21.15.160:0/1139121639 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0110060420 con 0x7f01100f5410 2023-11-17T05:36:37.782 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:37.779+0000 7f010d7fa700 1 -- 172.21.15.160:0/1139121639 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0104013f00 con 0x7f01100f5410 2023-11-17T05:36:37.782 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:37.779+0000 7f010d7fa700 1 -- 172.21.15.160:0/1139121639 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0104014d90 con 0x7f01100f5410 2023-11-17T05:36:37.782 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:37.780+0000 7f0116d84700 1 -- 172.21.15.160:0/1139121639 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0110109b00 con 0x7f01100f5410 2023-11-17T05:36:37.782 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:37.780+0000 7f010d7fa700 1 -- 172.21.15.160:0/1139121639 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f01040095f0 con 0x7f01100f5410 2023-11-17T05:36:37.782 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:37.780+0000 7f0116d84700 1 -- 172.21.15.160:0/1139121639 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0110059100 con 0x7f01100f5410 2023-11-17T05:36:37.782 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:37.780+0000 7f010d7fa700 1 --2- 172.21.15.160:0/1139121639 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f010002ffb0 0x7f0100032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:36:37.783 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:37.780+0000 7f010d7fa700 1 -- 172.21.15.160:0/1139121639 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0104040f90 con 0x7f01100f5410 2023-11-17T05:36:37.783 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:37.781+0000 7f010ffff700 1 --2- 172.21.15.160:0/1139121639 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f010002ffb0 0x7f0100032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:36:37.783 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:37.781+0000 7f010ffff700 1 --2- 172.21.15.160:0/1139121639 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f010002ffb0 0x7f0100032470 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f00fc0096a0 tx=0x7f00fc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:36:37.786 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:37.785+0000 7f010d7fa700 1 -- 172.21.15.160:0/1139121639 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f010401d7f0 con 0x7f01100f5410 2023-11-17T05:36:37.990 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:37.989+0000 7f0116d84700 1 -- 172.21.15.160:0/1139121639 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0110045ad0 con 0x7f01100f5410 2023-11-17T05:36:37.991 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:37.990+0000 7f010d7fa700 1 -- 172.21.15.160:0/1139121639 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f0104010230 con 0x7f01100f5410 2023-11-17T05:36:37.991 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:36:37.991 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:36:37.992 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:37.991+0000 7f0116d84700 1 -- 172.21.15.160:0/1139121639 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f010002ffb0 msgr2=0x7f0100032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:36:37.992 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:37.991+0000 7f0116d84700 1 --2- 172.21.15.160:0/1139121639 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f010002ffb0 0x7f0100032470 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f00fc0096a0 tx=0x7f00fc006b40).stop 2023-11-17T05:36:37.992 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:37.991+0000 7f0116d84700 1 -- 172.21.15.160:0/1139121639 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01100f5410 msgr2=0x7f0110109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:36:37.993 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:37.991+0000 7f0116d84700 1 --2- 172.21.15.160:0/1139121639 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01100f5410 0x7f0110109450 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f0104013510 tx=0x7f010400b040).stop 2023-11-17T05:36:37.993 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:37.991+0000 7f0116d84700 1 -- 172.21.15.160:0/1139121639 shutdown_connections 2023-11-17T05:36:37.993 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:37.991+0000 7f0116d84700 1 --2- 172.21.15.160:0/1139121639 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f010002ffb0 0x7f0100032470 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:36:37.993 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:37.991+0000 7f0116d84700 1 --2- 172.21.15.160:0/1139121639 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01100f5410 0x7f0110109450 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:36:37.993 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:37.991+0000 7f0116d84700 1 -- 172.21.15.160:0/1139121639 >> 172.21.15.160:0/1139121639 conn(0x7f01100f0b70 msgr2=0x7f01100f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:36:37.993 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:37.992+0000 7f0116d84700 1 -- 172.21.15.160:0/1139121639 shutdown_connections 2023-11-17T05:36:37.993 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:37.992+0000 7f0116d84700 1 -- 172.21.15.160:0/1139121639 wait complete. 2023-11-17T05:36:37.993 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:36:38.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:36:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:36:37.991030+0000 mon.smithi003 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.160:0/1139121639' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:36:39.403 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:36:39.403 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:36:39.520 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:36:39 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:36:38.366281+0000 mgr.smithi003.eddcuc (mgr.14164) 346 : cluster [DBG] pgmap v330: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:40.683 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:40.680+0000 7faa04612700 1 -- 172.21.15.160:0/646157159 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9fc0f5410 msgr2=0x7fa9fc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:36:40.683 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:40.680+0000 7faa04612700 1 --2- 172.21.15.160:0/646157159 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9fc0f5410 0x7fa9fc0f5830 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7fa9ec009fb0 tx=0x7fa9ec009b30).stop 2023-11-17T05:36:40.683 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:40.680+0000 7faa04612700 1 -- 172.21.15.160:0/646157159 shutdown_connections 2023-11-17T05:36:40.684 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:40.680+0000 7faa04612700 1 --2- 172.21.15.160:0/646157159 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9fc0f5410 0x7fa9fc0f5830 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:36:40.684 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:40.680+0000 7faa04612700 1 -- 172.21.15.160:0/646157159 >> 172.21.15.160:0/646157159 conn(0x7fa9fc0f0b70 msgr2=0x7fa9fc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:36:40.684 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:40.681+0000 7faa04612700 1 -- 172.21.15.160:0/646157159 shutdown_connections 2023-11-17T05:36:40.684 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:40.681+0000 7faa04612700 1 -- 172.21.15.160:0/646157159 wait complete. 2023-11-17T05:36:40.684 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:40.681+0000 7faa04612700 1 Processor -- start 2023-11-17T05:36:40.684 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:40.681+0000 7faa04612700 1 -- start start 2023-11-17T05:36:40.684 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:40.681+0000 7faa04612700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9fc0f5410 0x7fa9fc10b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:36:40.684 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:40.681+0000 7faa04612700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9fc107240 con 0x7fa9fc0f5410 2023-11-17T05:36:40.684 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:40.681+0000 7faa023ae700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9fc0f5410 0x7fa9fc10b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:36:40.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:40.681+0000 7faa023ae700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9fc0f5410 0x7fa9fc10b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:55940/0 (socket says 172.21.15.160:55940) 2023-11-17T05:36:40.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:40.681+0000 7faa023ae700 1 -- 172.21.15.160:0/1200909355 learned_addr learned my addr 172.21.15.160:0/1200909355 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:36:40.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:40.682+0000 7faa023ae700 1 -- 172.21.15.160:0/1200909355 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9ec005040 con 0x7fa9fc0f5410 2023-11-17T05:36:40.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:40.682+0000 7faa023ae700 1 --2- 172.21.15.160:0/1200909355 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9fc0f5410 0x7fa9fc10b850 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7fa9ec013510 tx=0x7fa9ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:36:40.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:40.682+0000 7fa9f2ffd700 1 -- 172.21.15.160:0/1200909355 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9ec009320 con 0x7fa9fc0f5410 2023-11-17T05:36:40.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:40.682+0000 7faa04612700 1 -- 172.21.15.160:0/1200909355 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9fc060220 con 0x7fa9fc0f5410 2023-11-17T05:36:40.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:40.682+0000 7fa9f2ffd700 1 -- 172.21.15.160:0/1200909355 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa9ec013f00 con 0x7fa9fc0f5410 2023-11-17T05:36:40.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:40.682+0000 7fa9f2ffd700 1 -- 172.21.15.160:0/1200909355 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9ec014d90 con 0x7fa9fc0f5410 2023-11-17T05:36:40.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:40.682+0000 7faa04612700 1 -- 172.21.15.160:0/1200909355 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9fc10bd40 con 0x7fa9fc0f5410 2023-11-17T05:36:40.686 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:40.683+0000 7fa9f2ffd700 1 -- 172.21.15.160:0/1200909355 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa9ec014420 con 0x7fa9fc0f5410 2023-11-17T05:36:40.686 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:40.683+0000 7faa04612700 1 -- 172.21.15.160:0/1200909355 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9fc045ad0 con 0x7fa9fc0f5410 2023-11-17T05:36:40.686 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:40.683+0000 7fa9f2ffd700 1 --2- 172.21.15.160:0/1200909355 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa9e802ff60 0x7fa9e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:36:40.686 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:40.683+0000 7fa9f2ffd700 1 -- 172.21.15.160:0/1200909355 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa9ec041110 con 0x7fa9fc0f5410 2023-11-17T05:36:40.686 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:40.683+0000 7faa01bad700 1 --2- 172.21.15.160:0/1200909355 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa9e802ff60 0x7fa9e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:36:40.687 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:40.684+0000 7faa01bad700 1 --2- 172.21.15.160:0/1200909355 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa9e802ff60 0x7fa9e8032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fa9f80096a0 tx=0x7fa9f8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:36:40.689 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:40.688+0000 7fa9f2ffd700 1 -- 172.21.15.160:0/1200909355 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa9ec0095f0 con 0x7fa9fc0f5410 2023-11-17T05:36:40.896 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:40.894+0000 7faa04612700 1 -- 172.21.15.160:0/1200909355 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa9fc0fae60 con 0x7fa9fc0f5410 2023-11-17T05:36:40.896 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:40.895+0000 7fa9f2ffd700 1 -- 172.21.15.160:0/1200909355 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fa9ec01db70 con 0x7fa9fc0f5410 2023-11-17T05:36:40.896 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:36:40.897 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:36:40.897 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:40.896+0000 7faa04612700 1 -- 172.21.15.160:0/1200909355 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa9e802ff60 msgr2=0x7fa9e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:36:40.898 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:40.896+0000 7faa04612700 1 --2- 172.21.15.160:0/1200909355 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa9e802ff60 0x7fa9e8032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fa9f80096a0 tx=0x7fa9f8006b40).stop 2023-11-17T05:36:40.898 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:40.896+0000 7faa04612700 1 -- 172.21.15.160:0/1200909355 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9fc0f5410 msgr2=0x7fa9fc10b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:36:40.898 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:40.896+0000 7faa04612700 1 --2- 172.21.15.160:0/1200909355 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9fc0f5410 0x7fa9fc10b850 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7fa9ec013510 tx=0x7fa9ec00b040).stop 2023-11-17T05:36:40.898 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:40.896+0000 7faa04612700 1 -- 172.21.15.160:0/1200909355 shutdown_connections 2023-11-17T05:36:40.898 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:40.896+0000 7faa04612700 1 --2- 172.21.15.160:0/1200909355 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa9e802ff60 0x7fa9e8032420 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:36:40.898 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:40.896+0000 7faa04612700 1 --2- 172.21.15.160:0/1200909355 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9fc0f5410 0x7fa9fc10b850 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:36:40.898 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:40.896+0000 7faa04612700 1 -- 172.21.15.160:0/1200909355 >> 172.21.15.160:0/1200909355 conn(0x7fa9fc0f0b70 msgr2=0x7fa9fc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:36:40.898 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:40.896+0000 7faa04612700 1 -- 172.21.15.160:0/1200909355 shutdown_connections 2023-11-17T05:36:40.898 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:40.896+0000 7faa04612700 1 -- 172.21.15.160:0/1200909355 wait complete. 2023-11-17T05:36:40.899 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:36:41.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:36:41 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:36:40.366506+0000 mgr.smithi003.eddcuc (mgr.14164) 347 : cluster [DBG] pgmap v331: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:41.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:36:41 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:36:40.896078+0000 mon.smithi003 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.160:0/1200909355' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:36:42.292 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:36:42.293 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:36:43.460 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:43.459+0000 7f2c42622700 1 -- 172.21.15.160:0/1697869196 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c3c060160 msgr2=0x7f2c3c0f8fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:36:43.460 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:43.459+0000 7f2c42622700 1 --2- 172.21.15.160:0/1697869196 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c3c060160 0x7f2c3c0f8fa0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f2c2c009fb0 tx=0x7f2c2c009b30).stop 2023-11-17T05:36:43.461 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:43.459+0000 7f2c42622700 1 -- 172.21.15.160:0/1697869196 shutdown_connections 2023-11-17T05:36:43.461 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:43.459+0000 7f2c42622700 1 --2- 172.21.15.160:0/1697869196 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c3c060160 0x7f2c3c0f8fa0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:36:43.461 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:43.459+0000 7f2c42622700 1 -- 172.21.15.160:0/1697869196 >> 172.21.15.160:0/1697869196 conn(0x7f2c3c0f09a0 msgr2=0x7f2c3c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:36:43.461 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:43.459+0000 7f2c42622700 1 -- 172.21.15.160:0/1697869196 shutdown_connections 2023-11-17T05:36:43.461 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:43.459+0000 7f2c42622700 1 -- 172.21.15.160:0/1697869196 wait complete. 2023-11-17T05:36:43.461 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:43.459+0000 7f2c42622700 1 Processor -- start 2023-11-17T05:36:43.461 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:43.459+0000 7f2c42622700 1 -- start start 2023-11-17T05:36:43.461 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:43.459+0000 7f2c42622700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c3c060160 0x7f2c3c1064f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:36:43.461 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:43.459+0000 7f2c42622700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c3c1069e0 con 0x7f2c3c060160 2023-11-17T05:36:43.461 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:43.460+0000 7f2c41620700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c3c060160 0x7f2c3c1064f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:36:43.461 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:43.460+0000 7f2c41620700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c3c060160 0x7f2c3c1064f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:55948/0 (socket says 172.21.15.160:55948) 2023-11-17T05:36:43.462 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:43.460+0000 7f2c41620700 1 -- 172.21.15.160:0/3958966536 learned_addr learned my addr 172.21.15.160:0/3958966536 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:36:43.463 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:43.460+0000 7f2c41620700 1 -- 172.21.15.160:0/3958966536 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c2c005040 con 0x7f2c3c060160 2023-11-17T05:36:43.463 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:43.460+0000 7f2c41620700 1 --2- 172.21.15.160:0/3958966536 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c3c060160 0x7f2c3c1064f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f2c2c013510 tx=0x7f2c2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:36:43.463 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:43.460+0000 7f2c31ffb700 1 -- 172.21.15.160:0/3958966536 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c2c009320 con 0x7f2c3c060160 2023-11-17T05:36:43.463 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:43.460+0000 7f2c42622700 1 -- 172.21.15.160:0/3958966536 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c3c106be0 con 0x7f2c3c060160 2023-11-17T05:36:43.463 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:43.460+0000 7f2c31ffb700 1 -- 172.21.15.160:0/3958966536 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2c2c013f00 con 0x7f2c3c060160 2023-11-17T05:36:43.463 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:43.460+0000 7f2c31ffb700 1 -- 172.21.15.160:0/3958966536 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c2c014d90 con 0x7f2c3c060160 2023-11-17T05:36:43.463 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:43.461+0000 7f2c42622700 1 -- 172.21.15.160:0/3958966536 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c3c107070 con 0x7f2c3c060160 2023-11-17T05:36:43.464 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:43.461+0000 7f2c31ffb700 1 -- 172.21.15.160:0/3958966536 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2c2c0095f0 con 0x7f2c3c060160 2023-11-17T05:36:43.464 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:43.461+0000 7f2c42622700 1 -- 172.21.15.160:0/3958966536 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c3c045ad0 con 0x7f2c3c060160 2023-11-17T05:36:43.464 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:43.461+0000 7f2c31ffb700 1 --2- 172.21.15.160:0/3958966536 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f2c2802ff60 0x7f2c28032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:36:43.464 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:43.462+0000 7f2c31ffb700 1 -- 172.21.15.160:0/3958966536 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2c2c040fc0 con 0x7f2c3c060160 2023-11-17T05:36:43.464 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:43.462+0000 7f2c40e1f700 1 --2- 172.21.15.160:0/3958966536 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f2c2802ff60 0x7f2c28032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:36:43.464 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:43.462+0000 7f2c40e1f700 1 --2- 172.21.15.160:0/3958966536 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f2c2802ff60 0x7f2c28032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f2c380096a0 tx=0x7f2c38006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:36:43.469 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:43.466+0000 7f2c31ffb700 1 -- 172.21.15.160:0/3958966536 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2c2c014420 con 0x7f2c3c060160 2023-11-17T05:36:43.677 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:43.676+0000 7f2c42622700 1 -- 172.21.15.160:0/3958966536 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2c3c107f20 con 0x7f2c3c060160 2023-11-17T05:36:43.677 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:43.677+0000 7f2c31ffb700 1 -- 172.21.15.160:0/3958966536 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f2c2c010230 con 0x7f2c3c060160 2023-11-17T05:36:43.678 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:36:43.678 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:36:43.679 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:43.678+0000 7f2c42622700 1 -- 172.21.15.160:0/3958966536 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f2c2802ff60 msgr2=0x7f2c28032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:36:43.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:43.678+0000 7f2c42622700 1 --2- 172.21.15.160:0/3958966536 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f2c2802ff60 0x7f2c28032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f2c380096a0 tx=0x7f2c38006b40).stop 2023-11-17T05:36:43.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:43.678+0000 7f2c42622700 1 -- 172.21.15.160:0/3958966536 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c3c060160 msgr2=0x7f2c3c1064f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:36:43.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:43.678+0000 7f2c42622700 1 --2- 172.21.15.160:0/3958966536 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c3c060160 0x7f2c3c1064f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f2c2c013510 tx=0x7f2c2c00b040).stop 2023-11-17T05:36:43.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:43.678+0000 7f2c42622700 1 -- 172.21.15.160:0/3958966536 shutdown_connections 2023-11-17T05:36:43.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:43.678+0000 7f2c42622700 1 --2- 172.21.15.160:0/3958966536 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f2c2802ff60 0x7f2c28032420 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:36:43.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:43.678+0000 7f2c42622700 1 --2- 172.21.15.160:0/3958966536 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c3c060160 0x7f2c3c1064f0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:36:43.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:43.678+0000 7f2c42622700 1 -- 172.21.15.160:0/3958966536 >> 172.21.15.160:0/3958966536 conn(0x7f2c3c0f09a0 msgr2=0x7f2c3c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:36:43.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:43.678+0000 7f2c42622700 1 -- 172.21.15.160:0/3958966536 shutdown_connections 2023-11-17T05:36:43.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:43.678+0000 7f2c42622700 1 -- 172.21.15.160:0/3958966536 wait complete. 2023-11-17T05:36:43.680 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:36:43.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:36:43 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:36:42.366784+0000 mgr.smithi003.eddcuc (mgr.14164) 348 : cluster [DBG] pgmap v332: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:44.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:36:44 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:36:43.677835+0000 mon.smithi003 (mon.0) 286 : audit [DBG] from='client.? 172.21.15.160:0/3958966536' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:36:45.067 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:36:45.067 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:36:45.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:36:45 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:36:44.367112+0000 mgr.smithi003.eddcuc (mgr.14164) 349 : cluster [DBG] pgmap v333: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:46.270 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:46.266+0000 7f09036e0700 1 -- 172.21.15.160:0/2220896141 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08fc0f5410 msgr2=0x7f08fc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:36:46.271 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:46.266+0000 7f09036e0700 1 --2- 172.21.15.160:0/2220896141 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08fc0f5410 0x7f08fc0f5830 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f08ec009fb0 tx=0x7f08ec009b30).stop 2023-11-17T05:36:46.271 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:46.266+0000 7f09036e0700 1 -- 172.21.15.160:0/2220896141 shutdown_connections 2023-11-17T05:36:46.271 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:46.266+0000 7f09036e0700 1 --2- 172.21.15.160:0/2220896141 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08fc0f5410 0x7f08fc0f5830 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:36:46.271 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:46.266+0000 7f09036e0700 1 -- 172.21.15.160:0/2220896141 >> 172.21.15.160:0/2220896141 conn(0x7f08fc0f0b70 msgr2=0x7f08fc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:36:46.271 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:46.266+0000 7f09036e0700 1 -- 172.21.15.160:0/2220896141 shutdown_connections 2023-11-17T05:36:46.272 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:46.266+0000 7f09036e0700 1 -- 172.21.15.160:0/2220896141 wait complete. 2023-11-17T05:36:46.272 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:46.267+0000 7f09036e0700 1 Processor -- start 2023-11-17T05:36:46.272 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:46.267+0000 7f09036e0700 1 -- start start 2023-11-17T05:36:46.272 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:46.267+0000 7f09036e0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08fc0f5410 0x7f08fc109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:36:46.272 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:46.267+0000 7f09036e0700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08fc060240 con 0x7f08fc0f5410 2023-11-17T05:36:46.273 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:46.267+0000 7f090147c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08fc0f5410 0x7f08fc109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:36:46.273 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:46.267+0000 7f090147c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08fc0f5410 0x7f08fc109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:55964/0 (socket says 172.21.15.160:55964) 2023-11-17T05:36:46.273 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:46.267+0000 7f090147c700 1 -- 172.21.15.160:0/1820918209 learned_addr learned my addr 172.21.15.160:0/1820918209 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:36:46.273 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:46.267+0000 7f090147c700 1 -- 172.21.15.160:0/1820918209 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08ec005040 con 0x7f08fc0f5410 2023-11-17T05:36:46.273 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:46.268+0000 7f090147c700 1 --2- 172.21.15.160:0/1820918209 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08fc0f5410 0x7f08fc109430 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f08ec014000 tx=0x7f08ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:36:46.273 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:46.268+0000 7f08f1ffb700 1 -- 172.21.15.160:0/1820918209 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f08ec0095f0 con 0x7f08fc0f5410 2023-11-17T05:36:46.273 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:46.268+0000 7f09036e0700 1 -- 172.21.15.160:0/1820918209 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08fc060440 con 0x7f08fc0f5410 2023-11-17T05:36:46.274 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:46.268+0000 7f08f1ffb700 1 -- 172.21.15.160:0/1820918209 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f08ec007e70 con 0x7f08fc0f5410 2023-11-17T05:36:46.274 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:46.268+0000 7f08f1ffb700 1 -- 172.21.15.160:0/1820918209 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f08ec014ea0 con 0x7f08fc0f5410 2023-11-17T05:36:46.274 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:46.268+0000 7f09036e0700 1 -- 172.21.15.160:0/1820918209 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08fc109ae0 con 0x7f08fc0f5410 2023-11-17T05:36:46.274 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:46.269+0000 7f08f1ffb700 1 -- 172.21.15.160:0/1820918209 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f08ec025070 con 0x7f08fc0f5410 2023-11-17T05:36:46.274 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:46.269+0000 7f09036e0700 1 -- 172.21.15.160:0/1820918209 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08fc045ad0 con 0x7f08fc0f5410 2023-11-17T05:36:46.274 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:46.269+0000 7f08f1ffb700 1 --2- 172.21.15.160:0/1820918209 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f08e802ff60 0x7f08e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:36:46.274 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:46.269+0000 7f08f1ffb700 1 -- 172.21.15.160:0/1820918209 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f08ec0409a0 con 0x7f08fc0f5410 2023-11-17T05:36:46.275 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:46.269+0000 7f0900c7b700 1 --2- 172.21.15.160:0/1820918209 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f08e802ff60 0x7f08e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:36:46.275 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:46.270+0000 7f0900c7b700 1 --2- 172.21.15.160:0/1820918209 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f08e802ff60 0x7f08e8032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f08f80096a0 tx=0x7f08f8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:36:46.275 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:46.274+0000 7f08f1ffb700 1 -- 172.21.15.160:0/1820918209 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f08ec00f4e0 con 0x7f08fc0f5410 2023-11-17T05:36:46.479 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:46.479+0000 7f09036e0700 1 -- 172.21.15.160:0/1820918209 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f08fc0fae60 con 0x7f08fc0f5410 2023-11-17T05:36:46.480 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:46.479+0000 7f08f1ffb700 1 -- 172.21.15.160:0/1820918209 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f08ec00f300 con 0x7f08fc0f5410 2023-11-17T05:36:46.480 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:36:46.480 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:36:46.482 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:46.481+0000 7f09036e0700 1 -- 172.21.15.160:0/1820918209 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f08e802ff60 msgr2=0x7f08e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:36:46.482 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:46.481+0000 7f09036e0700 1 --2- 172.21.15.160:0/1820918209 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f08e802ff60 0x7f08e8032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f08f80096a0 tx=0x7f08f8006b40).stop 2023-11-17T05:36:46.482 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:46.481+0000 7f09036e0700 1 -- 172.21.15.160:0/1820918209 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08fc0f5410 msgr2=0x7f08fc109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:36:46.482 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:46.481+0000 7f09036e0700 1 --2- 172.21.15.160:0/1820918209 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08fc0f5410 0x7f08fc109430 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f08ec014000 tx=0x7f08ec00b040).stop 2023-11-17T05:36:46.482 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:46.481+0000 7f09036e0700 1 -- 172.21.15.160:0/1820918209 shutdown_connections 2023-11-17T05:36:46.482 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:46.481+0000 7f09036e0700 1 --2- 172.21.15.160:0/1820918209 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f08e802ff60 0x7f08e8032420 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:36:46.482 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:46.481+0000 7f09036e0700 1 --2- 172.21.15.160:0/1820918209 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08fc0f5410 0x7f08fc109430 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:36:46.482 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:46.481+0000 7f09036e0700 1 -- 172.21.15.160:0/1820918209 >> 172.21.15.160:0/1820918209 conn(0x7f08fc0f0b70 msgr2=0x7f08fc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:36:46.482 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:46.481+0000 7f09036e0700 1 -- 172.21.15.160:0/1820918209 shutdown_connections 2023-11-17T05:36:46.483 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:46.481+0000 7f09036e0700 1 -- 172.21.15.160:0/1820918209 wait complete. 2023-11-17T05:36:46.483 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:36:47.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:36:47 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:36:46.367577+0000 mgr.smithi003.eddcuc (mgr.14164) 350 : cluster [DBG] pgmap v334: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:47.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:36:47 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:36:46.480573+0000 mon.smithi003 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.160:0/1820918209' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:36:47.918 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:36:47.918 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:36:49.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:49.081+0000 7fd8613f4700 1 -- 172.21.15.160:0/3446640322 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd85c0f5410 msgr2=0x7fd85c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:36:49.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:49.081+0000 7fd8613f4700 1 --2- 172.21.15.160:0/3446640322 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd85c0f5410 0x7fd85c0f5830 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7fd850009fb0 tx=0x7fd850009b30).stop 2023-11-17T05:36:49.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:49.081+0000 7fd8613f4700 1 -- 172.21.15.160:0/3446640322 shutdown_connections 2023-11-17T05:36:49.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:49.081+0000 7fd8613f4700 1 --2- 172.21.15.160:0/3446640322 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd85c0f5410 0x7fd85c0f5830 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:36:49.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:49.081+0000 7fd8613f4700 1 -- 172.21.15.160:0/3446640322 >> 172.21.15.160:0/3446640322 conn(0x7fd85c0f0b70 msgr2=0x7fd85c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:36:49.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:49.081+0000 7fd8613f4700 1 -- 172.21.15.160:0/3446640322 shutdown_connections 2023-11-17T05:36:49.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:49.081+0000 7fd8613f4700 1 -- 172.21.15.160:0/3446640322 wait complete. 2023-11-17T05:36:49.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:49.082+0000 7fd8613f4700 1 Processor -- start 2023-11-17T05:36:49.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:49.082+0000 7fd8613f4700 1 -- start start 2023-11-17T05:36:49.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:49.082+0000 7fd8613f4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd85c0f5410 0x7fd85c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:36:49.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:49.082+0000 7fd8613f4700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd85c105b20 con 0x7fd85c0f5410 2023-11-17T05:36:49.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:49.082+0000 7fd85affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd85c0f5410 0x7fd85c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:36:49.091 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:49.082+0000 7fd85affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd85c0f5410 0x7fd85c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:55982/0 (socket says 172.21.15.160:55982) 2023-11-17T05:36:49.091 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:49.082+0000 7fd85affd700 1 -- 172.21.15.160:0/1467900691 learned_addr learned my addr 172.21.15.160:0/1467900691 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:36:49.091 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:49.083+0000 7fd85affd700 1 -- 172.21.15.160:0/1467900691 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd850005040 con 0x7fd85c0f5410 2023-11-17T05:36:49.091 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:49.083+0000 7fd85affd700 1 --2- 172.21.15.160:0/1467900691 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd85c0f5410 0x7fd85c105630 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fd850013510 tx=0x7fd85000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:36:49.091 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:49.083+0000 7fd8437fe700 1 -- 172.21.15.160:0/1467900691 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd850009320 con 0x7fd85c0f5410 2023-11-17T05:36:49.091 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:49.083+0000 7fd8613f4700 1 -- 172.21.15.160:0/1467900691 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd85c105d20 con 0x7fd85c0f5410 2023-11-17T05:36:49.091 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:49.083+0000 7fd8437fe700 1 -- 172.21.15.160:0/1467900691 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd850013f00 con 0x7fd85c0f5410 2023-11-17T05:36:49.091 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:49.083+0000 7fd8437fe700 1 -- 172.21.15.160:0/1467900691 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd850014d90 con 0x7fd85c0f5410 2023-11-17T05:36:49.091 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:49.083+0000 7fd8613f4700 1 -- 172.21.15.160:0/1467900691 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd85c1061b0 con 0x7fd85c0f5410 2023-11-17T05:36:49.091 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:49.084+0000 7fd8437fe700 1 -- 172.21.15.160:0/1467900691 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd8500095f0 con 0x7fd85c0f5410 2023-11-17T05:36:49.091 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:49.084+0000 7fd8613f4700 1 -- 172.21.15.160:0/1467900691 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd85c045ad0 con 0x7fd85c0f5410 2023-11-17T05:36:49.091 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:49.084+0000 7fd8437fe700 1 --2- 172.21.15.160:0/1467900691 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fd84402ff60 0x7fd844032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:36:49.092 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:49.084+0000 7fd8437fe700 1 -- 172.21.15.160:0/1467900691 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd850040fe0 con 0x7fd85c0f5410 2023-11-17T05:36:49.092 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:49.084+0000 7fd85a7fc700 1 --2- 172.21.15.160:0/1467900691 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fd84402ff60 0x7fd844032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:36:49.092 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:49.085+0000 7fd85a7fc700 1 --2- 172.21.15.160:0/1467900691 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fd84402ff60 0x7fd844032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fd84c0096a0 tx=0x7fd84c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:36:49.092 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:49.089+0000 7fd8437fe700 1 -- 172.21.15.160:0/1467900691 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd850014420 con 0x7fd85c0f5410 2023-11-17T05:36:49.296 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:49.295+0000 7fd8613f4700 1 -- 172.21.15.160:0/1467900691 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd85c0fae60 con 0x7fd85c0f5410 2023-11-17T05:36:49.296 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:49.296+0000 7fd8437fe700 1 -- 172.21.15.160:0/1467900691 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fd850010230 con 0x7fd85c0f5410 2023-11-17T05:36:49.296 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:36:49.297 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:36:49.298 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:49.297+0000 7fd8613f4700 1 -- 172.21.15.160:0/1467900691 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fd84402ff60 msgr2=0x7fd844032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:36:49.298 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:49.297+0000 7fd8613f4700 1 --2- 172.21.15.160:0/1467900691 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fd84402ff60 0x7fd844032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fd84c0096a0 tx=0x7fd84c006b40).stop 2023-11-17T05:36:49.298 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:49.297+0000 7fd8613f4700 1 -- 172.21.15.160:0/1467900691 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd85c0f5410 msgr2=0x7fd85c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:36:49.298 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:49.297+0000 7fd8613f4700 1 --2- 172.21.15.160:0/1467900691 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd85c0f5410 0x7fd85c105630 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fd850013510 tx=0x7fd85000b040).stop 2023-11-17T05:36:49.298 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:49.297+0000 7fd8613f4700 1 -- 172.21.15.160:0/1467900691 shutdown_connections 2023-11-17T05:36:49.298 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:49.297+0000 7fd8613f4700 1 --2- 172.21.15.160:0/1467900691 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fd84402ff60 0x7fd844032420 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:36:49.299 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:49.297+0000 7fd8613f4700 1 --2- 172.21.15.160:0/1467900691 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd85c0f5410 0x7fd85c105630 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:36:49.299 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:49.297+0000 7fd8613f4700 1 -- 172.21.15.160:0/1467900691 >> 172.21.15.160:0/1467900691 conn(0x7fd85c0f0b70 msgr2=0x7fd85c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:36:49.299 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:49.297+0000 7fd8613f4700 1 -- 172.21.15.160:0/1467900691 shutdown_connections 2023-11-17T05:36:49.299 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:49.297+0000 7fd8613f4700 1 -- 172.21.15.160:0/1467900691 wait complete. 2023-11-17T05:36:49.299 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:36:49.774 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:36:49 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:36:48.367825+0000 mgr.smithi003.eddcuc (mgr.14164) 351 : cluster [DBG] pgmap v335: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:49.775 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:36:49 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:36:49.296765+0000 mon.smithi003 (mon.0) 288 : audit [DBG] from='client.? 172.21.15.160:0/1467900691' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:36:50.700 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:36:50.701 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:36:51.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:36:51 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:36:50.368179+0000 mgr.smithi003.eddcuc (mgr.14164) 352 : cluster [DBG] pgmap v336: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:52.034 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:52.032+0000 7faf2f352700 1 -- 172.21.15.160:0/4000926313 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf280f6510 msgr2=0x7faf280f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:36:52.034 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:52.032+0000 7faf2f352700 1 --2- 172.21.15.160:0/4000926313 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf280f6510 0x7faf280f6930 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7faf18009fb0 tx=0x7faf18009b30).stop 2023-11-17T05:36:52.034 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:52.032+0000 7faf2f352700 1 -- 172.21.15.160:0/4000926313 shutdown_connections 2023-11-17T05:36:52.034 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:52.032+0000 7faf2f352700 1 --2- 172.21.15.160:0/4000926313 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf280f6510 0x7faf280f6930 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:36:52.034 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:52.032+0000 7faf2f352700 1 -- 172.21.15.160:0/4000926313 >> 172.21.15.160:0/4000926313 conn(0x7faf280f1c70 msgr2=0x7faf280f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:36:52.034 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:52.032+0000 7faf2f352700 1 -- 172.21.15.160:0/4000926313 shutdown_connections 2023-11-17T05:36:52.034 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:52.032+0000 7faf2f352700 1 -- 172.21.15.160:0/4000926313 wait complete. 2023-11-17T05:36:52.034 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:52.033+0000 7faf2f352700 1 Processor -- start 2023-11-17T05:36:52.034 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:52.033+0000 7faf2f352700 1 -- start start 2023-11-17T05:36:52.035 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:52.033+0000 7faf2f352700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf280f6510 0x7faf281066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:36:52.035 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:52.033+0000 7faf2f352700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf28106bc0 con 0x7faf280f6510 2023-11-17T05:36:52.035 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:52.033+0000 7faf2d0ee700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf280f6510 0x7faf281066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:36:52.035 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:52.033+0000 7faf2d0ee700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf280f6510 0x7faf281066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:56314/0 (socket says 172.21.15.160:56314) 2023-11-17T05:36:52.035 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:52.033+0000 7faf2d0ee700 1 -- 172.21.15.160:0/1649501052 learned_addr learned my addr 172.21.15.160:0/1649501052 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:36:52.036 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:52.033+0000 7faf2d0ee700 1 -- 172.21.15.160:0/1649501052 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf18005040 con 0x7faf280f6510 2023-11-17T05:36:52.036 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:52.034+0000 7faf2d0ee700 1 --2- 172.21.15.160:0/1649501052 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf280f6510 0x7faf281066d0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7faf18013510 tx=0x7faf1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:36:52.036 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:52.034+0000 7faf1dffb700 1 -- 172.21.15.160:0/1649501052 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf18009320 con 0x7faf280f6510 2023-11-17T05:36:52.036 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:52.034+0000 7faf2f352700 1 -- 172.21.15.160:0/1649501052 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf28106dc0 con 0x7faf280f6510 2023-11-17T05:36:52.037 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:52.034+0000 7faf1dffb700 1 -- 172.21.15.160:0/1649501052 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faf18013f00 con 0x7faf280f6510 2023-11-17T05:36:52.037 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:52.034+0000 7faf1dffb700 1 -- 172.21.15.160:0/1649501052 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf18014d90 con 0x7faf280f6510 2023-11-17T05:36:52.037 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:52.034+0000 7faf2f352700 1 -- 172.21.15.160:0/1649501052 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf28107250 con 0x7faf280f6510 2023-11-17T05:36:52.037 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:52.035+0000 7faf1dffb700 1 -- 172.21.15.160:0/1649501052 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faf180263a0 con 0x7faf280f6510 2023-11-17T05:36:52.037 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:52.035+0000 7faf1dffb700 1 --2- 172.21.15.160:0/1649501052 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7faf1402ffb0 0x7faf14032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:36:52.037 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:52.035+0000 7faf1dffb700 1 -- 172.21.15.160:0/1649501052 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faf180410c0 con 0x7faf280f6510 2023-11-17T05:36:52.037 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:52.035+0000 7faf2f352700 1 -- 172.21.15.160:0/1649501052 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf28045ad0 con 0x7faf280f6510 2023-11-17T05:36:52.037 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:52.035+0000 7faf2c8ed700 1 --2- 172.21.15.160:0/1649501052 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7faf1402ffb0 0x7faf14032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:36:52.037 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:52.036+0000 7faf2c8ed700 1 --2- 172.21.15.160:0/1649501052 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7faf1402ffb0 0x7faf14032470 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7faf240096a0 tx=0x7faf24006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:36:52.041 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:52.040+0000 7faf1dffb700 1 -- 172.21.15.160:0/1649501052 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faf18012920 con 0x7faf280f6510 2023-11-17T05:36:52.246 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:52.245+0000 7faf2f352700 1 -- 172.21.15.160:0/1649501052 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faf28107f90 con 0x7faf280f6510 2023-11-17T05:36:52.246 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:52.245+0000 7faf1dffb700 1 -- 172.21.15.160:0/1649501052 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7faf1800f340 con 0x7faf280f6510 2023-11-17T05:36:52.246 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:36:52.247 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:36:52.248 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:52.247+0000 7faf2f352700 1 -- 172.21.15.160:0/1649501052 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7faf1402ffb0 msgr2=0x7faf14032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:36:52.248 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:52.247+0000 7faf2f352700 1 --2- 172.21.15.160:0/1649501052 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7faf1402ffb0 0x7faf14032470 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7faf240096a0 tx=0x7faf24006b40).stop 2023-11-17T05:36:52.248 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:52.247+0000 7faf2f352700 1 -- 172.21.15.160:0/1649501052 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf280f6510 msgr2=0x7faf281066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:36:52.248 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:52.247+0000 7faf2f352700 1 --2- 172.21.15.160:0/1649501052 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf280f6510 0x7faf281066d0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7faf18013510 tx=0x7faf1800b040).stop 2023-11-17T05:36:52.248 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:52.247+0000 7faf2f352700 1 -- 172.21.15.160:0/1649501052 shutdown_connections 2023-11-17T05:36:52.248 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:52.247+0000 7faf2f352700 1 --2- 172.21.15.160:0/1649501052 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7faf1402ffb0 0x7faf14032470 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:36:52.248 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:52.247+0000 7faf2f352700 1 --2- 172.21.15.160:0/1649501052 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf280f6510 0x7faf281066d0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:36:52.249 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:52.247+0000 7faf2f352700 1 -- 172.21.15.160:0/1649501052 >> 172.21.15.160:0/1649501052 conn(0x7faf280f1c70 msgr2=0x7faf280f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:36:52.249 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:52.247+0000 7faf2f352700 1 -- 172.21.15.160:0/1649501052 shutdown_connections 2023-11-17T05:36:52.249 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:52.247+0000 7faf2f352700 1 -- 172.21.15.160:0/1649501052 wait complete. 2023-11-17T05:36:52.249 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:36:52.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:36:52 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:36:52.246675+0000 mon.smithi003 (mon.0) 289 : audit [DBG] from='client.? 172.21.15.160:0/1649501052' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:36:53.644 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:36:53.646 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:36:53.771 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:36:53 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:36:52.368523+0000 mgr.smithi003.eddcuc (mgr.14164) 353 : cluster [DBG] pgmap v337: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:54.911 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:54.909+0000 7f169e72a700 1 -- 172.21.15.160:0/250025456 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16980f51a0 msgr2=0x7f16980f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:36:54.912 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:54.909+0000 7f169e72a700 1 --2- 172.21.15.160:0/250025456 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16980f51a0 0x7f16980f55c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f1694009fb0 tx=0x7f1694009b30).stop 2023-11-17T05:36:54.912 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:54.909+0000 7f169e72a700 1 -- 172.21.15.160:0/250025456 shutdown_connections 2023-11-17T05:36:54.912 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:54.909+0000 7f169e72a700 1 --2- 172.21.15.160:0/250025456 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16980f51a0 0x7f16980f55c0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:36:54.912 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:54.909+0000 7f169e72a700 1 -- 172.21.15.160:0/250025456 >> 172.21.15.160:0/250025456 conn(0x7f16980f09a0 msgr2=0x7f16980f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:36:54.912 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:54.909+0000 7f169e72a700 1 -- 172.21.15.160:0/250025456 shutdown_connections 2023-11-17T05:36:54.912 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:54.909+0000 7f169e72a700 1 -- 172.21.15.160:0/250025456 wait complete. 2023-11-17T05:36:54.912 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:54.909+0000 7f169e72a700 1 Processor -- start 2023-11-17T05:36:54.912 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:54.909+0000 7f169e72a700 1 -- start start 2023-11-17T05:36:54.912 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:54.910+0000 7f169e72a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16980f51a0 0x7f16981053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:36:54.912 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:54.910+0000 7f169e72a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16981058a0 con 0x7f16980f51a0 2023-11-17T05:36:54.912 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:54.910+0000 7f169d728700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16980f51a0 0x7f16981053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:36:54.913 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:54.910+0000 7f169d728700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16980f51a0 0x7f16981053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:56332/0 (socket says 172.21.15.160:56332) 2023-11-17T05:36:54.913 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:54.910+0000 7f169d728700 1 -- 172.21.15.160:0/1035591309 learned_addr learned my addr 172.21.15.160:0/1035591309 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:36:54.913 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:54.910+0000 7f169d728700 1 -- 172.21.15.160:0/1035591309 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1694005040 con 0x7f16980f51a0 2023-11-17T05:36:54.913 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:54.910+0000 7f169d728700 1 --2- 172.21.15.160:0/1035591309 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16980f51a0 0x7f16981053b0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f1694013510 tx=0x7f169400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:36:54.913 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:54.910+0000 7f168dffb700 1 -- 172.21.15.160:0/1035591309 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1694009320 con 0x7f16980f51a0 2023-11-17T05:36:54.913 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:54.910+0000 7f169e72a700 1 -- 172.21.15.160:0/1035591309 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1698105aa0 con 0x7f16980f51a0 2023-11-17T05:36:54.913 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:54.910+0000 7f168dffb700 1 -- 172.21.15.160:0/1035591309 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1694013f00 con 0x7f16980f51a0 2023-11-17T05:36:54.913 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:54.910+0000 7f168dffb700 1 -- 172.21.15.160:0/1035591309 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1694014dc0 con 0x7f16980f51a0 2023-11-17T05:36:54.913 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:54.911+0000 7f169e72a700 1 -- 172.21.15.160:0/1035591309 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1698105f30 con 0x7f16980f51a0 2023-11-17T05:36:54.914 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:54.911+0000 7f168dffb700 1 -- 172.21.15.160:0/1035591309 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f16940095f0 con 0x7f16980f51a0 2023-11-17T05:36:54.914 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:54.911+0000 7f169e72a700 1 -- 172.21.15.160:0/1035591309 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1698045ad0 con 0x7f16980f51a0 2023-11-17T05:36:54.914 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:54.912+0000 7f168dffb700 1 --2- 172.21.15.160:0/1035591309 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f168402ff60 0x7f1684032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:36:54.914 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:54.912+0000 7f168dffb700 1 -- 172.21.15.160:0/1035591309 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1694042e90 con 0x7f16980f51a0 2023-11-17T05:36:54.914 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:54.912+0000 7f169cf27700 1 --2- 172.21.15.160:0/1035591309 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f168402ff60 0x7f1684032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:36:54.914 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:54.912+0000 7f169cf27700 1 --2- 172.21.15.160:0/1035591309 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f168402ff60 0x7f1684032420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f16880096a0 tx=0x7f1688006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:36:55.011 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:54.916+0000 7f168dffb700 1 -- 172.21.15.160:0/1035591309 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1694014420 con 0x7f16980f51a0 2023-11-17T05:36:55.120 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:55.119+0000 7f169e72a700 1 -- 172.21.15.160:0/1035591309 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1698106d80 con 0x7f16980f51a0 2023-11-17T05:36:55.121 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:55.120+0000 7f168dffb700 1 -- 172.21.15.160:0/1035591309 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f169401f980 con 0x7f16980f51a0 2023-11-17T05:36:55.121 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:36:55.121 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:36:55.122 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:55.121+0000 7f169e72a700 1 -- 172.21.15.160:0/1035591309 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f168402ff60 msgr2=0x7f1684032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:36:55.122 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:55.122+0000 7f169e72a700 1 --2- 172.21.15.160:0/1035591309 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f168402ff60 0x7f1684032420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f16880096a0 tx=0x7f1688006b40).stop 2023-11-17T05:36:55.122 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:55.122+0000 7f169e72a700 1 -- 172.21.15.160:0/1035591309 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16980f51a0 msgr2=0x7f16981053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:36:55.123 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:55.122+0000 7f169e72a700 1 --2- 172.21.15.160:0/1035591309 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16980f51a0 0x7f16981053b0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f1694013510 tx=0x7f169400b040).stop 2023-11-17T05:36:55.123 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:55.122+0000 7f169e72a700 1 -- 172.21.15.160:0/1035591309 shutdown_connections 2023-11-17T05:36:55.123 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:55.122+0000 7f169e72a700 1 --2- 172.21.15.160:0/1035591309 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f168402ff60 0x7f1684032420 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:36:55.123 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:55.122+0000 7f169e72a700 1 --2- 172.21.15.160:0/1035591309 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16980f51a0 0x7f16981053b0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:36:55.123 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:55.122+0000 7f169e72a700 1 -- 172.21.15.160:0/1035591309 >> 172.21.15.160:0/1035591309 conn(0x7f16980f09a0 msgr2=0x7f16980f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:36:55.123 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:55.122+0000 7f169e72a700 1 -- 172.21.15.160:0/1035591309 shutdown_connections 2023-11-17T05:36:55.123 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:55.122+0000 7f169e72a700 1 -- 172.21.15.160:0/1035591309 wait complete. 2023-11-17T05:36:55.123 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:36:55.274 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:36:55 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:36:54.368761+0000 mgr.smithi003.eddcuc (mgr.14164) 354 : cluster [DBG] pgmap v338: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:56.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:36:56 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:36:55.121144+0000 mon.smithi003 (mon.0) 290 : audit [DBG] from='client.? 172.21.15.160:0/1035591309' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:36:56.484 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:36:56.484 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:36:57.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:36:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:36:56.369192+0000 mgr.smithi003.eddcuc (mgr.14164) 355 : cluster [DBG] pgmap v339: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:36:57.676 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:57.675+0000 7fa98e682700 1 -- 172.21.15.160:0/4226811914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9880f47c0 msgr2=0x7fa9880f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:36:57.677 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:57.675+0000 7fa98e682700 1 --2- 172.21.15.160:0/4226811914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9880f47c0 0x7fa9880f4be0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7fa978009fb0 tx=0x7fa978009b30).stop 2023-11-17T05:36:57.677 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:57.675+0000 7fa98e682700 1 -- 172.21.15.160:0/4226811914 shutdown_connections 2023-11-17T05:36:57.677 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:57.675+0000 7fa98e682700 1 --2- 172.21.15.160:0/4226811914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9880f47c0 0x7fa9880f4be0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:36:57.677 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:57.675+0000 7fa98e682700 1 -- 172.21.15.160:0/4226811914 >> 172.21.15.160:0/4226811914 conn(0x7fa9880f02e0 msgr2=0x7fa9880f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:36:57.677 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:57.675+0000 7fa98e682700 1 -- 172.21.15.160:0/4226811914 shutdown_connections 2023-11-17T05:36:57.677 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:57.675+0000 7fa98e682700 1 -- 172.21.15.160:0/4226811914 wait complete. 2023-11-17T05:36:57.677 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:57.676+0000 7fa98e682700 1 Processor -- start 2023-11-17T05:36:57.678 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:57.676+0000 7fa98e682700 1 -- start start 2023-11-17T05:36:57.678 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:57.676+0000 7fa98e682700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9880f47c0 0x7fa988106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:36:57.678 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:57.676+0000 7fa98e682700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa988106bf0 con 0x7fa9880f47c0 2023-11-17T05:36:57.679 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:57.677+0000 7fa987fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9880f47c0 0x7fa988106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:36:57.679 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:57.677+0000 7fa987fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9880f47c0 0x7fa988106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:56348/0 (socket says 172.21.15.160:56348) 2023-11-17T05:36:57.679 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:57.677+0000 7fa987fff700 1 -- 172.21.15.160:0/98977365 learned_addr learned my addr 172.21.15.160:0/98977365 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:36:57.679 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:57.677+0000 7fa987fff700 1 -- 172.21.15.160:0/98977365 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa978005040 con 0x7fa9880f47c0 2023-11-17T05:36:57.679 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:57.677+0000 7fa987fff700 1 --2- 172.21.15.160:0/98977365 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9880f47c0 0x7fa988106700 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7fa978013510 tx=0x7fa97800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:36:57.679 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:57.677+0000 7fa984ff9700 1 -- 172.21.15.160:0/98977365 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa978009320 con 0x7fa9880f47c0 2023-11-17T05:36:57.679 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:57.677+0000 7fa98e682700 1 -- 172.21.15.160:0/98977365 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa988106df0 con 0x7fa9880f47c0 2023-11-17T05:36:57.679 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:57.677+0000 7fa984ff9700 1 -- 172.21.15.160:0/98977365 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa978013f00 con 0x7fa9880f47c0 2023-11-17T05:36:57.679 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:57.677+0000 7fa984ff9700 1 -- 172.21.15.160:0/98977365 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa978014d90 con 0x7fa9880f47c0 2023-11-17T05:36:57.679 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:57.677+0000 7fa98e682700 1 -- 172.21.15.160:0/98977365 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa988107280 con 0x7fa9880f47c0 2023-11-17T05:36:57.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:57.678+0000 7fa984ff9700 1 -- 172.21.15.160:0/98977365 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa9780263a0 con 0x7fa9880f47c0 2023-11-17T05:36:57.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:57.678+0000 7fa984ff9700 1 --2- 172.21.15.160:0/98977365 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa97402ff60 0x7fa974032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:36:57.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:57.678+0000 7fa98e682700 1 -- 172.21.15.160:0/98977365 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa988045ad0 con 0x7fa9880f47c0 2023-11-17T05:36:57.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:57.678+0000 7fa984ff9700 1 -- 172.21.15.160:0/98977365 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa978041020 con 0x7fa9880f47c0 2023-11-17T05:36:57.680 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:57.679+0000 7fa9877fe700 1 --2- 172.21.15.160:0/98977365 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa97402ff60 0x7fa974032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:36:57.681 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:57.679+0000 7fa9877fe700 1 --2- 172.21.15.160:0/98977365 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa97402ff60 0x7fa974032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fa9700096a0 tx=0x7fa970006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:36:57.684 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:57.683+0000 7fa984ff9700 1 -- 172.21.15.160:0/98977365 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa978012920 con 0x7fa9880f47c0 2023-11-17T05:36:57.893 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:57.889+0000 7fa98e682700 1 -- 172.21.15.160:0/98977365 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa9881080d0 con 0x7fa9880f47c0 2023-11-17T05:36:57.893 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:57.890+0000 7fa984ff9700 1 -- 172.21.15.160:0/98977365 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fa97800f340 con 0x7fa9880f47c0 2023-11-17T05:36:57.893 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:36:57.894 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:36:57.894 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:57.891+0000 7fa98e682700 1 -- 172.21.15.160:0/98977365 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa97402ff60 msgr2=0x7fa974032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:36:57.894 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:57.891+0000 7fa98e682700 1 --2- 172.21.15.160:0/98977365 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa97402ff60 0x7fa974032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fa9700096a0 tx=0x7fa970006b40).stop 2023-11-17T05:36:57.894 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:57.891+0000 7fa98e682700 1 -- 172.21.15.160:0/98977365 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9880f47c0 msgr2=0x7fa988106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:36:57.894 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:57.891+0000 7fa98e682700 1 --2- 172.21.15.160:0/98977365 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9880f47c0 0x7fa988106700 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7fa978013510 tx=0x7fa97800b040).stop 2023-11-17T05:36:57.894 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:57.891+0000 7fa98e682700 1 -- 172.21.15.160:0/98977365 shutdown_connections 2023-11-17T05:36:57.895 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:57.891+0000 7fa98e682700 1 --2- 172.21.15.160:0/98977365 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa97402ff60 0x7fa974032420 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:36:57.895 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:57.891+0000 7fa98e682700 1 --2- 172.21.15.160:0/98977365 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9880f47c0 0x7fa988106700 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:36:57.895 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:57.891+0000 7fa98e682700 1 -- 172.21.15.160:0/98977365 >> 172.21.15.160:0/98977365 conn(0x7fa9880f02e0 msgr2=0x7fa9880f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:36:57.895 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:57.891+0000 7fa98e682700 1 -- 172.21.15.160:0/98977365 shutdown_connections 2023-11-17T05:36:57.895 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:36:57.891+0000 7fa98e682700 1 -- 172.21.15.160:0/98977365 wait complete. 2023-11-17T05:36:57.895 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:36:58.520 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:36:58 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:36:57.890825+0000 mon.smithi003 (mon.0) 291 : audit [DBG] from='client.? 172.21.15.160:0/98977365' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:36:59.284 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:36:59.285 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:36:59.520 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:36:59 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:36:58.369545+0000 mgr.smithi003.eddcuc (mgr.14164) 356 : cluster [DBG] pgmap v340: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:37:00.560 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:00.559+0000 7fdcc1c71700 1 -- 172.21.15.160:0/2061628066 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcbc0f51a0 msgr2=0x7fdcbc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:00.560 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:00.559+0000 7fdcc1c71700 1 --2- 172.21.15.160:0/2061628066 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcbc0f51a0 0x7fdcbc0f55c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7fdcb0009fb0 tx=0x7fdcb0009b30).stop 2023-11-17T05:37:00.560 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:00.559+0000 7fdcc1c71700 1 -- 172.21.15.160:0/2061628066 shutdown_connections 2023-11-17T05:37:00.560 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:00.559+0000 7fdcc1c71700 1 --2- 172.21.15.160:0/2061628066 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcbc0f51a0 0x7fdcbc0f55c0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:00.560 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:00.559+0000 7fdcc1c71700 1 -- 172.21.15.160:0/2061628066 >> 172.21.15.160:0/2061628066 conn(0x7fdcbc0f09a0 msgr2=0x7fdcbc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:37:00.560 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:00.559+0000 7fdcc1c71700 1 -- 172.21.15.160:0/2061628066 shutdown_connections 2023-11-17T05:37:00.560 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:00.559+0000 7fdcc1c71700 1 -- 172.21.15.160:0/2061628066 wait complete. 2023-11-17T05:37:00.561 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:00.559+0000 7fdcc1c71700 1 Processor -- start 2023-11-17T05:37:00.561 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:00.559+0000 7fdcc1c71700 1 -- start start 2023-11-17T05:37:00.562 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:00.560+0000 7fdcc1c71700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcbc0f51a0 0x7fdcbc0fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:37:00.562 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:00.560+0000 7fdcc1c71700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcbc0fb5a0 con 0x7fdcbc0f51a0 2023-11-17T05:37:00.562 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:00.560+0000 7fdcc0c6f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcbc0f51a0 0x7fdcbc0fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:37:00.562 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:00.560+0000 7fdcc0c6f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcbc0f51a0 0x7fdcbc0fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:51974/0 (socket says 172.21.15.160:51974) 2023-11-17T05:37:00.562 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:00.560+0000 7fdcc0c6f700 1 -- 172.21.15.160:0/3698339204 learned_addr learned my addr 172.21.15.160:0/3698339204 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:37:00.563 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:00.560+0000 7fdcc0c6f700 1 -- 172.21.15.160:0/3698339204 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb0005040 con 0x7fdcbc0f51a0 2023-11-17T05:37:00.563 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:00.560+0000 7fdcc0c6f700 1 --2- 172.21.15.160:0/3698339204 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcbc0f51a0 0x7fdcbc0fd120 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7fdcb0013510 tx=0x7fdcb000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:37:00.563 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:00.560+0000 7fdcb97fa700 1 -- 172.21.15.160:0/3698339204 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdcb0009320 con 0x7fdcbc0f51a0 2023-11-17T05:37:00.563 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:00.560+0000 7fdcc1c71700 1 -- 172.21.15.160:0/3698339204 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcbc0fb740 con 0x7fdcbc0f51a0 2023-11-17T05:37:00.563 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:00.560+0000 7fdcb97fa700 1 -- 172.21.15.160:0/3698339204 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdcb0013f00 con 0x7fdcbc0f51a0 2023-11-17T05:37:00.563 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:00.560+0000 7fdcb97fa700 1 -- 172.21.15.160:0/3698339204 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdcb0014dc0 con 0x7fdcbc0f51a0 2023-11-17T05:37:00.563 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:00.560+0000 7fdcc1c71700 1 -- 172.21.15.160:0/3698339204 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcbc0fbbd0 con 0x7fdcbc0f51a0 2023-11-17T05:37:00.563 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:00.561+0000 7fdcb97fa700 1 -- 172.21.15.160:0/3698339204 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdcb00095f0 con 0x7fdcbc0f51a0 2023-11-17T05:37:00.563 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:00.561+0000 7fdcc1c71700 1 -- 172.21.15.160:0/3698339204 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcbc059100 con 0x7fdcbc0f51a0 2023-11-17T05:37:00.563 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:00.561+0000 7fdcb97fa700 1 --2- 172.21.15.160:0/3698339204 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fdcac02ff60 0x7fdcac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:37:00.563 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:00.561+0000 7fdcb97fa700 1 -- 172.21.15.160:0/3698339204 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdcb00410c0 con 0x7fdcbc0f51a0 2023-11-17T05:37:00.563 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:00.562+0000 7fdcbbfff700 1 --2- 172.21.15.160:0/3698339204 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fdcac02ff60 0x7fdcac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:37:00.564 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:00.562+0000 7fdcbbfff700 1 --2- 172.21.15.160:0/3698339204 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fdcac02ff60 0x7fdcac032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fdca80096a0 tx=0x7fdca8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:37:00.567 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:00.567+0000 7fdcb97fa700 1 -- 172.21.15.160:0/3698339204 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdcb0014420 con 0x7fdcbc0f51a0 2023-11-17T05:37:00.771 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:00.771+0000 7fdcc1c71700 1 -- 172.21.15.160:0/3698339204 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdcbc0fabe0 con 0x7fdcbc0f51a0 2023-11-17T05:37:00.772 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:00.772+0000 7fdcb97fa700 1 -- 172.21.15.160:0/3698339204 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fdcb0010230 con 0x7fdcbc0f51a0 2023-11-17T05:37:00.772 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:37:00.772 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:37:00.774 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:00.773+0000 7fdcc1c71700 1 -- 172.21.15.160:0/3698339204 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fdcac02ff60 msgr2=0x7fdcac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:00.774 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:00.773+0000 7fdcc1c71700 1 --2- 172.21.15.160:0/3698339204 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fdcac02ff60 0x7fdcac032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fdca80096a0 tx=0x7fdca8006b40).stop 2023-11-17T05:37:00.774 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:00.773+0000 7fdcc1c71700 1 -- 172.21.15.160:0/3698339204 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcbc0f51a0 msgr2=0x7fdcbc0fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:00.775 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:00.773+0000 7fdcc1c71700 1 --2- 172.21.15.160:0/3698339204 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcbc0f51a0 0x7fdcbc0fd120 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7fdcb0013510 tx=0x7fdcb000b040).stop 2023-11-17T05:37:00.775 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:00.773+0000 7fdcc1c71700 1 -- 172.21.15.160:0/3698339204 shutdown_connections 2023-11-17T05:37:00.775 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:00.773+0000 7fdcc1c71700 1 --2- 172.21.15.160:0/3698339204 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fdcac02ff60 0x7fdcac032420 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:00.775 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:00.773+0000 7fdcc1c71700 1 --2- 172.21.15.160:0/3698339204 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdcbc0f51a0 0x7fdcbc0fd120 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:00.775 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:00.773+0000 7fdcc1c71700 1 -- 172.21.15.160:0/3698339204 >> 172.21.15.160:0/3698339204 conn(0x7fdcbc0f09a0 msgr2=0x7fdcbc05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:37:00.775 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:00.773+0000 7fdcc1c71700 1 -- 172.21.15.160:0/3698339204 shutdown_connections 2023-11-17T05:37:00.775 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:00.773+0000 7fdcc1c71700 1 -- 172.21.15.160:0/3698339204 wait complete. 2023-11-17T05:37:00.775 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:37:01.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:01 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:37:00.369862+0000 mgr.smithi003.eddcuc (mgr.14164) 357 : cluster [DBG] pgmap v341: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:37:01.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:01 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:37:00.772583+0000 mon.smithi003 (mon.0) 292 : audit [DBG] from='client.? 172.21.15.160:0/3698339204' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:37:02.156 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:37:02.157 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:37:03.392 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:03.371+0000 7f66408fa700 1 -- 172.21.15.160:0/2467376414 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f663c0f5410 msgr2=0x7f663c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:03.392 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:03.371+0000 7f66408fa700 1 --2- 172.21.15.160:0/2467376414 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f663c0f5410 0x7f663c0f5830 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f6624009fb0 tx=0x7f6624009b30).stop 2023-11-17T05:37:03.392 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:03.371+0000 7f66408fa700 1 -- 172.21.15.160:0/2467376414 shutdown_connections 2023-11-17T05:37:03.392 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:03.371+0000 7f66408fa700 1 --2- 172.21.15.160:0/2467376414 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f663c0f5410 0x7f663c0f5830 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:03.392 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:03.371+0000 7f66408fa700 1 -- 172.21.15.160:0/2467376414 >> 172.21.15.160:0/2467376414 conn(0x7f663c0f0b70 msgr2=0x7f663c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:37:03.392 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:03.371+0000 7f66408fa700 1 -- 172.21.15.160:0/2467376414 shutdown_connections 2023-11-17T05:37:03.392 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:03.371+0000 7f66408fa700 1 -- 172.21.15.160:0/2467376414 wait complete. 2023-11-17T05:37:03.392 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:03.372+0000 7f66408fa700 1 Processor -- start 2023-11-17T05:37:03.393 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:03.372+0000 7f66408fa700 1 -- start start 2023-11-17T05:37:03.393 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:03.372+0000 7f66408fa700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f663c0f5410 0x7f663c108910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:37:03.393 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:03.372+0000 7f66408fa700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f663c108e00 con 0x7f663c0f5410 2023-11-17T05:37:03.393 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:03.372+0000 7f663a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f663c0f5410 0x7f663c108910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:37:03.393 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:03.372+0000 7f663a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f663c0f5410 0x7f663c108910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:51998/0 (socket says 172.21.15.160:51998) 2023-11-17T05:37:03.393 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:03.372+0000 7f663a59c700 1 -- 172.21.15.160:0/1596146779 learned_addr learned my addr 172.21.15.160:0/1596146779 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:37:03.393 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:03.372+0000 7f663a59c700 1 -- 172.21.15.160:0/1596146779 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6624005040 con 0x7f663c0f5410 2023-11-17T05:37:03.393 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:03.373+0000 7f663a59c700 1 --2- 172.21.15.160:0/1596146779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f663c0f5410 0x7f663c108910 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f6624014000 tx=0x7f662400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:37:03.393 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:03.373+0000 7f6632ffd700 1 -- 172.21.15.160:0/1596146779 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66240095f0 con 0x7f663c0f5410 2023-11-17T05:37:03.393 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:03.373+0000 7f66408fa700 1 -- 172.21.15.160:0/1596146779 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f663c109000 con 0x7f663c0f5410 2023-11-17T05:37:03.393 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:03.373+0000 7f6632ffd700 1 -- 172.21.15.160:0/1596146779 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6624007e70 con 0x7f663c0f5410 2023-11-17T05:37:03.393 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:03.373+0000 7f6632ffd700 1 -- 172.21.15.160:0/1596146779 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6624014ea0 con 0x7f663c0f5410 2023-11-17T05:37:03.393 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:03.373+0000 7f66408fa700 1 -- 172.21.15.160:0/1596146779 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f663c109490 con 0x7f663c0f5410 2023-11-17T05:37:03.393 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:03.374+0000 7f6632ffd700 1 -- 172.21.15.160:0/1596146779 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6624025070 con 0x7f663c0f5410 2023-11-17T05:37:03.394 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:03.374+0000 7f66408fa700 1 -- 172.21.15.160:0/1596146779 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f663c045ad0 con 0x7f663c0f5410 2023-11-17T05:37:03.394 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:03.374+0000 7f6632ffd700 1 --2- 172.21.15.160:0/1596146779 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f662802ff60 0x7f6628032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:37:03.394 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:03.374+0000 7f6632ffd700 1 -- 172.21.15.160:0/1596146779 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6624040970 con 0x7f663c0f5410 2023-11-17T05:37:03.394 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:03.374+0000 7f6639d9b700 1 --2- 172.21.15.160:0/1596146779 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f662802ff60 0x7f6628032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:37:03.394 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:03.374+0000 7f6639d9b700 1 --2- 172.21.15.160:0/1596146779 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f662802ff60 0x7f6628032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f662c0096a0 tx=0x7f662c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:37:03.394 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:03.380+0000 7f6632ffd700 1 -- 172.21.15.160:0/1596146779 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f66240079c0 con 0x7f663c0f5410 2023-11-17T05:37:03.587 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:03.587+0000 7f66408fa700 1 -- 172.21.15.160:0/1596146779 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f663c0fae60 con 0x7f663c0f5410 2023-11-17T05:37:03.587 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:03.587+0000 7f6632ffd700 1 -- 172.21.15.160:0/1596146779 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f662400f300 con 0x7f663c0f5410 2023-11-17T05:37:03.587 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:37:03.587 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:37:03.589 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:03.589+0000 7f66408fa700 1 -- 172.21.15.160:0/1596146779 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f662802ff60 msgr2=0x7f6628032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:03.589 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:03.589+0000 7f66408fa700 1 --2- 172.21.15.160:0/1596146779 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f662802ff60 0x7f6628032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f662c0096a0 tx=0x7f662c006b40).stop 2023-11-17T05:37:03.589 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:03.589+0000 7f66408fa700 1 -- 172.21.15.160:0/1596146779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f663c0f5410 msgr2=0x7f663c108910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:03.589 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:03.589+0000 7f66408fa700 1 --2- 172.21.15.160:0/1596146779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f663c0f5410 0x7f663c108910 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f6624014000 tx=0x7f662400b040).stop 2023-11-17T05:37:03.589 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:03.589+0000 7f66408fa700 1 -- 172.21.15.160:0/1596146779 shutdown_connections 2023-11-17T05:37:03.589 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:03.589+0000 7f66408fa700 1 --2- 172.21.15.160:0/1596146779 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f662802ff60 0x7f6628032420 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:03.589 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:03.589+0000 7f66408fa700 1 --2- 172.21.15.160:0/1596146779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f663c0f5410 0x7f663c108910 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:03.589 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:03.589+0000 7f66408fa700 1 -- 172.21.15.160:0/1596146779 >> 172.21.15.160:0/1596146779 conn(0x7f663c0f0b70 msgr2=0x7f663c05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:37:03.589 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:03.589+0000 7f66408fa700 1 -- 172.21.15.160:0/1596146779 shutdown_connections 2023-11-17T05:37:03.589 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:03.589+0000 7f66408fa700 1 -- 172.21.15.160:0/1596146779 wait complete. 2023-11-17T05:37:03.590 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:37:03.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:03 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:37:02.370150+0000 mgr.smithi003.eddcuc (mgr.14164) 358 : cluster [DBG] pgmap v342: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:37:04.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:04 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:37:03.587620+0000 mon.smithi003 (mon.0) 293 : audit [DBG] from='client.? 172.21.15.160:0/1596146779' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:37:04.977 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:37:04.978 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:37:05.771 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:05 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:37:04.370554+0000 mgr.smithi003.eddcuc (mgr.14164) 359 : cluster [DBG] pgmap v343: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:37:06.158 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:06.157+0000 7f2940ce8700 1 -- 172.21.15.160:0/3128773328 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f293c0f6510 msgr2=0x7f293c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:06.158 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:06.157+0000 7f2940ce8700 1 --2- 172.21.15.160:0/3128773328 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f293c0f6510 0x7f293c0f6930 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f2930009fb0 tx=0x7f2930009b30).stop 2023-11-17T05:37:06.158 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:06.158+0000 7f2940ce8700 1 -- 172.21.15.160:0/3128773328 shutdown_connections 2023-11-17T05:37:06.158 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:06.158+0000 7f2940ce8700 1 --2- 172.21.15.160:0/3128773328 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f293c0f6510 0x7f293c0f6930 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:06.158 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:06.158+0000 7f2940ce8700 1 -- 172.21.15.160:0/3128773328 >> 172.21.15.160:0/3128773328 conn(0x7f293c0f1c70 msgr2=0x7f293c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:37:06.159 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:06.158+0000 7f2940ce8700 1 -- 172.21.15.160:0/3128773328 shutdown_connections 2023-11-17T05:37:06.159 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:06.158+0000 7f2940ce8700 1 -- 172.21.15.160:0/3128773328 wait complete. 2023-11-17T05:37:06.159 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:06.158+0000 7f2940ce8700 1 Processor -- start 2023-11-17T05:37:06.159 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:06.158+0000 7f2940ce8700 1 -- start start 2023-11-17T05:37:06.159 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:06.158+0000 7f2940ce8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f293c0f6510 0x7f293c1066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:37:06.159 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:06.158+0000 7f2940ce8700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f293c106bc0 con 0x7f293c0f6510 2023-11-17T05:37:06.161 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:06.159+0000 7f293a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f293c0f6510 0x7f293c1066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:37:06.161 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:06.159+0000 7f293a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f293c0f6510 0x7f293c1066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:52014/0 (socket says 172.21.15.160:52014) 2023-11-17T05:37:06.161 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:06.159+0000 7f293a59c700 1 -- 172.21.15.160:0/3970991668 learned_addr learned my addr 172.21.15.160:0/3970991668 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:37:06.161 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:06.159+0000 7f293a59c700 1 -- 172.21.15.160:0/3970991668 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2930005040 con 0x7f293c0f6510 2023-11-17T05:37:06.161 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:06.159+0000 7f293a59c700 1 --2- 172.21.15.160:0/3970991668 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f293c0f6510 0x7f293c1066d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f2930013510 tx=0x7f293000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:37:06.161 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:06.159+0000 7f292affd700 1 -- 172.21.15.160:0/3970991668 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2930009320 con 0x7f293c0f6510 2023-11-17T05:37:06.161 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:06.159+0000 7f2940ce8700 1 -- 172.21.15.160:0/3970991668 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f293c106dc0 con 0x7f293c0f6510 2023-11-17T05:37:06.161 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:06.159+0000 7f292affd700 1 -- 172.21.15.160:0/3970991668 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2930013f00 con 0x7f293c0f6510 2023-11-17T05:37:06.161 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:06.159+0000 7f292affd700 1 -- 172.21.15.160:0/3970991668 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2930014d90 con 0x7f293c0f6510 2023-11-17T05:37:06.161 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:06.159+0000 7f2940ce8700 1 -- 172.21.15.160:0/3970991668 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f293c107250 con 0x7f293c0f6510 2023-11-17T05:37:06.161 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:06.160+0000 7f292affd700 1 -- 172.21.15.160:0/3970991668 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f29300095f0 con 0x7f293c0f6510 2023-11-17T05:37:06.161 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:06.160+0000 7f2940ce8700 1 -- 172.21.15.160:0/3970991668 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f293c045ad0 con 0x7f293c0f6510 2023-11-17T05:37:06.161 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:06.160+0000 7f292affd700 1 --2- 172.21.15.160:0/3970991668 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f292402ff60 0x7f2924032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:37:06.162 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:06.160+0000 7f292affd700 1 -- 172.21.15.160:0/3970991668 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2930040fc0 con 0x7f293c0f6510 2023-11-17T05:37:06.162 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:06.161+0000 7f2939d9b700 1 --2- 172.21.15.160:0/3970991668 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f292402ff60 0x7f2924032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:37:06.162 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:06.161+0000 7f2939d9b700 1 --2- 172.21.15.160:0/3970991668 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f292402ff60 0x7f2924032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f292c0096a0 tx=0x7f292c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:37:06.165 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:06.165+0000 7f292affd700 1 -- 172.21.15.160:0/3970991668 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2930014420 con 0x7f293c0f6510 2023-11-17T05:37:06.370 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:06.370+0000 7f2940ce8700 1 -- 172.21.15.160:0/3970991668 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f293c1080a0 con 0x7f293c0f6510 2023-11-17T05:37:06.370 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:06.371+0000 7f292affd700 1 -- 172.21.15.160:0/3970991668 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f2930010230 con 0x7f293c0f6510 2023-11-17T05:37:06.371 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:37:06.371 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:37:06.372 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:06.372+0000 7f2940ce8700 1 -- 172.21.15.160:0/3970991668 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f292402ff60 msgr2=0x7f2924032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:06.372 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:06.372+0000 7f2940ce8700 1 --2- 172.21.15.160:0/3970991668 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f292402ff60 0x7f2924032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f292c0096a0 tx=0x7f292c006b40).stop 2023-11-17T05:37:06.372 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:06.372+0000 7f2940ce8700 1 -- 172.21.15.160:0/3970991668 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f293c0f6510 msgr2=0x7f293c1066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:06.372 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:06.372+0000 7f2940ce8700 1 --2- 172.21.15.160:0/3970991668 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f293c0f6510 0x7f293c1066d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f2930013510 tx=0x7f293000b040).stop 2023-11-17T05:37:06.372 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:06.372+0000 7f2940ce8700 1 -- 172.21.15.160:0/3970991668 shutdown_connections 2023-11-17T05:37:06.373 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:06.372+0000 7f2940ce8700 1 --2- 172.21.15.160:0/3970991668 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f292402ff60 0x7f2924032420 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:06.373 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:06.372+0000 7f2940ce8700 1 --2- 172.21.15.160:0/3970991668 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f293c0f6510 0x7f293c1066d0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:06.373 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:06.372+0000 7f2940ce8700 1 -- 172.21.15.160:0/3970991668 >> 172.21.15.160:0/3970991668 conn(0x7f293c0f1c70 msgr2=0x7f293c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:37:06.373 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:06.372+0000 7f2940ce8700 1 -- 172.21.15.160:0/3970991668 shutdown_connections 2023-11-17T05:37:06.373 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:06.373+0000 7f2940ce8700 1 -- 172.21.15.160:0/3970991668 wait complete. 2023-11-17T05:37:06.373 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:37:06.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:06 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:37:06.371023+0000 mon.smithi003 (mon.0) 294 : audit [DBG] from='client.? 172.21.15.160:0/3970991668' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:37:07.765 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:37:07.766 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:37:07.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:07 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:37:06.370916+0000 mgr.smithi003.eddcuc (mgr.14164) 360 : cluster [DBG] pgmap v344: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:37:08.909 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:08.907+0000 7fb9d3923700 1 -- 172.21.15.160:0/2576956078 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9cc0f6510 msgr2=0x7fb9cc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:08.909 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:08.907+0000 7fb9d3923700 1 --2- 172.21.15.160:0/2576956078 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9cc0f6510 0x7fb9cc0f6930 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7fb9c8009fb0 tx=0x7fb9c8009b30).stop 2023-11-17T05:37:08.909 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:08.907+0000 7fb9d3923700 1 -- 172.21.15.160:0/2576956078 shutdown_connections 2023-11-17T05:37:08.909 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:08.907+0000 7fb9d3923700 1 --2- 172.21.15.160:0/2576956078 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9cc0f6510 0x7fb9cc0f6930 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:08.910 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:08.907+0000 7fb9d3923700 1 -- 172.21.15.160:0/2576956078 >> 172.21.15.160:0/2576956078 conn(0x7fb9cc0f1c70 msgr2=0x7fb9cc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:37:08.910 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:08.907+0000 7fb9d3923700 1 -- 172.21.15.160:0/2576956078 shutdown_connections 2023-11-17T05:37:08.910 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:08.907+0000 7fb9d3923700 1 -- 172.21.15.160:0/2576956078 wait complete. 2023-11-17T05:37:08.910 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:08.908+0000 7fb9d3923700 1 Processor -- start 2023-11-17T05:37:08.910 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:08.908+0000 7fb9d3923700 1 -- start start 2023-11-17T05:37:08.910 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:08.908+0000 7fb9d3923700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9cc0f6510 0x7fb9cc06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:37:08.910 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:08.908+0000 7fb9d3923700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9cc069e20 con 0x7fb9cc0f6510 2023-11-17T05:37:08.910 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:08.908+0000 7fb9d16bf700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9cc0f6510 0x7fb9cc06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:37:08.911 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:08.908+0000 7fb9d16bf700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9cc0f6510 0x7fb9cc06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:52034/0 (socket says 172.21.15.160:52034) 2023-11-17T05:37:08.911 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:08.908+0000 7fb9d16bf700 1 -- 172.21.15.160:0/2717767997 learned_addr learned my addr 172.21.15.160:0/2717767997 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:37:08.911 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:08.908+0000 7fb9d16bf700 1 -- 172.21.15.160:0/2717767997 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c8005040 con 0x7fb9cc0f6510 2023-11-17T05:37:08.911 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:08.909+0000 7fb9d16bf700 1 --2- 172.21.15.160:0/2717767997 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9cc0f6510 0x7fb9cc06b9a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7fb9c8013510 tx=0x7fb9c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:37:08.912 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:08.909+0000 7fb9c1ffb700 1 -- 172.21.15.160:0/2717767997 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9c8009320 con 0x7fb9cc0f6510 2023-11-17T05:37:08.912 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:08.909+0000 7fb9d3923700 1 -- 172.21.15.160:0/2717767997 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9cc069fc0 con 0x7fb9cc0f6510 2023-11-17T05:37:08.913 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:08.909+0000 7fb9c1ffb700 1 -- 172.21.15.160:0/2717767997 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb9c8013f00 con 0x7fb9cc0f6510 2023-11-17T05:37:08.913 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:08.909+0000 7fb9c1ffb700 1 -- 172.21.15.160:0/2717767997 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9c8014d90 con 0x7fb9cc0f6510 2023-11-17T05:37:08.913 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:08.909+0000 7fb9d3923700 1 -- 172.21.15.160:0/2717767997 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9cc06a450 con 0x7fb9cc0f6510 2023-11-17T05:37:08.913 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:08.910+0000 7fb9c1ffb700 1 -- 172.21.15.160:0/2717767997 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb9c80095f0 con 0x7fb9cc0f6510 2023-11-17T05:37:08.913 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:08.910+0000 7fb9d3923700 1 -- 172.21.15.160:0/2717767997 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9cc059100 con 0x7fb9cc0f6510 2023-11-17T05:37:08.913 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:08.910+0000 7fb9c1ffb700 1 --2- 172.21.15.160:0/2717767997 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fb9b802ff60 0x7fb9b8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:37:08.913 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:08.910+0000 7fb9c1ffb700 1 -- 172.21.15.160:0/2717767997 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb9c8041000 con 0x7fb9cc0f6510 2023-11-17T05:37:08.913 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:08.910+0000 7fb9d0ebe700 1 --2- 172.21.15.160:0/2717767997 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fb9b802ff60 0x7fb9b8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:37:08.913 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:08.910+0000 7fb9d0ebe700 1 --2- 172.21.15.160:0/2717767997 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fb9b802ff60 0x7fb9b8032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fb9bc0096a0 tx=0x7fb9bc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:37:08.917 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:08.915+0000 7fb9c1ffb700 1 -- 172.21.15.160:0/2717767997 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb9c8014420 con 0x7fb9cc0f6510 2023-11-17T05:37:09.121 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:09.121+0000 7fb9d3923700 1 -- 172.21.15.160:0/2717767997 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb9cc0fbf60 con 0x7fb9cc0f6510 2023-11-17T05:37:09.122 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:09.121+0000 7fb9c1ffb700 1 -- 172.21.15.160:0/2717767997 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fb9c801d7f0 con 0x7fb9cc0f6510 2023-11-17T05:37:09.122 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:37:09.122 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:37:09.123 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:09.123+0000 7fb9d3923700 1 -- 172.21.15.160:0/2717767997 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fb9b802ff60 msgr2=0x7fb9b8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:09.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:09.123+0000 7fb9d3923700 1 --2- 172.21.15.160:0/2717767997 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fb9b802ff60 0x7fb9b8032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fb9bc0096a0 tx=0x7fb9bc006b40).stop 2023-11-17T05:37:09.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:09.123+0000 7fb9d3923700 1 -- 172.21.15.160:0/2717767997 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9cc0f6510 msgr2=0x7fb9cc06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:09.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:09.123+0000 7fb9d3923700 1 --2- 172.21.15.160:0/2717767997 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9cc0f6510 0x7fb9cc06b9a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7fb9c8013510 tx=0x7fb9c800b040).stop 2023-11-17T05:37:09.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:09.123+0000 7fb9d3923700 1 -- 172.21.15.160:0/2717767997 shutdown_connections 2023-11-17T05:37:09.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:09.123+0000 7fb9d3923700 1 --2- 172.21.15.160:0/2717767997 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fb9b802ff60 0x7fb9b8032420 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:09.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:09.123+0000 7fb9d3923700 1 --2- 172.21.15.160:0/2717767997 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb9cc0f6510 0x7fb9cc06b9a0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:09.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:09.123+0000 7fb9d3923700 1 -- 172.21.15.160:0/2717767997 >> 172.21.15.160:0/2717767997 conn(0x7fb9cc0f1c70 msgr2=0x7fb9cc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:37:09.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:09.123+0000 7fb9d3923700 1 -- 172.21.15.160:0/2717767997 shutdown_connections 2023-11-17T05:37:09.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:09.123+0000 7fb9d3923700 1 -- 172.21.15.160:0/2717767997 wait complete. 2023-11-17T05:37:09.125 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:37:09.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:09 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:37:08.371387+0000 mgr.smithi003.eddcuc (mgr.14164) 361 : cluster [DBG] pgmap v345: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:37:09.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:09 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:37:09.122406+0000 mon.smithi003 (mon.0) 295 : audit [DBG] from='client.? 172.21.15.160:0/2717767997' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:37:10.518 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:37:10.518 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:37:11.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:11 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:37:10.371814+0000 mgr.smithi003.eddcuc (mgr.14164) 362 : cluster [DBG] pgmap v346: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:37:11.775 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:11.773+0000 7f1fcc9b3700 1 -- 172.21.15.160:0/1133862304 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1fc80f79a0 msgr2=0x7f1fc80f7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:11.775 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:11.773+0000 7f1fcc9b3700 1 --2- 172.21.15.160:0/1133862304 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1fc80f79a0 0x7f1fc80f7dc0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f1fb0009fb0 tx=0x7f1fb0009b30).stop 2023-11-17T05:37:11.775 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:11.773+0000 7f1fcc9b3700 1 -- 172.21.15.160:0/1133862304 shutdown_connections 2023-11-17T05:37:11.775 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:11.773+0000 7f1fcc9b3700 1 --2- 172.21.15.160:0/1133862304 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1fc80f79a0 0x7f1fc80f7dc0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:11.775 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:11.773+0000 7f1fcc9b3700 1 -- 172.21.15.160:0/1133862304 >> 172.21.15.160:0/1133862304 conn(0x7f1fc80f0120 msgr2=0x7f1fc80f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:37:11.775 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:11.774+0000 7f1fcc9b3700 1 -- 172.21.15.160:0/1133862304 shutdown_connections 2023-11-17T05:37:11.775 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:11.774+0000 7f1fcc9b3700 1 -- 172.21.15.160:0/1133862304 wait complete. 2023-11-17T05:37:11.776 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:11.774+0000 7f1fcc9b3700 1 Processor -- start 2023-11-17T05:37:11.777 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:11.774+0000 7f1fcc9b3700 1 -- start start 2023-11-17T05:37:11.777 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:11.775+0000 7f1fcc9b3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1fc80f79a0 0x7f1fc81064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:37:11.777 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:11.775+0000 7f1fcc9b3700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc81069a0 con 0x7f1fc80f79a0 2023-11-17T05:37:11.777 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:11.775+0000 7f1fc6d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1fc80f79a0 0x7f1fc81064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:37:11.777 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:11.775+0000 7f1fc6d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1fc80f79a0 0x7f1fc81064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:46576/0 (socket says 172.21.15.160:46576) 2023-11-17T05:37:11.777 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:11.775+0000 7f1fc6d9d700 1 -- 172.21.15.160:0/4257750316 learned_addr learned my addr 172.21.15.160:0/4257750316 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:37:11.777 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:11.775+0000 7f1fc6d9d700 1 -- 172.21.15.160:0/4257750316 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb0005040 con 0x7f1fc80f79a0 2023-11-17T05:37:11.777 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:11.775+0000 7f1fc6d9d700 1 --2- 172.21.15.160:0/4257750316 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1fc80f79a0 0x7f1fc81064b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f1fb0013510 tx=0x7f1fb000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:37:11.777 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:11.775+0000 7f1fbf7fe700 1 -- 172.21.15.160:0/4257750316 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fb0009320 con 0x7f1fc80f79a0 2023-11-17T05:37:11.777 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:11.776+0000 7f1fcc9b3700 1 -- 172.21.15.160:0/4257750316 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fc8106ba0 con 0x7f1fc80f79a0 2023-11-17T05:37:11.777 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:11.776+0000 7f1fbf7fe700 1 -- 172.21.15.160:0/4257750316 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1fb0013f00 con 0x7f1fc80f79a0 2023-11-17T05:37:11.777 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:11.776+0000 7f1fbf7fe700 1 -- 172.21.15.160:0/4257750316 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fb0014dc0 con 0x7f1fc80f79a0 2023-11-17T05:37:11.778 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:11.776+0000 7f1fcc9b3700 1 -- 172.21.15.160:0/4257750316 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fc8107030 con 0x7f1fc80f79a0 2023-11-17T05:37:11.778 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:11.776+0000 7f1fbf7fe700 1 -- 172.21.15.160:0/4257750316 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1fb0014420 con 0x7f1fc80f79a0 2023-11-17T05:37:11.778 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:11.777+0000 7f1fcc9b3700 1 -- 172.21.15.160:0/4257750316 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fc8045ad0 con 0x7f1fc80f79a0 2023-11-17T05:37:11.778 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:11.777+0000 7f1fbf7fe700 1 --2- 172.21.15.160:0/4257750316 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f1fb402ff60 0x7f1fb4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:37:11.779 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:11.777+0000 7f1fbf7fe700 1 -- 172.21.15.160:0/4257750316 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1fb0041050 con 0x7f1fc80f79a0 2023-11-17T05:37:11.779 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:11.777+0000 7f1fc659c700 1 --2- 172.21.15.160:0/4257750316 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f1fb402ff60 0x7f1fb4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:37:11.779 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:11.777+0000 7f1fc659c700 1 --2- 172.21.15.160:0/4257750316 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f1fb402ff60 0x7f1fb4032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f1fb80096a0 tx=0x7f1fb8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:37:11.783 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:11.782+0000 7f1fbf7fe700 1 -- 172.21.15.160:0/4257750316 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1fb00095f0 con 0x7f1fc80f79a0 2023-11-17T05:37:11.985 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:11.984+0000 7f1fcc9b3700 1 -- 172.21.15.160:0/4257750316 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1fc8107ee0 con 0x7f1fc80f79a0 2023-11-17T05:37:11.985 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:11.985+0000 7f1fbf7fe700 1 -- 172.21.15.160:0/4257750316 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f1fb001cb10 con 0x7f1fc80f79a0 2023-11-17T05:37:11.986 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:37:11.986 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:37:11.987 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:11.986+0000 7f1fcc9b3700 1 -- 172.21.15.160:0/4257750316 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f1fb402ff60 msgr2=0x7f1fb4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:11.987 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:11.986+0000 7f1fcc9b3700 1 --2- 172.21.15.160:0/4257750316 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f1fb402ff60 0x7f1fb4032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f1fb80096a0 tx=0x7f1fb8006b40).stop 2023-11-17T05:37:11.987 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:11.986+0000 7f1fcc9b3700 1 -- 172.21.15.160:0/4257750316 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1fc80f79a0 msgr2=0x7f1fc81064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:11.988 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:11.986+0000 7f1fcc9b3700 1 --2- 172.21.15.160:0/4257750316 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1fc80f79a0 0x7f1fc81064b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f1fb0013510 tx=0x7f1fb000b040).stop 2023-11-17T05:37:11.988 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:11.986+0000 7f1fcc9b3700 1 -- 172.21.15.160:0/4257750316 shutdown_connections 2023-11-17T05:37:11.988 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:11.986+0000 7f1fcc9b3700 1 --2- 172.21.15.160:0/4257750316 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f1fb402ff60 0x7f1fb4032420 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:11.988 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:11.986+0000 7f1fcc9b3700 1 --2- 172.21.15.160:0/4257750316 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1fc80f79a0 0x7f1fc81064b0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:11.988 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:11.986+0000 7f1fcc9b3700 1 -- 172.21.15.160:0/4257750316 >> 172.21.15.160:0/4257750316 conn(0x7f1fc80f0120 msgr2=0x7f1fc80f0bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:37:11.988 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:11.986+0000 7f1fcc9b3700 1 -- 172.21.15.160:0/4257750316 shutdown_connections 2023-11-17T05:37:11.988 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:11.986+0000 7f1fcc9b3700 1 -- 172.21.15.160:0/4257750316 wait complete. 2023-11-17T05:37:11.988 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:37:12.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:12 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:37:11.986071+0000 mon.smithi003 (mon.0) 296 : audit [DBG] from='client.? 172.21.15.160:0/4257750316' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:37:13.390 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:37:13.390 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:37:13.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:13 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:37:12.372095+0000 mgr.smithi003.eddcuc (mgr.14164) 363 : cluster [DBG] pgmap v347: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:37:14.538 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:14.536+0000 7f50f717c700 1 -- 172.21.15.160:0/3166250525 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f50f00f5410 msgr2=0x7f50f00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:14.538 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:14.536+0000 7f50f717c700 1 --2- 172.21.15.160:0/3166250525 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f50f00f5410 0x7f50f00f5830 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f50e4009fb0 tx=0x7f50e4009b30).stop 2023-11-17T05:37:14.538 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:14.536+0000 7f50f717c700 1 -- 172.21.15.160:0/3166250525 shutdown_connections 2023-11-17T05:37:14.538 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:14.536+0000 7f50f717c700 1 --2- 172.21.15.160:0/3166250525 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f50f00f5410 0x7f50f00f5830 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:14.538 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:14.537+0000 7f50f717c700 1 -- 172.21.15.160:0/3166250525 >> 172.21.15.160:0/3166250525 conn(0x7f50f00f0b70 msgr2=0x7f50f00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:37:14.539 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:14.537+0000 7f50f717c700 1 -- 172.21.15.160:0/3166250525 shutdown_connections 2023-11-17T05:37:14.539 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:14.537+0000 7f50f717c700 1 -- 172.21.15.160:0/3166250525 wait complete. 2023-11-17T05:37:14.539 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:14.537+0000 7f50f717c700 1 Processor -- start 2023-11-17T05:37:14.539 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:14.537+0000 7f50f717c700 1 -- start start 2023-11-17T05:37:14.539 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:14.537+0000 7f50f717c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f50f00f5410 0x7f50f0109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:37:14.539 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:14.537+0000 7f50f717c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50f0060240 con 0x7f50f00f5410 2023-11-17T05:37:14.540 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:14.538+0000 7f50f4f18700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f50f00f5410 0x7f50f0109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:37:14.540 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:14.538+0000 7f50f4f18700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f50f00f5410 0x7f50f0109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:46592/0 (socket says 172.21.15.160:46592) 2023-11-17T05:37:14.541 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:14.538+0000 7f50f4f18700 1 -- 172.21.15.160:0/1152693830 learned_addr learned my addr 172.21.15.160:0/1152693830 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:37:14.541 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:14.538+0000 7f50f4f18700 1 -- 172.21.15.160:0/1152693830 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e4005040 con 0x7f50f00f5410 2023-11-17T05:37:14.541 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:14.538+0000 7f50f4f18700 1 --2- 172.21.15.160:0/1152693830 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f50f00f5410 0x7f50f0109430 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f50e4013510 tx=0x7f50e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:37:14.541 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:14.538+0000 7f50ed7fa700 1 -- 172.21.15.160:0/1152693830 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50e4009320 con 0x7f50f00f5410 2023-11-17T05:37:14.541 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:14.538+0000 7f50f717c700 1 -- 172.21.15.160:0/1152693830 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50f0060440 con 0x7f50f00f5410 2023-11-17T05:37:14.541 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:14.538+0000 7f50ed7fa700 1 -- 172.21.15.160:0/1152693830 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f50e4013f00 con 0x7f50f00f5410 2023-11-17T05:37:14.541 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:14.538+0000 7f50ed7fa700 1 -- 172.21.15.160:0/1152693830 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50e4014d90 con 0x7f50f00f5410 2023-11-17T05:37:14.541 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:14.538+0000 7f50f717c700 1 -- 172.21.15.160:0/1152693830 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50f0109ae0 con 0x7f50f00f5410 2023-11-17T05:37:14.541 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:14.539+0000 7f50ed7fa700 1 -- 172.21.15.160:0/1152693830 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f50e40095f0 con 0x7f50f00f5410 2023-11-17T05:37:14.541 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:14.539+0000 7f50f717c700 1 -- 172.21.15.160:0/1152693830 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50f0045ad0 con 0x7f50f00f5410 2023-11-17T05:37:14.541 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:14.539+0000 7f50ed7fa700 1 --2- 172.21.15.160:0/1152693830 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f50e002ffb0 0x7f50e0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:37:14.541 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:14.539+0000 7f50ed7fa700 1 -- 172.21.15.160:0/1152693830 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f50e4040fe0 con 0x7f50f00f5410 2023-11-17T05:37:14.542 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:14.540+0000 7f50effff700 1 --2- 172.21.15.160:0/1152693830 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f50e002ffb0 0x7f50e0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:37:14.542 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:14.540+0000 7f50effff700 1 --2- 172.21.15.160:0/1152693830 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f50e002ffb0 0x7f50e0032470 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f50dc0096a0 tx=0x7f50dc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:37:14.545 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:14.544+0000 7f50ed7fa700 1 -- 172.21.15.160:0/1152693830 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f50e406d2b0 con 0x7f50f00f5410 2023-11-17T05:37:14.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:14.753+0000 7f50f717c700 1 -- 172.21.15.160:0/1152693830 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f50f00fae60 con 0x7f50f00f5410 2023-11-17T05:37:14.755 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:14.753+0000 7f50ed7fa700 1 -- 172.21.15.160:0/1152693830 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f50e4014420 con 0x7f50f00f5410 2023-11-17T05:37:14.755 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:37:14.755 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:37:14.756 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:14.755+0000 7f50f717c700 1 -- 172.21.15.160:0/1152693830 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f50e002ffb0 msgr2=0x7f50e0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:14.756 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:14.755+0000 7f50f717c700 1 --2- 172.21.15.160:0/1152693830 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f50e002ffb0 0x7f50e0032470 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f50dc0096a0 tx=0x7f50dc006b40).stop 2023-11-17T05:37:14.756 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:14.755+0000 7f50f717c700 1 -- 172.21.15.160:0/1152693830 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f50f00f5410 msgr2=0x7f50f0109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:14.756 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:14.755+0000 7f50f717c700 1 --2- 172.21.15.160:0/1152693830 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f50f00f5410 0x7f50f0109430 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f50e4013510 tx=0x7f50e400b040).stop 2023-11-17T05:37:14.756 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:14.755+0000 7f50f717c700 1 -- 172.21.15.160:0/1152693830 shutdown_connections 2023-11-17T05:37:14.756 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:14.755+0000 7f50f717c700 1 --2- 172.21.15.160:0/1152693830 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f50e002ffb0 0x7f50e0032470 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:14.756 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:14.755+0000 7f50f717c700 1 --2- 172.21.15.160:0/1152693830 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f50f00f5410 0x7f50f0109430 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:14.756 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:14.755+0000 7f50f717c700 1 -- 172.21.15.160:0/1152693830 >> 172.21.15.160:0/1152693830 conn(0x7f50f00f0b70 msgr2=0x7f50f00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:37:14.756 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:14.755+0000 7f50f717c700 1 -- 172.21.15.160:0/1152693830 shutdown_connections 2023-11-17T05:37:14.756 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:14.755+0000 7f50f717c700 1 -- 172.21.15.160:0/1152693830 wait complete. 2023-11-17T05:37:14.757 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:37:15.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:15 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:37:14.372394+0000 mgr.smithi003.eddcuc (mgr.14164) 364 : cluster [DBG] pgmap v348: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:37:15.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:15 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:37:14.754618+0000 mon.smithi003 (mon.0) 297 : audit [DBG] from='client.? 172.21.15.160:0/1152693830' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:37:16.146 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:37:16.146 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:37:17.343 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:17.341+0000 7f6ae2e63700 1 -- 172.21.15.160:0/1506171824 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6adc0f6510 msgr2=0x7f6adc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:17.343 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:17.341+0000 7f6ae2e63700 1 --2- 172.21.15.160:0/1506171824 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6adc0f6510 0x7f6adc0f6930 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f6ad0009fb0 tx=0x7f6ad0009b30).stop 2023-11-17T05:37:17.343 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:17.341+0000 7f6ae2e63700 1 -- 172.21.15.160:0/1506171824 shutdown_connections 2023-11-17T05:37:17.343 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:17.341+0000 7f6ae2e63700 1 --2- 172.21.15.160:0/1506171824 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6adc0f6510 0x7f6adc0f6930 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:17.343 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:17.341+0000 7f6ae2e63700 1 -- 172.21.15.160:0/1506171824 >> 172.21.15.160:0/1506171824 conn(0x7f6adc0f1c70 msgr2=0x7f6adc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:37:17.343 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:17.341+0000 7f6ae2e63700 1 -- 172.21.15.160:0/1506171824 shutdown_connections 2023-11-17T05:37:17.343 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:17.341+0000 7f6ae2e63700 1 -- 172.21.15.160:0/1506171824 wait complete. 2023-11-17T05:37:17.343 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:17.342+0000 7f6ae2e63700 1 Processor -- start 2023-11-17T05:37:17.344 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:17.342+0000 7f6ae2e63700 1 -- start start 2023-11-17T05:37:17.344 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:17.342+0000 7f6ae2e63700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6adc0f6510 0x7f6adc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:37:17.344 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:17.342+0000 7f6ae2e63700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6adc106c20 con 0x7f6adc0f6510 2023-11-17T05:37:17.345 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:17.342+0000 7f6ae0bff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6adc0f6510 0x7f6adc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:37:17.346 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:17.342+0000 7f6ae0bff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6adc0f6510 0x7f6adc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:46602/0 (socket says 172.21.15.160:46602) 2023-11-17T05:37:17.346 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:17.342+0000 7f6ae0bff700 1 -- 172.21.15.160:0/1449537471 learned_addr learned my addr 172.21.15.160:0/1449537471 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:37:17.346 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:17.343+0000 7f6ae0bff700 1 -- 172.21.15.160:0/1449537471 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad0005040 con 0x7f6adc0f6510 2023-11-17T05:37:17.346 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:17.343+0000 7f6ae0bff700 1 --2- 172.21.15.160:0/1449537471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6adc0f6510 0x7f6adc106730 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f6ad0013510 tx=0x7f6ad000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:37:17.346 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:17.343+0000 7f6ad97fa700 1 -- 172.21.15.160:0/1449537471 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ad0009320 con 0x7f6adc0f6510 2023-11-17T05:37:17.346 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:17.343+0000 7f6ae2e63700 1 -- 172.21.15.160:0/1449537471 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6adc106e20 con 0x7f6adc0f6510 2023-11-17T05:37:17.346 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:17.343+0000 7f6ad97fa700 1 -- 172.21.15.160:0/1449537471 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6ad0013f00 con 0x7f6adc0f6510 2023-11-17T05:37:17.346 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:17.343+0000 7f6ad97fa700 1 -- 172.21.15.160:0/1449537471 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ad0014d90 con 0x7f6adc0f6510 2023-11-17T05:37:17.346 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:17.343+0000 7f6ae2e63700 1 -- 172.21.15.160:0/1449537471 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6adc1072b0 con 0x7f6adc0f6510 2023-11-17T05:37:17.346 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:17.344+0000 7f6ad97fa700 1 -- 172.21.15.160:0/1449537471 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6ad00095f0 con 0x7f6adc0f6510 2023-11-17T05:37:17.347 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:17.344+0000 7f6ae2e63700 1 -- 172.21.15.160:0/1449537471 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6adc045ad0 con 0x7f6adc0f6510 2023-11-17T05:37:17.347 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:17.344+0000 7f6ad97fa700 1 --2- 172.21.15.160:0/1449537471 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f6acc02ff60 0x7f6acc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:37:17.347 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:17.344+0000 7f6ad97fa700 1 -- 172.21.15.160:0/1449537471 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6ad0041000 con 0x7f6adc0f6510 2023-11-17T05:37:17.347 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:17.344+0000 7f6adbfff700 1 --2- 172.21.15.160:0/1449537471 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f6acc02ff60 0x7f6acc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:37:17.347 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:17.345+0000 7f6adbfff700 1 --2- 172.21.15.160:0/1449537471 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f6acc02ff60 0x7f6acc032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f6ac80096a0 tx=0x7f6ac8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:37:17.362 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:17.349+0000 7f6ad97fa700 1 -- 172.21.15.160:0/1449537471 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6ad0014420 con 0x7f6adc0f6510 2023-11-17T05:37:17.555 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:17.555+0000 7f6ae2e63700 1 -- 172.21.15.160:0/1449537471 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6adc0fbf60 con 0x7f6adc0f6510 2023-11-17T05:37:17.557 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:17.555+0000 7f6ad97fa700 1 -- 172.21.15.160:0/1449537471 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f6ad0010230 con 0x7f6adc0f6510 2023-11-17T05:37:17.557 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:37:17.557 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:37:17.559 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:17.557+0000 7f6ae2e63700 1 -- 172.21.15.160:0/1449537471 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f6acc02ff60 msgr2=0x7f6acc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:17.559 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:17.557+0000 7f6ae2e63700 1 --2- 172.21.15.160:0/1449537471 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f6acc02ff60 0x7f6acc032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f6ac80096a0 tx=0x7f6ac8006b40).stop 2023-11-17T05:37:17.559 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:17.557+0000 7f6ae2e63700 1 -- 172.21.15.160:0/1449537471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6adc0f6510 msgr2=0x7f6adc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:17.559 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:17.557+0000 7f6ae2e63700 1 --2- 172.21.15.160:0/1449537471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6adc0f6510 0x7f6adc106730 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f6ad0013510 tx=0x7f6ad000b040).stop 2023-11-17T05:37:17.559 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:17.557+0000 7f6ae2e63700 1 -- 172.21.15.160:0/1449537471 shutdown_connections 2023-11-17T05:37:17.559 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:17.557+0000 7f6ae2e63700 1 --2- 172.21.15.160:0/1449537471 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f6acc02ff60 0x7f6acc032420 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:17.559 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:17.557+0000 7f6ae2e63700 1 --2- 172.21.15.160:0/1449537471 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6adc0f6510 0x7f6adc106730 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:17.559 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:17.557+0000 7f6ae2e63700 1 -- 172.21.15.160:0/1449537471 >> 172.21.15.160:0/1449537471 conn(0x7f6adc0f1c70 msgr2=0x7f6adc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:37:17.559 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:17.557+0000 7f6ae2e63700 1 -- 172.21.15.160:0/1449537471 shutdown_connections 2023-11-17T05:37:17.560 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:17.557+0000 7f6ae2e63700 1 -- 172.21.15.160:0/1449537471 wait complete. 2023-11-17T05:37:17.560 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:37:17.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:17 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:37:16.372832+0000 mgr.smithi003.eddcuc (mgr.14164) 365 : cluster [DBG] pgmap v349: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:37:18.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:18 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:37:17.556710+0000 mon.smithi003 (mon.0) 298 : audit [DBG] from='client.? 172.21.15.160:0/1449537471' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:37:18.987 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:37:18.987 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:37:19.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:19 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:37:18.373225+0000 mgr.smithi003.eddcuc (mgr.14164) 366 : cluster [DBG] pgmap v350: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:37:20.129 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:20.128+0000 7f8593a7c700 1 -- 172.21.15.160:0/713277834 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f858c0f6950 msgr2=0x7f858c0f6d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:20.130 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:20.128+0000 7f8593a7c700 1 --2- 172.21.15.160:0/713277834 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f858c0f6950 0x7f858c0f6d70 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f857c009fb0 tx=0x7f857c009b30).stop 2023-11-17T05:37:20.130 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:20.128+0000 7f8593a7c700 1 -- 172.21.15.160:0/713277834 shutdown_connections 2023-11-17T05:37:20.130 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:20.128+0000 7f8593a7c700 1 --2- 172.21.15.160:0/713277834 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f858c0f6950 0x7f858c0f6d70 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:20.130 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:20.128+0000 7f8593a7c700 1 -- 172.21.15.160:0/713277834 >> 172.21.15.160:0/713277834 conn(0x7f858c0f0260 msgr2=0x7f858c0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:37:20.130 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:20.128+0000 7f8593a7c700 1 -- 172.21.15.160:0/713277834 shutdown_connections 2023-11-17T05:37:20.130 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:20.128+0000 7f8593a7c700 1 -- 172.21.15.160:0/713277834 wait complete. 2023-11-17T05:37:20.130 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:20.128+0000 7f8593a7c700 1 Processor -- start 2023-11-17T05:37:20.130 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:20.128+0000 7f8593a7c700 1 -- start start 2023-11-17T05:37:20.130 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:20.128+0000 7f8593a7c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f858c0f6950 0x7f858c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:37:20.130 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:20.128+0000 7f8593a7c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f858c106bd0 con 0x7f858c0f6950 2023-11-17T05:37:20.131 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:20.129+0000 7f8591818700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f858c0f6950 0x7f858c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:37:20.131 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:20.129+0000 7f8591818700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f858c0f6950 0x7f858c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:46622/0 (socket says 172.21.15.160:46622) 2023-11-17T05:37:20.131 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:20.129+0000 7f8591818700 1 -- 172.21.15.160:0/563686295 learned_addr learned my addr 172.21.15.160:0/563686295 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:37:20.132 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:20.129+0000 7f8591818700 1 -- 172.21.15.160:0/563686295 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f857c005040 con 0x7f858c0f6950 2023-11-17T05:37:20.132 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:20.129+0000 7f8591818700 1 --2- 172.21.15.160:0/563686295 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f858c0f6950 0x7f858c1066e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f857c013510 tx=0x7f857c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:37:20.132 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:20.129+0000 7f85827fc700 1 -- 172.21.15.160:0/563686295 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f857c009320 con 0x7f858c0f6950 2023-11-17T05:37:20.132 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:20.129+0000 7f8593a7c700 1 -- 172.21.15.160:0/563686295 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f858c106dd0 con 0x7f858c0f6950 2023-11-17T05:37:20.132 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:20.129+0000 7f85827fc700 1 -- 172.21.15.160:0/563686295 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f857c013f00 con 0x7f858c0f6950 2023-11-17T05:37:20.132 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:20.129+0000 7f85827fc700 1 -- 172.21.15.160:0/563686295 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f857c014d90 con 0x7f858c0f6950 2023-11-17T05:37:20.132 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:20.129+0000 7f8593a7c700 1 -- 172.21.15.160:0/563686295 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f858c107260 con 0x7f858c0f6950 2023-11-17T05:37:20.132 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:20.130+0000 7f8593a7c700 1 -- 172.21.15.160:0/563686295 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f858c045ad0 con 0x7f858c0f6950 2023-11-17T05:37:20.133 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:20.130+0000 7f85827fc700 1 -- 172.21.15.160:0/563686295 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f857c0095f0 con 0x7f858c0f6950 2023-11-17T05:37:20.133 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:20.131+0000 7f85827fc700 1 --2- 172.21.15.160:0/563686295 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f857802ff60 0x7f8578032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:37:20.133 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:20.131+0000 7f85827fc700 1 -- 172.21.15.160:0/563686295 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f857c040fc0 con 0x7f858c0f6950 2023-11-17T05:37:20.133 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:20.131+0000 7f8591017700 1 --2- 172.21.15.160:0/563686295 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f857802ff60 0x7f8578032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:37:20.133 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:20.131+0000 7f8591017700 1 --2- 172.21.15.160:0/563686295 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f857802ff60 0x7f8578032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f85880096a0 tx=0x7f8588006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:37:20.136 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:20.135+0000 7f85827fc700 1 -- 172.21.15.160:0/563686295 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f857c014420 con 0x7f858c0f6950 2023-11-17T05:37:20.343 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:20.341+0000 7f8593a7c700 1 -- 172.21.15.160:0/563686295 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f858c1080b0 con 0x7f858c0f6950 2023-11-17T05:37:20.344 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:20.342+0000 7f85827fc700 1 -- 172.21.15.160:0/563686295 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f857c010230 con 0x7f858c0f6950 2023-11-17T05:37:20.347 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:37:20.348 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:37:20.352 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:20.343+0000 7f8593a7c700 1 -- 172.21.15.160:0/563686295 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f857802ff60 msgr2=0x7f8578032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:20.353 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:20.343+0000 7f8593a7c700 1 --2- 172.21.15.160:0/563686295 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f857802ff60 0x7f8578032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f85880096a0 tx=0x7f8588006b40).stop 2023-11-17T05:37:20.354 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:20.343+0000 7f8593a7c700 1 -- 172.21.15.160:0/563686295 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f858c0f6950 msgr2=0x7f858c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:20.354 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:20.343+0000 7f8593a7c700 1 --2- 172.21.15.160:0/563686295 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f858c0f6950 0x7f858c1066e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f857c013510 tx=0x7f857c00b040).stop 2023-11-17T05:37:20.355 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:20.343+0000 7f8593a7c700 1 -- 172.21.15.160:0/563686295 shutdown_connections 2023-11-17T05:37:20.355 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:20.343+0000 7f8593a7c700 1 --2- 172.21.15.160:0/563686295 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f857802ff60 0x7f8578032420 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:20.356 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:20.343+0000 7f8593a7c700 1 --2- 172.21.15.160:0/563686295 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f858c0f6950 0x7f858c1066e0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:20.356 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:20.343+0000 7f8593a7c700 1 -- 172.21.15.160:0/563686295 >> 172.21.15.160:0/563686295 conn(0x7f858c0f0260 msgr2=0x7f858c0f4d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:37:20.356 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:20.344+0000 7f8593a7c700 1 -- 172.21.15.160:0/563686295 shutdown_connections 2023-11-17T05:37:20.357 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:20.344+0000 7f8593a7c700 1 -- 172.21.15.160:0/563686295 wait complete. 2023-11-17T05:37:20.358 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:37:20.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:20 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:37:20.343021+0000 mon.smithi003 (mon.0) 299 : audit [DBG] from='client.? 172.21.15.160:0/563686295' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:37:21.726 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:37:21.727 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:37:21.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:21 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:37:20.373601+0000 mgr.smithi003.eddcuc (mgr.14164) 367 : cluster [DBG] pgmap v351: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:37:22.910 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:22.905+0000 7fed7147c700 1 -- 172.21.15.160:0/2528657199 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed6c0f47c0 msgr2=0x7fed6c0f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:22.911 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:22.905+0000 7fed7147c700 1 --2- 172.21.15.160:0/2528657199 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed6c0f47c0 0x7fed6c0f4be0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7fed5c009fb0 tx=0x7fed5c009b30).stop 2023-11-17T05:37:22.911 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:22.905+0000 7fed7147c700 1 -- 172.21.15.160:0/2528657199 shutdown_connections 2023-11-17T05:37:22.911 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:22.905+0000 7fed7147c700 1 --2- 172.21.15.160:0/2528657199 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed6c0f47c0 0x7fed6c0f4be0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:22.911 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:22.905+0000 7fed7147c700 1 -- 172.21.15.160:0/2528657199 >> 172.21.15.160:0/2528657199 conn(0x7fed6c0f02e0 msgr2=0x7fed6c0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:37:22.911 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:22.905+0000 7fed7147c700 1 -- 172.21.15.160:0/2528657199 shutdown_connections 2023-11-17T05:37:22.911 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:22.905+0000 7fed7147c700 1 -- 172.21.15.160:0/2528657199 wait complete. 2023-11-17T05:37:22.911 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:22.906+0000 7fed7147c700 1 Processor -- start 2023-11-17T05:37:22.911 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:22.906+0000 7fed7147c700 1 -- start start 2023-11-17T05:37:22.911 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:22.906+0000 7fed7147c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed6c0f47c0 0x7fed6c0f7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:37:22.911 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:22.906+0000 7fed7147c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed6c0f57f0 con 0x7fed6c0f47c0 2023-11-17T05:37:22.911 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:22.906+0000 7fed6affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed6c0f47c0 0x7fed6c0f7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:37:22.911 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:22.906+0000 7fed6affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed6c0f47c0 0x7fed6c0f7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:33774/0 (socket says 172.21.15.160:33774) 2023-11-17T05:37:22.912 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:22.906+0000 7fed6affd700 1 -- 172.21.15.160:0/557329374 learned_addr learned my addr 172.21.15.160:0/557329374 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:37:22.912 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:22.907+0000 7fed6affd700 1 -- 172.21.15.160:0/557329374 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed5c005040 con 0x7fed6c0f47c0 2023-11-17T05:37:22.912 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:22.907+0000 7fed6affd700 1 --2- 172.21.15.160:0/557329374 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed6c0f47c0 0x7fed6c0f7370 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7fed5c013510 tx=0x7fed5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:37:22.912 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:22.907+0000 7fed537fe700 1 -- 172.21.15.160:0/557329374 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed5c009320 con 0x7fed6c0f47c0 2023-11-17T05:37:22.912 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:22.907+0000 7fed7147c700 1 -- 172.21.15.160:0/557329374 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed6c0f5990 con 0x7fed6c0f47c0 2023-11-17T05:37:22.912 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:22.907+0000 7fed537fe700 1 -- 172.21.15.160:0/557329374 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fed5c013f00 con 0x7fed6c0f47c0 2023-11-17T05:37:22.912 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:22.907+0000 7fed537fe700 1 -- 172.21.15.160:0/557329374 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed5c014d90 con 0x7fed6c0f47c0 2023-11-17T05:37:22.912 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:22.907+0000 7fed7147c700 1 -- 172.21.15.160:0/557329374 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed6c0f5e20 con 0x7fed6c0f47c0 2023-11-17T05:37:22.912 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:22.908+0000 7fed7147c700 1 -- 172.21.15.160:0/557329374 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed6c059100 con 0x7fed6c0f47c0 2023-11-17T05:37:22.912 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:22.908+0000 7fed537fe700 1 -- 172.21.15.160:0/557329374 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fed5c0095f0 con 0x7fed6c0f47c0 2023-11-17T05:37:22.912 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:22.908+0000 7fed537fe700 1 --2- 172.21.15.160:0/557329374 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fed5402ff60 0x7fed54032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:37:22.912 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:22.908+0000 7fed537fe700 1 -- 172.21.15.160:0/557329374 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fed5c041080 con 0x7fed6c0f47c0 2023-11-17T05:37:22.912 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:22.908+0000 7fed6a7fc700 1 --2- 172.21.15.160:0/557329374 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fed5402ff60 0x7fed54032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:37:22.913 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:22.909+0000 7fed6a7fc700 1 --2- 172.21.15.160:0/557329374 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fed5402ff60 0x7fed54032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fed600096a0 tx=0x7fed60006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:37:22.914 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:22.913+0000 7fed537fe700 1 -- 172.21.15.160:0/557329374 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fed5c014420 con 0x7fed6c0f47c0 2023-11-17T05:37:23.126 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:23.121+0000 7fed7147c700 1 -- 172.21.15.160:0/557329374 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fed6c0fbf60 con 0x7fed6c0f47c0 2023-11-17T05:37:23.127 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:23.121+0000 7fed537fe700 1 -- 172.21.15.160:0/557329374 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fed5c01d7f0 con 0x7fed6c0f47c0 2023-11-17T05:37:23.127 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:37:23.127 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:37:23.128 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:23.123+0000 7fed7147c700 1 -- 172.21.15.160:0/557329374 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fed5402ff60 msgr2=0x7fed54032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:23.128 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:23.123+0000 7fed7147c700 1 --2- 172.21.15.160:0/557329374 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fed5402ff60 0x7fed54032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fed600096a0 tx=0x7fed60006b40).stop 2023-11-17T05:37:23.128 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:23.123+0000 7fed7147c700 1 -- 172.21.15.160:0/557329374 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed6c0f47c0 msgr2=0x7fed6c0f7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:23.128 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:23.123+0000 7fed7147c700 1 --2- 172.21.15.160:0/557329374 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed6c0f47c0 0x7fed6c0f7370 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7fed5c013510 tx=0x7fed5c00b040).stop 2023-11-17T05:37:23.128 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:23.123+0000 7fed7147c700 1 -- 172.21.15.160:0/557329374 shutdown_connections 2023-11-17T05:37:23.128 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:23.123+0000 7fed7147c700 1 --2- 172.21.15.160:0/557329374 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fed5402ff60 0x7fed54032420 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:23.128 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:23.123+0000 7fed7147c700 1 --2- 172.21.15.160:0/557329374 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed6c0f47c0 0x7fed6c0f7370 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:23.128 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:23.123+0000 7fed7147c700 1 -- 172.21.15.160:0/557329374 >> 172.21.15.160:0/557329374 conn(0x7fed6c0f02e0 msgr2=0x7fed6c0f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:37:23.128 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:23.123+0000 7fed7147c700 1 -- 172.21.15.160:0/557329374 shutdown_connections 2023-11-17T05:37:23.128 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:23.123+0000 7fed7147c700 1 -- 172.21.15.160:0/557329374 wait complete. 2023-11-17T05:37:23.128 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:37:23.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:23 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:37:22.373950+0000 mgr.smithi003.eddcuc (mgr.14164) 368 : cluster [DBG] pgmap v352: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:37:23.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:23 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:37:23.122504+0000 mon.smithi003 (mon.0) 300 : audit [DBG] from='client.? 172.21.15.160:0/557329374' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:37:24.538 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:37:24.538 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:37:25.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:25 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:37:24.374362+0000 mgr.smithi003.eddcuc (mgr.14164) 369 : cluster [DBG] pgmap v353: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:37:25.654 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:25.652+0000 7fc4b5f46700 1 -- 172.21.15.160:0/2966193840 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4b0060220 msgr2=0x7fc4b00f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:25.654 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:25.652+0000 7fc4b5f46700 1 --2- 172.21.15.160:0/2966193840 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4b0060220 0x7fc4b00f9150 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7fc4a4009fb0 tx=0x7fc4a4009b30).stop 2023-11-17T05:37:25.654 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:25.652+0000 7fc4b5f46700 1 -- 172.21.15.160:0/2966193840 shutdown_connections 2023-11-17T05:37:25.654 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:25.652+0000 7fc4b5f46700 1 --2- 172.21.15.160:0/2966193840 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4b0060220 0x7fc4b00f9150 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:25.654 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:25.652+0000 7fc4b5f46700 1 -- 172.21.15.160:0/2966193840 >> 172.21.15.160:0/2966193840 conn(0x7fc4b00f0b70 msgr2=0x7fc4b00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:37:25.654 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:25.652+0000 7fc4b5f46700 1 -- 172.21.15.160:0/2966193840 shutdown_connections 2023-11-17T05:37:25.654 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:25.652+0000 7fc4b5f46700 1 -- 172.21.15.160:0/2966193840 wait complete. 2023-11-17T05:37:25.654 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:25.653+0000 7fc4b5f46700 1 Processor -- start 2023-11-17T05:37:25.654 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:25.653+0000 7fc4b5f46700 1 -- start start 2023-11-17T05:37:25.655 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:25.653+0000 7fc4b5f46700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4b0060220 0x7fc4b00f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:37:25.655 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:25.653+0000 7fc4b5f46700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b00f5050 con 0x7fc4b0060220 2023-11-17T05:37:25.655 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:25.653+0000 7fc4af7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4b0060220 0x7fc4b00f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:37:25.655 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:25.653+0000 7fc4af7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4b0060220 0x7fc4b00f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:33790/0 (socket says 172.21.15.160:33790) 2023-11-17T05:37:25.655 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:25.653+0000 7fc4af7fe700 1 -- 172.21.15.160:0/1310114535 learned_addr learned my addr 172.21.15.160:0/1310114535 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:37:25.656 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:25.653+0000 7fc4af7fe700 1 -- 172.21.15.160:0/1310114535 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a4005040 con 0x7fc4b0060220 2023-11-17T05:37:25.656 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:25.654+0000 7fc4af7fe700 1 --2- 172.21.15.160:0/1310114535 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4b0060220 0x7fc4b00f6bd0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7fc4a4013510 tx=0x7fc4a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:37:25.656 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:25.654+0000 7fc497fff700 1 -- 172.21.15.160:0/1310114535 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4a4009320 con 0x7fc4b0060220 2023-11-17T05:37:25.656 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:25.654+0000 7fc4b5f46700 1 -- 172.21.15.160:0/1310114535 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4b00f51f0 con 0x7fc4b0060220 2023-11-17T05:37:25.656 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:25.654+0000 7fc497fff700 1 -- 172.21.15.160:0/1310114535 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc4a4013f00 con 0x7fc4b0060220 2023-11-17T05:37:25.657 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:25.654+0000 7fc497fff700 1 -- 172.21.15.160:0/1310114535 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4a4014dc0 con 0x7fc4b0060220 2023-11-17T05:37:25.657 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:25.654+0000 7fc4b5f46700 1 -- 172.21.15.160:0/1310114535 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4b00f5680 con 0x7fc4b0060220 2023-11-17T05:37:25.657 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:25.655+0000 7fc4b5f46700 1 -- 172.21.15.160:0/1310114535 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4b0059100 con 0x7fc4b0060220 2023-11-17T05:37:25.657 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:25.655+0000 7fc497fff700 1 -- 172.21.15.160:0/1310114535 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc4a4014420 con 0x7fc4b0060220 2023-11-17T05:37:25.657 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:25.655+0000 7fc497fff700 1 --2- 172.21.15.160:0/1310114535 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc49802ff60 0x7fc498032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:37:25.657 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:25.655+0000 7fc497fff700 1 -- 172.21.15.160:0/1310114535 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc4a40410a0 con 0x7fc4b0060220 2023-11-17T05:37:25.657 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:25.655+0000 7fc4aeffd700 1 --2- 172.21.15.160:0/1310114535 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc49802ff60 0x7fc498032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:37:25.657 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:25.655+0000 7fc4aeffd700 1 --2- 172.21.15.160:0/1310114535 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc49802ff60 0x7fc498032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fc4a00096a0 tx=0x7fc4a0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:37:25.661 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:25.660+0000 7fc497fff700 1 -- 172.21.15.160:0/1310114535 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc4a40095f0 con 0x7fc4b0060220 2023-11-17T05:37:25.869 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:25.868+0000 7fc4b5f46700 1 -- 172.21.15.160:0/1310114535 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc4b0045ad0 con 0x7fc4b0060220 2023-11-17T05:37:25.870 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:25.869+0000 7fc497fff700 1 -- 172.21.15.160:0/1310114535 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fc4a401cb10 con 0x7fc4b0060220 2023-11-17T05:37:25.870 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:37:25.870 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:37:25.871 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:25.871+0000 7fc4b5f46700 1 -- 172.21.15.160:0/1310114535 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc49802ff60 msgr2=0x7fc498032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:25.872 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:25.871+0000 7fc4b5f46700 1 --2- 172.21.15.160:0/1310114535 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc49802ff60 0x7fc498032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fc4a00096a0 tx=0x7fc4a0006b40).stop 2023-11-17T05:37:25.872 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:25.871+0000 7fc4b5f46700 1 -- 172.21.15.160:0/1310114535 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4b0060220 msgr2=0x7fc4b00f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:25.872 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:25.871+0000 7fc4b5f46700 1 --2- 172.21.15.160:0/1310114535 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4b0060220 0x7fc4b00f6bd0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7fc4a4013510 tx=0x7fc4a400b040).stop 2023-11-17T05:37:25.872 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:25.871+0000 7fc4b5f46700 1 -- 172.21.15.160:0/1310114535 shutdown_connections 2023-11-17T05:37:25.872 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:25.871+0000 7fc4b5f46700 1 --2- 172.21.15.160:0/1310114535 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc49802ff60 0x7fc498032420 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:25.872 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:25.871+0000 7fc4b5f46700 1 --2- 172.21.15.160:0/1310114535 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4b0060220 0x7fc4b00f6bd0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:25.872 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:25.871+0000 7fc4b5f46700 1 -- 172.21.15.160:0/1310114535 >> 172.21.15.160:0/1310114535 conn(0x7fc4b00f0b70 msgr2=0x7fc4b00f16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:37:25.872 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:25.871+0000 7fc4b5f46700 1 -- 172.21.15.160:0/1310114535 shutdown_connections 2023-11-17T05:37:25.872 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:25.871+0000 7fc4b5f46700 1 -- 172.21.15.160:0/1310114535 wait complete. 2023-11-17T05:37:25.873 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:37:26.270 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:26 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:37:25.870096+0000 mon.smithi003 (mon.0) 301 : audit [DBG] from='client.? 172.21.15.160:0/1310114535' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:37:27.272 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:27 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:37:26.374730+0000 mgr.smithi003.eddcuc (mgr.14164) 370 : cluster [DBG] pgmap v354: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:37:27.279 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:37:27.279 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:37:28.423 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:28.421+0000 7f59b9a10700 1 -- 172.21.15.160:0/2773910731 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f59b40f5410 msgr2=0x7f59b40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:28.423 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:28.421+0000 7f59b9a10700 1 --2- 172.21.15.160:0/2773910731 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f59b40f5410 0x7f59b40f5830 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f59a4009fb0 tx=0x7f59a4009b30).stop 2023-11-17T05:37:28.423 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:28.421+0000 7f59b9a10700 1 -- 172.21.15.160:0/2773910731 shutdown_connections 2023-11-17T05:37:28.424 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:28.421+0000 7f59b9a10700 1 --2- 172.21.15.160:0/2773910731 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f59b40f5410 0x7f59b40f5830 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:28.424 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:28.421+0000 7f59b9a10700 1 -- 172.21.15.160:0/2773910731 >> 172.21.15.160:0/2773910731 conn(0x7f59b40f0b70 msgr2=0x7f59b40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:37:28.424 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:28.421+0000 7f59b9a10700 1 -- 172.21.15.160:0/2773910731 shutdown_connections 2023-11-17T05:37:28.424 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:28.421+0000 7f59b9a10700 1 -- 172.21.15.160:0/2773910731 wait complete. 2023-11-17T05:37:28.424 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:28.422+0000 7f59b9a10700 1 Processor -- start 2023-11-17T05:37:28.424 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:28.422+0000 7f59b9a10700 1 -- start start 2023-11-17T05:37:28.424 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:28.422+0000 7f59b9a10700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f59b40f5410 0x7f59b410b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:37:28.424 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:28.422+0000 7f59b9a10700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59b41071e0 con 0x7f59b40f5410 2023-11-17T05:37:28.424 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:28.422+0000 7f59b2ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f59b40f5410 0x7f59b410b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:37:28.424 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:28.422+0000 7f59b2ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f59b40f5410 0x7f59b410b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:33812/0 (socket says 172.21.15.160:33812) 2023-11-17T05:37:28.424 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:28.422+0000 7f59b2ffd700 1 -- 172.21.15.160:0/651622377 learned_addr learned my addr 172.21.15.160:0/651622377 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:37:28.424 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:28.423+0000 7f59b2ffd700 1 -- 172.21.15.160:0/651622377 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a4005040 con 0x7f59b40f5410 2023-11-17T05:37:28.424 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:28.423+0000 7f59b2ffd700 1 --2- 172.21.15.160:0/651622377 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f59b40f5410 0x7f59b410b7f0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f59a4013510 tx=0x7f59a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:37:28.425 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:28.423+0000 7f599b7fe700 1 -- 172.21.15.160:0/651622377 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59a4009320 con 0x7f59b40f5410 2023-11-17T05:37:28.426 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:28.423+0000 7f59b9a10700 1 -- 172.21.15.160:0/651622377 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59b4060220 con 0x7f59b40f5410 2023-11-17T05:37:28.426 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:28.423+0000 7f599b7fe700 1 -- 172.21.15.160:0/651622377 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f59a4013f00 con 0x7f59b40f5410 2023-11-17T05:37:28.426 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:28.423+0000 7f599b7fe700 1 -- 172.21.15.160:0/651622377 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59a4014d90 con 0x7f59b40f5410 2023-11-17T05:37:28.426 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:28.423+0000 7f59b9a10700 1 -- 172.21.15.160:0/651622377 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59b410bce0 con 0x7f59b40f5410 2023-11-17T05:37:28.426 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:28.424+0000 7f599b7fe700 1 -- 172.21.15.160:0/651622377 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f59a40095f0 con 0x7f59b40f5410 2023-11-17T05:37:28.426 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:28.424+0000 7f59b9a10700 1 -- 172.21.15.160:0/651622377 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59b4045ad0 con 0x7f59b40f5410 2023-11-17T05:37:28.426 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:28.424+0000 7f599b7fe700 1 --2- 172.21.15.160:0/651622377 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f599c02ff60 0x7f599c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:37:28.426 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:28.424+0000 7f599b7fe700 1 -- 172.21.15.160:0/651622377 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f59a4040fe0 con 0x7f59b40f5410 2023-11-17T05:37:28.426 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:28.424+0000 7f59b27fc700 1 --2- 172.21.15.160:0/651622377 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f599c02ff60 0x7f599c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:37:28.426 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:28.425+0000 7f59b27fc700 1 --2- 172.21.15.160:0/651622377 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f599c02ff60 0x7f599c032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f59a80096a0 tx=0x7f59a8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:37:28.429 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:28.429+0000 7f599b7fe700 1 -- 172.21.15.160:0/651622377 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f59a4014420 con 0x7f59b40f5410 2023-11-17T05:37:28.526 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:37:27.558500+0000 mon.smithi003 (mon.0) 302 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.eddcuc/mirror_snapshot_schedule"}]: dispatch 2023-11-17T05:37:28.527 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:37:27.561633+0000 mon.smithi003 (mon.0) 303 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.eddcuc/trash_purge_schedule"}]: dispatch 2023-11-17T05:37:28.636 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:28.633+0000 7f59b9a10700 1 -- 172.21.15.160:0/651622377 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f59b410c4d0 con 0x7f59b40f5410 2023-11-17T05:37:28.636 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:28.634+0000 7f599b7fe700 1 -- 172.21.15.160:0/651622377 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f59a4010230 con 0x7f59b40f5410 2023-11-17T05:37:28.636 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:37:28.636 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:37:28.637 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:28.635+0000 7f59b9a10700 1 -- 172.21.15.160:0/651622377 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f599c02ff60 msgr2=0x7f599c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:28.637 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:28.635+0000 7f59b9a10700 1 --2- 172.21.15.160:0/651622377 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f599c02ff60 0x7f599c032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f59a80096a0 tx=0x7f59a8006b40).stop 2023-11-17T05:37:28.637 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:28.635+0000 7f59b9a10700 1 -- 172.21.15.160:0/651622377 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f59b40f5410 msgr2=0x7f59b410b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:28.637 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:28.635+0000 7f59b9a10700 1 --2- 172.21.15.160:0/651622377 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f59b40f5410 0x7f59b410b7f0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f59a4013510 tx=0x7f59a400b040).stop 2023-11-17T05:37:28.638 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:28.636+0000 7f59b9a10700 1 -- 172.21.15.160:0/651622377 shutdown_connections 2023-11-17T05:37:28.638 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:28.636+0000 7f59b9a10700 1 --2- 172.21.15.160:0/651622377 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f599c02ff60 0x7f599c032420 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:28.638 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:28.636+0000 7f59b9a10700 1 --2- 172.21.15.160:0/651622377 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f59b40f5410 0x7f59b410b7f0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:28.638 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:28.636+0000 7f59b9a10700 1 -- 172.21.15.160:0/651622377 >> 172.21.15.160:0/651622377 conn(0x7f59b40f0b70 msgr2=0x7f59b40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:37:28.638 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:28.636+0000 7f59b9a10700 1 -- 172.21.15.160:0/651622377 shutdown_connections 2023-11-17T05:37:28.638 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:28.636+0000 7f59b9a10700 1 -- 172.21.15.160:0/651622377 wait complete. 2023-11-17T05:37:28.638 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:37:29.520 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:29 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:37:28.375020+0000 mgr.smithi003.eddcuc (mgr.14164) 371 : cluster [DBG] pgmap v355: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:37:29.520 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:29 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:37:28.635233+0000 mon.smithi003 (mon.0) 304 : audit [DBG] from='client.? 172.21.15.160:0/651622377' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:37:30.018 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:37:30.018 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:37:31.160 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:31.159+0000 7f67259d9700 1 -- 172.21.15.160:0/2764761816 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67200f6510 msgr2=0x7f67200f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:31.161 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:31.159+0000 7f67259d9700 1 --2- 172.21.15.160:0/2764761816 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67200f6510 0x7f67200f6930 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f6714009fb0 tx=0x7f6714009b30).stop 2023-11-17T05:37:31.161 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:31.159+0000 7f67259d9700 1 -- 172.21.15.160:0/2764761816 shutdown_connections 2023-11-17T05:37:31.161 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:31.159+0000 7f67259d9700 1 --2- 172.21.15.160:0/2764761816 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67200f6510 0x7f67200f6930 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:31.161 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:31.159+0000 7f67259d9700 1 -- 172.21.15.160:0/2764761816 >> 172.21.15.160:0/2764761816 conn(0x7f67200f1c70 msgr2=0x7f67200f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:37:31.161 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:31.160+0000 7f67259d9700 1 -- 172.21.15.160:0/2764761816 shutdown_connections 2023-11-17T05:37:31.161 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:31.160+0000 7f67259d9700 1 -- 172.21.15.160:0/2764761816 wait complete. 2023-11-17T05:37:31.162 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:31.160+0000 7f67259d9700 1 Processor -- start 2023-11-17T05:37:31.162 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:31.160+0000 7f67259d9700 1 -- start start 2023-11-17T05:37:31.162 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:31.161+0000 7f67259d9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67200f6510 0x7f6720106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:37:31.163 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:31.161+0000 7f67259d9700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6720106c20 con 0x7f67200f6510 2023-11-17T05:37:31.163 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:31.161+0000 7f671effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67200f6510 0x7f6720106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:37:31.163 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:31.161+0000 7f671effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67200f6510 0x7f6720106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:47572/0 (socket says 172.21.15.160:47572) 2023-11-17T05:37:31.163 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:31.161+0000 7f671effd700 1 -- 172.21.15.160:0/3086203170 learned_addr learned my addr 172.21.15.160:0/3086203170 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:37:31.163 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:31.161+0000 7f671effd700 1 -- 172.21.15.160:0/3086203170 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6714005040 con 0x7f67200f6510 2023-11-17T05:37:31.163 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:31.161+0000 7f671effd700 1 --2- 172.21.15.160:0/3086203170 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67200f6510 0x7f6720106730 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f6714014000 tx=0x7f671400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:37:31.163 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:31.162+0000 7f67077fe700 1 -- 172.21.15.160:0/3086203170 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67140095f0 con 0x7f67200f6510 2023-11-17T05:37:31.163 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:31.162+0000 7f67259d9700 1 -- 172.21.15.160:0/3086203170 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6720106e20 con 0x7f67200f6510 2023-11-17T05:37:31.163 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:31.162+0000 7f67077fe700 1 -- 172.21.15.160:0/3086203170 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6714007e70 con 0x7f67200f6510 2023-11-17T05:37:31.163 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:31.162+0000 7f67077fe700 1 -- 172.21.15.160:0/3086203170 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6714014ea0 con 0x7f67200f6510 2023-11-17T05:37:31.163 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:31.162+0000 7f67259d9700 1 -- 172.21.15.160:0/3086203170 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67201072b0 con 0x7f67200f6510 2023-11-17T05:37:31.164 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:31.162+0000 7f67077fe700 1 -- 172.21.15.160:0/3086203170 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f67140079c0 con 0x7f67200f6510 2023-11-17T05:37:31.164 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:31.162+0000 7f67259d9700 1 -- 172.21.15.160:0/3086203170 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6720045ad0 con 0x7f67200f6510 2023-11-17T05:37:31.164 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:31.163+0000 7f67077fe700 1 --2- 172.21.15.160:0/3086203170 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f670802ff60 0x7f6708032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:37:31.164 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:31.163+0000 7f67077fe700 1 -- 172.21.15.160:0/3086203170 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6714040860 con 0x7f67200f6510 2023-11-17T05:37:31.164 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:31.163+0000 7f671e7fc700 1 --2- 172.21.15.160:0/3086203170 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f670802ff60 0x7f6708032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:37:31.164 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:31.163+0000 7f671e7fc700 1 --2- 172.21.15.160:0/3086203170 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f670802ff60 0x7f6708032420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f67100096a0 tx=0x7f6710006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:37:31.168 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:31.167+0000 7f67077fe700 1 -- 172.21.15.160:0/3086203170 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f671400f950 con 0x7f67200f6510 2023-11-17T05:37:31.373 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:31.372+0000 7f67259d9700 1 -- 172.21.15.160:0/3086203170 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f67200fbf60 con 0x7f67200f6510 2023-11-17T05:37:31.373 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:31.373+0000 7f67077fe700 1 -- 172.21.15.160:0/3086203170 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f6714010230 con 0x7f67200f6510 2023-11-17T05:37:31.374 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:37:31.374 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:37:31.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:31.374+0000 7f67259d9700 1 -- 172.21.15.160:0/3086203170 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f670802ff60 msgr2=0x7f6708032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:31.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:31.374+0000 7f67259d9700 1 --2- 172.21.15.160:0/3086203170 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f670802ff60 0x7f6708032420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f67100096a0 tx=0x7f6710006b40).stop 2023-11-17T05:37:31.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:31.374+0000 7f67259d9700 1 -- 172.21.15.160:0/3086203170 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67200f6510 msgr2=0x7f6720106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:31.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:31.374+0000 7f67259d9700 1 --2- 172.21.15.160:0/3086203170 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67200f6510 0x7f6720106730 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f6714014000 tx=0x7f671400b040).stop 2023-11-17T05:37:31.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:31.374+0000 7f67259d9700 1 -- 172.21.15.160:0/3086203170 shutdown_connections 2023-11-17T05:37:31.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:31.374+0000 7f67259d9700 1 --2- 172.21.15.160:0/3086203170 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f670802ff60 0x7f6708032420 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:31.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:31.374+0000 7f67259d9700 1 --2- 172.21.15.160:0/3086203170 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67200f6510 0x7f6720106730 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:31.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:31.374+0000 7f67259d9700 1 -- 172.21.15.160:0/3086203170 >> 172.21.15.160:0/3086203170 conn(0x7f67200f1c70 msgr2=0x7f67200f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:37:31.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:31.374+0000 7f67259d9700 1 -- 172.21.15.160:0/3086203170 shutdown_connections 2023-11-17T05:37:31.376 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:31.374+0000 7f67259d9700 1 -- 172.21.15.160:0/3086203170 wait complete. 2023-11-17T05:37:31.376 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:37:31.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:31 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:37:30.375348+0000 mgr.smithi003.eddcuc (mgr.14164) 372 : cluster [DBG] pgmap v356: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:37:31.771 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:31 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:37:31.373850+0000 mon.smithi003 (mon.0) 305 : audit [DBG] from='client.? 172.21.15.160:0/3086203170' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:37:32.767 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:37:32.767 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:37:33.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:37:32.375641+0000 mgr.smithi003.eddcuc (mgr.14164) 373 : cluster [DBG] pgmap v357: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:37:33.919 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:33.917+0000 7f2c52c10700 1 -- 172.21.15.160:0/3328841229 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c4c0f6510 msgr2=0x7f2c4c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:33.919 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:33.917+0000 7f2c52c10700 1 --2- 172.21.15.160:0/3328841229 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c4c0f6510 0x7f2c4c0f6930 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f2c40009fb0 tx=0x7f2c40009b30).stop 2023-11-17T05:37:33.919 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:33.917+0000 7f2c52c10700 1 -- 172.21.15.160:0/3328841229 shutdown_connections 2023-11-17T05:37:33.919 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:33.917+0000 7f2c52c10700 1 --2- 172.21.15.160:0/3328841229 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c4c0f6510 0x7f2c4c0f6930 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:33.919 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:33.917+0000 7f2c52c10700 1 -- 172.21.15.160:0/3328841229 >> 172.21.15.160:0/3328841229 conn(0x7f2c4c0f1c70 msgr2=0x7f2c4c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:37:33.919 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:33.917+0000 7f2c52c10700 1 -- 172.21.15.160:0/3328841229 shutdown_connections 2023-11-17T05:37:33.919 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:33.917+0000 7f2c52c10700 1 -- 172.21.15.160:0/3328841229 wait complete. 2023-11-17T05:37:33.919 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:33.918+0000 7f2c52c10700 1 Processor -- start 2023-11-17T05:37:33.919 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:33.918+0000 7f2c52c10700 1 -- start start 2023-11-17T05:37:33.919 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:33.918+0000 7f2c52c10700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c4c0f6510 0x7f2c4c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:37:33.920 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:33.918+0000 7f2c52c10700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4c106c20 con 0x7f2c4c0f6510 2023-11-17T05:37:33.921 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:33.918+0000 7f2c509ac700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c4c0f6510 0x7f2c4c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:37:33.921 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:33.918+0000 7f2c509ac700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c4c0f6510 0x7f2c4c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:47586/0 (socket says 172.21.15.160:47586) 2023-11-17T05:37:33.921 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:33.918+0000 7f2c509ac700 1 -- 172.21.15.160:0/2330917688 learned_addr learned my addr 172.21.15.160:0/2330917688 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:37:33.921 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:33.919+0000 7f2c509ac700 1 -- 172.21.15.160:0/2330917688 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c40005040 con 0x7f2c4c0f6510 2023-11-17T05:37:33.921 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:33.919+0000 7f2c509ac700 1 --2- 172.21.15.160:0/2330917688 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c4c0f6510 0x7f2c4c106730 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f2c40013510 tx=0x7f2c4000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:37:33.921 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:33.919+0000 7f2c497fa700 1 -- 172.21.15.160:0/2330917688 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c40009320 con 0x7f2c4c0f6510 2023-11-17T05:37:33.921 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:33.919+0000 7f2c52c10700 1 -- 172.21.15.160:0/2330917688 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c4c106e20 con 0x7f2c4c0f6510 2023-11-17T05:37:33.922 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:33.919+0000 7f2c52c10700 1 -- 172.21.15.160:0/2330917688 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c4c1072b0 con 0x7f2c4c0f6510 2023-11-17T05:37:33.922 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:33.919+0000 7f2c497fa700 1 -- 172.21.15.160:0/2330917688 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2c40013f00 con 0x7f2c4c0f6510 2023-11-17T05:37:33.922 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:33.919+0000 7f2c497fa700 1 -- 172.21.15.160:0/2330917688 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c40014d90 con 0x7f2c4c0f6510 2023-11-17T05:37:33.922 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:33.920+0000 7f2c497fa700 1 -- 172.21.15.160:0/2330917688 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2c400095f0 con 0x7f2c4c0f6510 2023-11-17T05:37:33.922 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:33.920+0000 7f2c52c10700 1 -- 172.21.15.160:0/2330917688 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c4c045ad0 con 0x7f2c4c0f6510 2023-11-17T05:37:33.922 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:33.920+0000 7f2c497fa700 1 --2- 172.21.15.160:0/2330917688 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f2c3c02ff60 0x7f2c3c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:37:33.922 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:33.920+0000 7f2c497fa700 1 -- 172.21.15.160:0/2330917688 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2c400410a0 con 0x7f2c4c0f6510 2023-11-17T05:37:33.922 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:33.920+0000 7f2c4bfff700 1 --2- 172.21.15.160:0/2330917688 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f2c3c02ff60 0x7f2c3c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:37:33.922 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:33.921+0000 7f2c4bfff700 1 --2- 172.21.15.160:0/2330917688 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f2c3c02ff60 0x7f2c3c032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f2c380096a0 tx=0x7f2c38006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:37:33.926 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:33.925+0000 7f2c497fa700 1 -- 172.21.15.160:0/2330917688 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2c40010230 con 0x7f2c4c0f6510 2023-11-17T05:37:34.133 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:34.132+0000 7f2c52c10700 1 -- 172.21.15.160:0/2330917688 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2c4c0fbf60 con 0x7f2c4c0f6510 2023-11-17T05:37:34.133 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:34.132+0000 7f2c497fa700 1 -- 172.21.15.160:0/2330917688 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f2c40010230 con 0x7f2c4c0f6510 2023-11-17T05:37:34.134 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:37:34.134 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:37:34.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:34.134+0000 7f2c52c10700 1 -- 172.21.15.160:0/2330917688 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f2c3c02ff60 msgr2=0x7f2c3c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:34.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:34.134+0000 7f2c52c10700 1 --2- 172.21.15.160:0/2330917688 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f2c3c02ff60 0x7f2c3c032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f2c380096a0 tx=0x7f2c38006b40).stop 2023-11-17T05:37:34.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:34.134+0000 7f2c52c10700 1 -- 172.21.15.160:0/2330917688 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c4c0f6510 msgr2=0x7f2c4c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:34.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:34.134+0000 7f2c52c10700 1 --2- 172.21.15.160:0/2330917688 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c4c0f6510 0x7f2c4c106730 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f2c40013510 tx=0x7f2c4000b040).stop 2023-11-17T05:37:34.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:34.134+0000 7f2c52c10700 1 -- 172.21.15.160:0/2330917688 shutdown_connections 2023-11-17T05:37:34.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:34.134+0000 7f2c52c10700 1 --2- 172.21.15.160:0/2330917688 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f2c3c02ff60 0x7f2c3c032420 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:34.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:34.134+0000 7f2c52c10700 1 --2- 172.21.15.160:0/2330917688 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c4c0f6510 0x7f2c4c106730 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:34.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:34.134+0000 7f2c52c10700 1 -- 172.21.15.160:0/2330917688 >> 172.21.15.160:0/2330917688 conn(0x7f2c4c0f1c70 msgr2=0x7f2c4c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:37:34.135 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:34.134+0000 7f2c52c10700 1 -- 172.21.15.160:0/2330917688 shutdown_connections 2023-11-17T05:37:34.136 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:34.134+0000 7f2c52c10700 1 -- 172.21.15.160:0/2330917688 wait complete. 2023-11-17T05:37:34.136 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:37:35.520 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:35 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:37:34.133643+0000 mon.smithi003 (mon.0) 306 : audit [DBG] from='client.? 172.21.15.160:0/2330917688' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:37:35.520 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:35 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:37:34.375989+0000 mgr.smithi003.eddcuc (mgr.14164) 374 : cluster [DBG] pgmap v358: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:37:35.547 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:37:35.548 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:37:36.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:36.708+0000 7f1969bb4700 1 -- 172.21.15.160:0/66589639 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19640f6510 msgr2=0x7f19640f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:36.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:36.708+0000 7f1969bb4700 1 --2- 172.21.15.160:0/66589639 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19640f6510 0x7f19640f6930 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f194c009fb0 tx=0x7f194c009b30).stop 2023-11-17T05:37:36.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:36.708+0000 7f1969bb4700 1 -- 172.21.15.160:0/66589639 shutdown_connections 2023-11-17T05:37:36.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:36.708+0000 7f1969bb4700 1 --2- 172.21.15.160:0/66589639 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19640f6510 0x7f19640f6930 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:36.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:36.708+0000 7f1969bb4700 1 -- 172.21.15.160:0/66589639 >> 172.21.15.160:0/66589639 conn(0x7f19640f1c70 msgr2=0x7f19640f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:37:36.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:36.708+0000 7f1969bb4700 1 -- 172.21.15.160:0/66589639 shutdown_connections 2023-11-17T05:37:36.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:36.708+0000 7f1969bb4700 1 -- 172.21.15.160:0/66589639 wait complete. 2023-11-17T05:37:36.714 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:36.709+0000 7f1969bb4700 1 Processor -- start 2023-11-17T05:37:36.715 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:36.709+0000 7f1969bb4700 1 -- start start 2023-11-17T05:37:36.715 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:36.709+0000 7f1969bb4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19640f6510 0x7f196406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:37:36.715 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:36.709+0000 7f1969bb4700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1964069e20 con 0x7f19640f6510 2023-11-17T05:37:36.715 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:36.710+0000 7f19637fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19640f6510 0x7f196406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:37:36.715 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:36.710+0000 7f19637fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19640f6510 0x7f196406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:47608/0 (socket says 172.21.15.160:47608) 2023-11-17T05:37:36.715 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:36.710+0000 7f19637fe700 1 -- 172.21.15.160:0/3932038385 learned_addr learned my addr 172.21.15.160:0/3932038385 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:37:36.715 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:36.710+0000 7f19637fe700 1 -- 172.21.15.160:0/3932038385 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f194c005040 con 0x7f19640f6510 2023-11-17T05:37:36.715 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:36.710+0000 7f19637fe700 1 --2- 172.21.15.160:0/3932038385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19640f6510 0x7f196406b9a0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f194c013510 tx=0x7f194c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:37:36.715 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:36.710+0000 7f195bfff700 1 -- 172.21.15.160:0/3932038385 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f194c009320 con 0x7f19640f6510 2023-11-17T05:37:36.715 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:36.710+0000 7f1969bb4700 1 -- 172.21.15.160:0/3932038385 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1964069fc0 con 0x7f19640f6510 2023-11-17T05:37:36.716 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:36.710+0000 7f195bfff700 1 -- 172.21.15.160:0/3932038385 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f194c013f00 con 0x7f19640f6510 2023-11-17T05:37:36.716 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:36.710+0000 7f195bfff700 1 -- 172.21.15.160:0/3932038385 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f194c014d90 con 0x7f19640f6510 2023-11-17T05:37:36.716 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:36.710+0000 7f1969bb4700 1 -- 172.21.15.160:0/3932038385 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f196406a450 con 0x7f19640f6510 2023-11-17T05:37:36.716 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:36.711+0000 7f195bfff700 1 -- 172.21.15.160:0/3932038385 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f194c0263a0 con 0x7f19640f6510 2023-11-17T05:37:36.716 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:36.711+0000 7f1969bb4700 1 -- 172.21.15.160:0/3932038385 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1964059100 con 0x7f19640f6510 2023-11-17T05:37:36.716 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:36.711+0000 7f195bfff700 1 --2- 172.21.15.160:0/3932038385 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f195002ff60 0x7f1950032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:37:36.716 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:36.711+0000 7f195bfff700 1 -- 172.21.15.160:0/3932038385 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f194c0410c0 con 0x7f19640f6510 2023-11-17T05:37:36.716 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:36.711+0000 7f1962ffd700 1 --2- 172.21.15.160:0/3932038385 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f195002ff60 0x7f1950032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:37:36.716 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:36.712+0000 7f1962ffd700 1 --2- 172.21.15.160:0/3932038385 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f195002ff60 0x7f1950032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f19540096a0 tx=0x7f1954006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:37:36.717 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:36.716+0000 7f195bfff700 1 -- 172.21.15.160:0/3932038385 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f194c0095f0 con 0x7f19640f6510 2023-11-17T05:37:36.922 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:36.921+0000 7f1969bb4700 1 -- 172.21.15.160:0/3932038385 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f19640fbf60 con 0x7f19640f6510 2023-11-17T05:37:36.923 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:36.922+0000 7f195bfff700 1 -- 172.21.15.160:0/3932038385 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f194c01db70 con 0x7f19640f6510 2023-11-17T05:37:36.923 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:37:36.923 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:37:36.925 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:36.924+0000 7f1969bb4700 1 -- 172.21.15.160:0/3932038385 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f195002ff60 msgr2=0x7f1950032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:36.925 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:36.924+0000 7f1969bb4700 1 --2- 172.21.15.160:0/3932038385 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f195002ff60 0x7f1950032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f19540096a0 tx=0x7f1954006b40).stop 2023-11-17T05:37:36.925 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:36.924+0000 7f1969bb4700 1 -- 172.21.15.160:0/3932038385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19640f6510 msgr2=0x7f196406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:36.925 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:36.924+0000 7f1969bb4700 1 --2- 172.21.15.160:0/3932038385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19640f6510 0x7f196406b9a0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f194c013510 tx=0x7f194c00b040).stop 2023-11-17T05:37:36.925 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:36.924+0000 7f1969bb4700 1 -- 172.21.15.160:0/3932038385 shutdown_connections 2023-11-17T05:37:36.925 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:36.924+0000 7f1969bb4700 1 --2- 172.21.15.160:0/3932038385 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f195002ff60 0x7f1950032420 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:36.925 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:36.924+0000 7f1969bb4700 1 --2- 172.21.15.160:0/3932038385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19640f6510 0x7f196406b9a0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:36.925 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:36.924+0000 7f1969bb4700 1 -- 172.21.15.160:0/3932038385 >> 172.21.15.160:0/3932038385 conn(0x7f19640f1c70 msgr2=0x7f19640f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:37:36.925 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:36.924+0000 7f1969bb4700 1 -- 172.21.15.160:0/3932038385 shutdown_connections 2023-11-17T05:37:36.925 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:36.924+0000 7f1969bb4700 1 -- 172.21.15.160:0/3932038385 wait complete. 2023-11-17T05:37:36.926 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:37:37.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:37 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:37:36.376454+0000 mgr.smithi003.eddcuc (mgr.14164) 375 : cluster [DBG] pgmap v359: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:37:37.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:37 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:37:36.923155+0000 mon.smithi003 (mon.0) 307 : audit [DBG] from='client.? 172.21.15.160:0/3932038385' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:37:38.310 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:37:38.311 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:37:39.596 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:39.595+0000 7f006a582700 1 -- 172.21.15.160:0/3826201992 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f00640f6510 msgr2=0x7f00640f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:39.596 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:39.595+0000 7f006a582700 1 --2- 172.21.15.160:0/3826201992 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f00640f6510 0x7f00640f6930 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f0058009fb0 tx=0x7f0058009b30).stop 2023-11-17T05:37:39.596 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:39.595+0000 7f006a582700 1 -- 172.21.15.160:0/3826201992 shutdown_connections 2023-11-17T05:37:39.596 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:39.595+0000 7f006a582700 1 --2- 172.21.15.160:0/3826201992 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f00640f6510 0x7f00640f6930 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:39.596 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:39.595+0000 7f006a582700 1 -- 172.21.15.160:0/3826201992 >> 172.21.15.160:0/3826201992 conn(0x7f00640f1c70 msgr2=0x7f00640f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:37:39.596 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:39.595+0000 7f006a582700 1 -- 172.21.15.160:0/3826201992 shutdown_connections 2023-11-17T05:37:39.596 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:39.595+0000 7f006a582700 1 -- 172.21.15.160:0/3826201992 wait complete. 2023-11-17T05:37:39.598 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:39.595+0000 7f006a582700 1 Processor -- start 2023-11-17T05:37:39.598 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:39.596+0000 7f006a582700 1 -- start start 2023-11-17T05:37:39.598 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:39.596+0000 7f006a582700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0064106840 0x7f0064106c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:37:39.598 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:39.596+0000 7f006a582700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0064107150 con 0x7f0064106840 2023-11-17T05:37:39.598 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:39.596+0000 7f0063fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0064106840 0x7f0064106c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:37:39.598 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:39.596+0000 7f0063fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0064106840 0x7f0064106c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:47616/0 (socket says 172.21.15.160:47616) 2023-11-17T05:37:39.598 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:39.596+0000 7f0063fff700 1 -- 172.21.15.160:0/1844157590 learned_addr learned my addr 172.21.15.160:0/1844157590 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:37:39.598 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:39.596+0000 7f0063fff700 1 -- 172.21.15.160:0/1844157590 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0058005040 con 0x7f0064106840 2023-11-17T05:37:39.598 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:39.596+0000 7f0063fff700 1 --2- 172.21.15.160:0/1844157590 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0064106840 0x7f0064106c60 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f0058009f80 tx=0x7f0058009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:37:39.598 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:39.596+0000 7f0060ff9700 1 -- 172.21.15.160:0/1844157590 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f00580136c0 con 0x7f0064106840 2023-11-17T05:37:39.598 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:39.597+0000 7f0060ff9700 1 -- 172.21.15.160:0/1844157590 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f00580095f0 con 0x7f0064106840 2023-11-17T05:37:39.599 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:39.597+0000 7f006a582700 1 -- 172.21.15.160:0/1844157590 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0064107350 con 0x7f0064106840 2023-11-17T05:37:39.599 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:39.597+0000 7f0060ff9700 1 -- 172.21.15.160:0/1844157590 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0058014420 con 0x7f0064106840 2023-11-17T05:37:39.599 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:39.597+0000 7f006a582700 1 -- 172.21.15.160:0/1844157590 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0064107ea0 con 0x7f0064106840 2023-11-17T05:37:39.600 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:39.597+0000 7f0060ff9700 1 -- 172.21.15.160:0/1844157590 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0058025060 con 0x7f0064106840 2023-11-17T05:37:39.600 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:39.597+0000 7f006a582700 1 -- 172.21.15.160:0/1844157590 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0064059100 con 0x7f0064106840 2023-11-17T05:37:39.600 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:39.598+0000 7f0060ff9700 1 --2- 172.21.15.160:0/1844157590 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f004c02ffb0 0x7f004c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:37:39.600 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:39.598+0000 7f0060ff9700 1 -- 172.21.15.160:0/1844157590 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0058040e10 con 0x7f0064106840 2023-11-17T05:37:39.600 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:39.598+0000 7f00637fe700 1 --2- 172.21.15.160:0/1844157590 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f004c02ffb0 0x7f004c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:37:39.600 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:39.598+0000 7f00637fe700 1 --2- 172.21.15.160:0/1844157590 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f004c02ffb0 0x7f004c032470 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f00540096a0 tx=0x7f0054006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:37:39.604 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:39.602+0000 7f0060ff9700 1 -- 172.21.15.160:0/1844157590 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0058044030 con 0x7f0064106840 2023-11-17T05:37:39.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:39 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:37:38.376837+0000 mgr.smithi003.eddcuc (mgr.14164) 376 : cluster [DBG] pgmap v360: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:37:39.810 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:39.810+0000 7f006a582700 1 -- 172.21.15.160:0/1844157590 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0064108660 con 0x7f0064106840 2023-11-17T05:37:39.811 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:39.810+0000 7f0060ff9700 1 -- 172.21.15.160:0/1844157590 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f005801e960 con 0x7f0064106840 2023-11-17T05:37:39.811 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:37:39.811 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:37:39.813 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:39.812+0000 7f006a582700 1 -- 172.21.15.160:0/1844157590 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f004c02ffb0 msgr2=0x7f004c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:39.813 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:39.812+0000 7f006a582700 1 --2- 172.21.15.160:0/1844157590 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f004c02ffb0 0x7f004c032470 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f00540096a0 tx=0x7f0054006b40).stop 2023-11-17T05:37:39.813 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:39.812+0000 7f006a582700 1 -- 172.21.15.160:0/1844157590 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0064106840 msgr2=0x7f0064106c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:39.813 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:39.812+0000 7f006a582700 1 --2- 172.21.15.160:0/1844157590 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0064106840 0x7f0064106c60 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f0058009f80 tx=0x7f0058009b30).stop 2023-11-17T05:37:39.814 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:39.812+0000 7f006a582700 1 -- 172.21.15.160:0/1844157590 shutdown_connections 2023-11-17T05:37:39.814 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:39.812+0000 7f006a582700 1 --2- 172.21.15.160:0/1844157590 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f004c02ffb0 0x7f004c032470 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:39.814 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:39.812+0000 7f006a582700 1 --2- 172.21.15.160:0/1844157590 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0064106840 0x7f0064106c60 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:39.814 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:39.812+0000 7f006a582700 1 -- 172.21.15.160:0/1844157590 >> 172.21.15.160:0/1844157590 conn(0x7f00640f1c70 msgr2=0x7f00640f27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:37:39.814 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:39.812+0000 7f006a582700 1 -- 172.21.15.160:0/1844157590 shutdown_connections 2023-11-17T05:37:39.814 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:39.812+0000 7f006a582700 1 -- 172.21.15.160:0/1844157590 wait complete. 2023-11-17T05:37:39.815 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:37:40.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:40 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:37:39.811578+0000 mon.smithi003 (mon.0) 308 : audit [DBG] from='client.? 172.21.15.160:0/1844157590' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:37:41.167 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:37:41.167 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:37:41.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:41 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:37:40.377122+0000 mgr.smithi003.eddcuc (mgr.14164) 377 : cluster [DBG] pgmap v361: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:37:42.312 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:42.310+0000 7f5f04fca700 1 -- 172.21.15.160:0/1989149137 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f000f5410 msgr2=0x7f5f000f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:42.312 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:42.310+0000 7f5f04fca700 1 --2- 172.21.15.160:0/1989149137 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f000f5410 0x7f5f000f5830 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f5ee8009fb0 tx=0x7f5ee8009b30).stop 2023-11-17T05:37:42.312 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:42.310+0000 7f5f04fca700 1 -- 172.21.15.160:0/1989149137 shutdown_connections 2023-11-17T05:37:42.312 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:42.310+0000 7f5f04fca700 1 --2- 172.21.15.160:0/1989149137 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f000f5410 0x7f5f000f5830 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:42.312 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:42.310+0000 7f5f04fca700 1 -- 172.21.15.160:0/1989149137 >> 172.21.15.160:0/1989149137 conn(0x7f5f000f0b70 msgr2=0x7f5f000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:37:42.312 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:42.310+0000 7f5f04fca700 1 -- 172.21.15.160:0/1989149137 shutdown_connections 2023-11-17T05:37:42.312 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:42.310+0000 7f5f04fca700 1 -- 172.21.15.160:0/1989149137 wait complete. 2023-11-17T05:37:42.312 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:42.311+0000 7f5f04fca700 1 Processor -- start 2023-11-17T05:37:42.312 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:42.311+0000 7f5f04fca700 1 -- start start 2023-11-17T05:37:42.312 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:42.311+0000 7f5f04fca700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f000f5410 0x7f5f00106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:37:42.312 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:42.311+0000 7f5f04fca700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f00106c10 con 0x7f5f000f5410 2023-11-17T05:37:42.314 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:42.311+0000 7f5efe59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f000f5410 0x7f5f00106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:37:42.314 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:42.311+0000 7f5efe59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f000f5410 0x7f5f00106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:52536/0 (socket says 172.21.15.160:52536) 2023-11-17T05:37:42.314 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:42.311+0000 7f5efe59c700 1 -- 172.21.15.160:0/526560305 learned_addr learned my addr 172.21.15.160:0/526560305 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:37:42.314 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:42.312+0000 7f5efe59c700 1 -- 172.21.15.160:0/526560305 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee8005040 con 0x7f5f000f5410 2023-11-17T05:37:42.314 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:42.312+0000 7f5efe59c700 1 --2- 172.21.15.160:0/526560305 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f000f5410 0x7f5f00106720 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f5ee8013510 tx=0x7f5ee800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:37:42.314 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:42.312+0000 7f5ef6ffd700 1 -- 172.21.15.160:0/526560305 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ee8009320 con 0x7f5f000f5410 2023-11-17T05:37:42.314 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:42.312+0000 7f5f04fca700 1 -- 172.21.15.160:0/526560305 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f00106e10 con 0x7f5f000f5410 2023-11-17T05:37:42.315 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:42.312+0000 7f5ef6ffd700 1 -- 172.21.15.160:0/526560305 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5ee8013f00 con 0x7f5f000f5410 2023-11-17T05:37:42.315 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:42.312+0000 7f5ef6ffd700 1 -- 172.21.15.160:0/526560305 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ee8014d90 con 0x7f5f000f5410 2023-11-17T05:37:42.315 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:42.312+0000 7f5f04fca700 1 -- 172.21.15.160:0/526560305 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f001072a0 con 0x7f5f000f5410 2023-11-17T05:37:42.315 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:42.313+0000 7f5ef6ffd700 1 -- 172.21.15.160:0/526560305 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5ee80095f0 con 0x7f5f000f5410 2023-11-17T05:37:42.315 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:42.313+0000 7f5f04fca700 1 -- 172.21.15.160:0/526560305 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f00045ad0 con 0x7f5f000f5410 2023-11-17T05:37:42.315 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:42.313+0000 7f5ef6ffd700 1 --2- 172.21.15.160:0/526560305 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f5eec02ff60 0x7f5eec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:37:42.315 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:42.313+0000 7f5ef6ffd700 1 -- 172.21.15.160:0/526560305 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5ee80410c0 con 0x7f5f000f5410 2023-11-17T05:37:42.315 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:42.313+0000 7f5efdd9b700 1 --2- 172.21.15.160:0/526560305 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f5eec02ff60 0x7f5eec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:37:42.315 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:42.313+0000 7f5efdd9b700 1 --2- 172.21.15.160:0/526560305 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f5eec02ff60 0x7f5eec032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f5ef00096a0 tx=0x7f5ef0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:37:42.318 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:42.318+0000 7f5ef6ffd700 1 -- 172.21.15.160:0/526560305 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5ee8014420 con 0x7f5f000f5410 2023-11-17T05:37:42.529 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:42.525+0000 7f5f04fca700 1 -- 172.21.15.160:0/526560305 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5f00108150 con 0x7f5f000f5410 2023-11-17T05:37:42.529 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:42.526+0000 7f5ef6ffd700 1 -- 172.21.15.160:0/526560305 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f5ee8010230 con 0x7f5f000f5410 2023-11-17T05:37:42.530 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:37:42.530 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:37:42.530 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:42.527+0000 7f5f04fca700 1 -- 172.21.15.160:0/526560305 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f5eec02ff60 msgr2=0x7f5eec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:42.530 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:42.527+0000 7f5f04fca700 1 --2- 172.21.15.160:0/526560305 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f5eec02ff60 0x7f5eec032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f5ef00096a0 tx=0x7f5ef0006b40).stop 2023-11-17T05:37:42.530 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:42.527+0000 7f5f04fca700 1 -- 172.21.15.160:0/526560305 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f000f5410 msgr2=0x7f5f00106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:42.531 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:42.527+0000 7f5f04fca700 1 --2- 172.21.15.160:0/526560305 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f000f5410 0x7f5f00106720 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f5ee8013510 tx=0x7f5ee800b040).stop 2023-11-17T05:37:42.531 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:42.528+0000 7f5f04fca700 1 -- 172.21.15.160:0/526560305 shutdown_connections 2023-11-17T05:37:42.531 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:42.528+0000 7f5f04fca700 1 --2- 172.21.15.160:0/526560305 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f5eec02ff60 0x7f5eec032420 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:42.531 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:42.528+0000 7f5f04fca700 1 --2- 172.21.15.160:0/526560305 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f000f5410 0x7f5f00106720 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:42.531 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:42.528+0000 7f5f04fca700 1 -- 172.21.15.160:0/526560305 >> 172.21.15.160:0/526560305 conn(0x7f5f000f0b70 msgr2=0x7f5f0005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:37:42.531 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:42.528+0000 7f5f04fca700 1 -- 172.21.15.160:0/526560305 shutdown_connections 2023-11-17T05:37:42.531 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:42.528+0000 7f5f04fca700 1 -- 172.21.15.160:0/526560305 wait complete. 2023-11-17T05:37:42.531 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:37:43.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:43 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:37:42.377431+0000 mgr.smithi003.eddcuc (mgr.14164) 378 : cluster [DBG] pgmap v362: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:37:43.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:43 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:37:42.527113+0000 mon.smithi003 (mon.0) 309 : audit [DBG] from='client.? 172.21.15.160:0/526560305' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:37:43.906 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:37:43.906 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:37:45.176 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:45.169+0000 7f014a0dc700 1 -- 172.21.15.160:0/376705950 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01440f6510 msgr2=0x7f01440f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:45.176 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:45.169+0000 7f014a0dc700 1 --2- 172.21.15.160:0/376705950 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01440f6510 0x7f01440f6930 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f012c009fb0 tx=0x7f012c009b30).stop 2023-11-17T05:37:45.176 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:45.169+0000 7f014a0dc700 1 -- 172.21.15.160:0/376705950 shutdown_connections 2023-11-17T05:37:45.176 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:45.169+0000 7f014a0dc700 1 --2- 172.21.15.160:0/376705950 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01440f6510 0x7f01440f6930 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:45.176 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:45.169+0000 7f014a0dc700 1 -- 172.21.15.160:0/376705950 >> 172.21.15.160:0/376705950 conn(0x7f01440f1c70 msgr2=0x7f01440f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:37:45.176 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:45.169+0000 7f014a0dc700 1 -- 172.21.15.160:0/376705950 shutdown_connections 2023-11-17T05:37:45.176 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:45.169+0000 7f014a0dc700 1 -- 172.21.15.160:0/376705950 wait complete. 2023-11-17T05:37:45.177 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:45.170+0000 7f014a0dc700 1 Processor -- start 2023-11-17T05:37:45.177 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:45.170+0000 7f014a0dc700 1 -- start start 2023-11-17T05:37:45.177 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:45.170+0000 7f014a0dc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01440f6510 0x7f014406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:37:45.177 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:45.170+0000 7f014a0dc700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0144069e20 con 0x7f01440f6510 2023-11-17T05:37:45.177 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:45.170+0000 7f01437fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01440f6510 0x7f014406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:37:45.177 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:45.170+0000 7f01437fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01440f6510 0x7f014406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:52552/0 (socket says 172.21.15.160:52552) 2023-11-17T05:37:45.177 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:45.170+0000 7f01437fe700 1 -- 172.21.15.160:0/2553152218 learned_addr learned my addr 172.21.15.160:0/2553152218 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:37:45.177 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:45.171+0000 7f01437fe700 1 -- 172.21.15.160:0/2553152218 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f012c005040 con 0x7f01440f6510 2023-11-17T05:37:45.177 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:45.171+0000 7f01437fe700 1 --2- 172.21.15.160:0/2553152218 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01440f6510 0x7f014406b9a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f012c013510 tx=0x7f012c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:37:45.177 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:45.171+0000 7f013bfff700 1 -- 172.21.15.160:0/2553152218 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f012c009320 con 0x7f01440f6510 2023-11-17T05:37:45.177 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:45.171+0000 7f014a0dc700 1 -- 172.21.15.160:0/2553152218 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0144069fc0 con 0x7f01440f6510 2023-11-17T05:37:45.177 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:45.171+0000 7f013bfff700 1 -- 172.21.15.160:0/2553152218 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f012c013f00 con 0x7f01440f6510 2023-11-17T05:37:45.177 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:45.171+0000 7f013bfff700 1 -- 172.21.15.160:0/2553152218 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f012c014d90 con 0x7f01440f6510 2023-11-17T05:37:45.178 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:45.171+0000 7f014a0dc700 1 -- 172.21.15.160:0/2553152218 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f014406a450 con 0x7f01440f6510 2023-11-17T05:37:45.178 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:45.172+0000 7f013bfff700 1 -- 172.21.15.160:0/2553152218 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f012c0095f0 con 0x7f01440f6510 2023-11-17T05:37:45.178 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:45.172+0000 7f014a0dc700 1 -- 172.21.15.160:0/2553152218 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0144059100 con 0x7f01440f6510 2023-11-17T05:37:45.178 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:45.172+0000 7f013bfff700 1 --2- 172.21.15.160:0/2553152218 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f013002ff60 0x7f0130032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:37:45.178 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:45.172+0000 7f013bfff700 1 -- 172.21.15.160:0/2553152218 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f012c040fc0 con 0x7f01440f6510 2023-11-17T05:37:45.178 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:45.172+0000 7f0142ffd700 1 --2- 172.21.15.160:0/2553152218 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f013002ff60 0x7f0130032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:37:45.178 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:45.173+0000 7f0142ffd700 1 --2- 172.21.15.160:0/2553152218 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f013002ff60 0x7f0130032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f01340096a0 tx=0x7f0134006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:37:45.178 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:45.177+0000 7f013bfff700 1 -- 172.21.15.160:0/2553152218 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f012c014420 con 0x7f01440f6510 2023-11-17T05:37:45.385 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:45.383+0000 7f014a0dc700 1 -- 172.21.15.160:0/2553152218 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0144045ad0 con 0x7f01440f6510 2023-11-17T05:37:45.385 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:45.384+0000 7f013bfff700 1 -- 172.21.15.160:0/2553152218 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f012c010230 con 0x7f01440f6510 2023-11-17T05:37:45.386 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:37:45.386 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:37:45.388 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:45.385+0000 7f014a0dc700 1 -- 172.21.15.160:0/2553152218 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f013002ff60 msgr2=0x7f0130032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:45.389 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:45.385+0000 7f014a0dc700 1 --2- 172.21.15.160:0/2553152218 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f013002ff60 0x7f0130032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f01340096a0 tx=0x7f0134006b40).stop 2023-11-17T05:37:45.389 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:45.385+0000 7f014a0dc700 1 -- 172.21.15.160:0/2553152218 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01440f6510 msgr2=0x7f014406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:45.389 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:45.385+0000 7f014a0dc700 1 --2- 172.21.15.160:0/2553152218 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01440f6510 0x7f014406b9a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f012c013510 tx=0x7f012c00b040).stop 2023-11-17T05:37:45.390 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:45.385+0000 7f014a0dc700 1 -- 172.21.15.160:0/2553152218 shutdown_connections 2023-11-17T05:37:45.390 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:45.385+0000 7f014a0dc700 1 --2- 172.21.15.160:0/2553152218 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f013002ff60 0x7f0130032420 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:45.390 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:45.385+0000 7f014a0dc700 1 --2- 172.21.15.160:0/2553152218 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f01440f6510 0x7f014406b9a0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:45.391 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:45.385+0000 7f014a0dc700 1 -- 172.21.15.160:0/2553152218 >> 172.21.15.160:0/2553152218 conn(0x7f01440f1c70 msgr2=0x7f01440f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:37:45.391 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:45.385+0000 7f014a0dc700 1 -- 172.21.15.160:0/2553152218 shutdown_connections 2023-11-17T05:37:45.391 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:45.385+0000 7f014a0dc700 1 -- 172.21.15.160:0/2553152218 wait complete. 2023-11-17T05:37:45.391 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:37:45.457 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:45 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:37:44.377815+0000 mgr.smithi003.eddcuc (mgr.14164) 379 : cluster [DBG] pgmap v363: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:37:46.522 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:46 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:37:45.384861+0000 mon.smithi003 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.160:0/2553152218' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:37:46.797 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:37:46.797 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:37:47.521 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:47 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:37:46.378189+0000 mgr.smithi003.eddcuc (mgr.14164) 380 : cluster [DBG] pgmap v364: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:37:47.951 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:47.948+0000 7fa094e4d700 1 -- 172.21.15.160:0/1067681393 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0900f5410 msgr2=0x7fa0900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:47.998 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:47.948+0000 7fa094e4d700 1 --2- 172.21.15.160:0/1067681393 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0900f5410 0x7fa0900f5830 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7fa078009fb0 tx=0x7fa078009b30).stop 2023-11-17T05:37:47.998 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:47.948+0000 7fa094e4d700 1 -- 172.21.15.160:0/1067681393 shutdown_connections 2023-11-17T05:37:47.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:47.948+0000 7fa094e4d700 1 --2- 172.21.15.160:0/1067681393 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0900f5410 0x7fa0900f5830 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:47.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:47.948+0000 7fa094e4d700 1 -- 172.21.15.160:0/1067681393 >> 172.21.15.160:0/1067681393 conn(0x7fa0900f0b70 msgr2=0x7fa0900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:37:47.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:47.948+0000 7fa094e4d700 1 -- 172.21.15.160:0/1067681393 shutdown_connections 2023-11-17T05:37:47.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:47.948+0000 7fa094e4d700 1 -- 172.21.15.160:0/1067681393 wait complete. 2023-11-17T05:37:47.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:47.949+0000 7fa094e4d700 1 Processor -- start 2023-11-17T05:37:47.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:47.949+0000 7fa094e4d700 1 -- start start 2023-11-17T05:37:47.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:47.949+0000 7fa094e4d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0900f5410 0x7fa090105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:37:47.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:47.949+0000 7fa094e4d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa090105b20 con 0x7fa0900f5410 2023-11-17T05:37:47.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:47.949+0000 7fa08e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0900f5410 0x7fa090105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:37:47.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:47.949+0000 7fa08e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0900f5410 0x7fa090105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:52570/0 (socket says 172.21.15.160:52570) 2023-11-17T05:37:47.999 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:47.949+0000 7fa08e59c700 1 -- 172.21.15.160:0/3061828419 learned_addr learned my addr 172.21.15.160:0/3061828419 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:37:48.000 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:47.950+0000 7fa08e59c700 1 -- 172.21.15.160:0/3061828419 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa078005040 con 0x7fa0900f5410 2023-11-17T05:37:48.000 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:47.950+0000 7fa08e59c700 1 --2- 172.21.15.160:0/3061828419 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0900f5410 0x7fa090105630 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7fa078013510 tx=0x7fa07800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:37:48.000 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:47.950+0000 7fa086ffd700 1 -- 172.21.15.160:0/3061828419 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa078009320 con 0x7fa0900f5410 2023-11-17T05:37:48.001 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:47.950+0000 7fa094e4d700 1 -- 172.21.15.160:0/3061828419 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa090105d20 con 0x7fa0900f5410 2023-11-17T05:37:48.001 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:47.950+0000 7fa086ffd700 1 -- 172.21.15.160:0/3061828419 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa078013f00 con 0x7fa0900f5410 2023-11-17T05:37:48.001 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:47.950+0000 7fa086ffd700 1 -- 172.21.15.160:0/3061828419 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa078014d90 con 0x7fa0900f5410 2023-11-17T05:37:48.001 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:47.950+0000 7fa094e4d700 1 -- 172.21.15.160:0/3061828419 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0901061b0 con 0x7fa0900f5410 2023-11-17T05:37:48.002 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:47.951+0000 7fa086ffd700 1 -- 172.21.15.160:0/3061828419 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa078014420 con 0x7fa0900f5410 2023-11-17T05:37:48.003 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:47.951+0000 7fa094e4d700 1 -- 172.21.15.160:0/3061828419 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa090045ad0 con 0x7fa0900f5410 2023-11-17T05:37:48.003 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:47.951+0000 7fa086ffd700 1 --2- 172.21.15.160:0/3061828419 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa07c02ffb0 0x7fa07c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:37:48.003 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:47.951+0000 7fa086ffd700 1 -- 172.21.15.160:0/3061828419 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa078040fd0 con 0x7fa0900f5410 2023-11-17T05:37:48.003 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:47.951+0000 7fa08dd9b700 1 --2- 172.21.15.160:0/3061828419 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa07c02ffb0 0x7fa07c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:37:48.003 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:47.951+0000 7fa08dd9b700 1 --2- 172.21.15.160:0/3061828419 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa07c02ffb0 0x7fa07c032470 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fa0800096a0 tx=0x7fa080006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:37:48.003 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:47.956+0000 7fa086ffd700 1 -- 172.21.15.160:0/3061828419 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa07806d2b0 con 0x7fa0900f5410 2023-11-17T05:37:48.165 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:48.164+0000 7fa094e4d700 1 -- 172.21.15.160:0/3061828419 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa0900fae60 con 0x7fa0900f5410 2023-11-17T05:37:48.165 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:48.165+0000 7fa086ffd700 1 -- 172.21.15.160:0/3061828419 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fa0780095f0 con 0x7fa0900f5410 2023-11-17T05:37:48.165 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:37:48.166 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:37:48.167 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:48.166+0000 7fa094e4d700 1 -- 172.21.15.160:0/3061828419 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa07c02ffb0 msgr2=0x7fa07c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:48.167 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:48.166+0000 7fa094e4d700 1 --2- 172.21.15.160:0/3061828419 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa07c02ffb0 0x7fa07c032470 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fa0800096a0 tx=0x7fa080006b40).stop 2023-11-17T05:37:48.167 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:48.166+0000 7fa094e4d700 1 -- 172.21.15.160:0/3061828419 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0900f5410 msgr2=0x7fa090105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:48.167 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:48.166+0000 7fa094e4d700 1 --2- 172.21.15.160:0/3061828419 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0900f5410 0x7fa090105630 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7fa078013510 tx=0x7fa07800b040).stop 2023-11-17T05:37:48.167 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:48.166+0000 7fa094e4d700 1 -- 172.21.15.160:0/3061828419 shutdown_connections 2023-11-17T05:37:48.167 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:48.166+0000 7fa094e4d700 1 --2- 172.21.15.160:0/3061828419 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa07c02ffb0 0x7fa07c032470 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:48.167 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:48.166+0000 7fa094e4d700 1 --2- 172.21.15.160:0/3061828419 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa0900f5410 0x7fa090105630 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:48.168 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:48.166+0000 7fa094e4d700 1 -- 172.21.15.160:0/3061828419 >> 172.21.15.160:0/3061828419 conn(0x7fa0900f0b70 msgr2=0x7fa0900f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:37:48.168 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:48.166+0000 7fa094e4d700 1 -- 172.21.15.160:0/3061828419 shutdown_connections 2023-11-17T05:37:48.168 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:48.166+0000 7fa094e4d700 1 -- 172.21.15.160:0/3061828419 wait complete. 2023-11-17T05:37:48.168 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:37:48.520 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:48 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:37:48.165777+0000 mon.smithi003 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.160:0/3061828419' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:37:49.520 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:49 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:37:48.378548+0000 mgr.smithi003.eddcuc (mgr.14164) 381 : cluster [DBG] pgmap v365: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:37:49.558 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:37:49.559 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:37:50.753 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:50.750+0000 7f15d1fa8700 1 -- 172.21.15.160:0/2065477663 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15cc0f6510 msgr2=0x7f15cc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:50.753 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:50.750+0000 7f15d1fa8700 1 --2- 172.21.15.160:0/2065477663 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15cc0f6510 0x7f15cc0f6930 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f15b4009fb0 tx=0x7f15b4009b30).stop 2023-11-17T05:37:50.753 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:50.750+0000 7f15d1fa8700 1 -- 172.21.15.160:0/2065477663 shutdown_connections 2023-11-17T05:37:50.753 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:50.750+0000 7f15d1fa8700 1 --2- 172.21.15.160:0/2065477663 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15cc0f6510 0x7f15cc0f6930 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:50.753 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:50.750+0000 7f15d1fa8700 1 -- 172.21.15.160:0/2065477663 >> 172.21.15.160:0/2065477663 conn(0x7f15cc0f1c70 msgr2=0x7f15cc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:37:50.753 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:50.750+0000 7f15d1fa8700 1 -- 172.21.15.160:0/2065477663 shutdown_connections 2023-11-17T05:37:50.753 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:50.750+0000 7f15d1fa8700 1 -- 172.21.15.160:0/2065477663 wait complete. 2023-11-17T05:37:50.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:50.751+0000 7f15d1fa8700 1 Processor -- start 2023-11-17T05:37:50.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:50.751+0000 7f15d1fa8700 1 -- start start 2023-11-17T05:37:50.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:50.751+0000 7f15d1fa8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15cc106840 0x7f15cc106c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:37:50.754 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:50.751+0000 7f15d1fa8700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15cc107150 con 0x7f15cc106840 2023-11-17T05:37:50.755 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:50.751+0000 7f15cb7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15cc106840 0x7f15cc106c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:37:50.755 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:50.751+0000 7f15cb7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15cc106840 0x7f15cc106c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:60570/0 (socket says 172.21.15.160:60570) 2023-11-17T05:37:50.755 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:50.751+0000 7f15cb7fe700 1 -- 172.21.15.160:0/3278238204 learned_addr learned my addr 172.21.15.160:0/3278238204 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:37:50.755 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:50.752+0000 7f15cb7fe700 1 -- 172.21.15.160:0/3278238204 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b4005040 con 0x7f15cc106840 2023-11-17T05:37:50.756 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:50.752+0000 7f15cb7fe700 1 --2- 172.21.15.160:0/3278238204 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15cc106840 0x7f15cc106c60 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f15b4009f80 tx=0x7f15b4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:37:50.756 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:50.752+0000 7f15c3fff700 1 -- 172.21.15.160:0/3278238204 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f15b40136c0 con 0x7f15cc106840 2023-11-17T05:37:50.756 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:50.752+0000 7f15c3fff700 1 -- 172.21.15.160:0/3278238204 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f15b40095f0 con 0x7f15cc106840 2023-11-17T05:37:50.756 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:50.752+0000 7f15d1fa8700 1 -- 172.21.15.160:0/3278238204 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15cc107350 con 0x7f15cc106840 2023-11-17T05:37:50.756 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:50.752+0000 7f15c3fff700 1 -- 172.21.15.160:0/3278238204 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f15b4014420 con 0x7f15cc106840 2023-11-17T05:37:50.756 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:50.752+0000 7f15d1fa8700 1 -- 172.21.15.160:0/3278238204 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15cc107ea0 con 0x7f15cc106840 2023-11-17T05:37:50.757 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:50.753+0000 7f15c3fff700 1 -- 172.21.15.160:0/3278238204 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f15b4025060 con 0x7f15cc106840 2023-11-17T05:37:50.757 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:50.753+0000 7f15d1fa8700 1 -- 172.21.15.160:0/3278238204 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15cc059100 con 0x7f15cc106840 2023-11-17T05:37:50.757 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:50.753+0000 7f15c3fff700 1 --2- 172.21.15.160:0/3278238204 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f15b802ff60 0x7f15b8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:37:50.757 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:50.753+0000 7f15c3fff700 1 -- 172.21.15.160:0/3278238204 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f15b40411c0 con 0x7f15cc106840 2023-11-17T05:37:50.757 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:50.753+0000 7f15caffd700 1 --2- 172.21.15.160:0/3278238204 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f15b802ff60 0x7f15b8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:37:50.757 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:50.754+0000 7f15caffd700 1 --2- 172.21.15.160:0/3278238204 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f15b802ff60 0x7f15b8032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f15bc0096a0 tx=0x7f15bc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:37:50.758 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:50.758+0000 7f15c3fff700 1 -- 172.21.15.160:0/3278238204 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f15b4026b30 con 0x7f15cc106840 2023-11-17T05:37:50.965 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:50.964+0000 7f15d1fa8700 1 -- 172.21.15.160:0/3278238204 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f15cc045ad0 con 0x7f15cc106840 2023-11-17T05:37:50.965 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:50.965+0000 7f15c3fff700 1 -- 172.21.15.160:0/3278238204 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f15b4026b30 con 0x7f15cc106840 2023-11-17T05:37:50.966 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:37:50.966 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:37:50.967 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:50.966+0000 7f15d1fa8700 1 -- 172.21.15.160:0/3278238204 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f15b802ff60 msgr2=0x7f15b8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:50.967 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:50.966+0000 7f15d1fa8700 1 --2- 172.21.15.160:0/3278238204 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f15b802ff60 0x7f15b8032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f15bc0096a0 tx=0x7f15bc006b40).stop 2023-11-17T05:37:50.967 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:50.966+0000 7f15d1fa8700 1 -- 172.21.15.160:0/3278238204 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15cc106840 msgr2=0x7f15cc106c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:50.967 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:50.966+0000 7f15d1fa8700 1 --2- 172.21.15.160:0/3278238204 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15cc106840 0x7f15cc106c60 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f15b4009f80 tx=0x7f15b4009b30).stop 2023-11-17T05:37:50.967 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:50.966+0000 7f15d1fa8700 1 -- 172.21.15.160:0/3278238204 shutdown_connections 2023-11-17T05:37:50.967 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:50.966+0000 7f15d1fa8700 1 --2- 172.21.15.160:0/3278238204 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f15b802ff60 0x7f15b8032420 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:50.967 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:50.966+0000 7f15d1fa8700 1 --2- 172.21.15.160:0/3278238204 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15cc106840 0x7f15cc106c60 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:50.968 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:50.966+0000 7f15d1fa8700 1 -- 172.21.15.160:0/3278238204 >> 172.21.15.160:0/3278238204 conn(0x7f15cc0f1c70 msgr2=0x7f15cc0f27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:37:50.968 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:50.966+0000 7f15d1fa8700 1 -- 172.21.15.160:0/3278238204 shutdown_connections 2023-11-17T05:37:50.968 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:50.966+0000 7f15d1fa8700 1 -- 172.21.15.160:0/3278238204 wait complete. 2023-11-17T05:37:50.968 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:37:51.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:51 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:37:50.378888+0000 mgr.smithi003.eddcuc (mgr.14164) 382 : cluster [DBG] pgmap v366: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:37:51.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:51 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:37:50.965701+0000 mon.smithi003 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.160:0/3278238204' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:37:52.397 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:37:52.397 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:37:53.588 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:53.581+0000 7fa8a604b700 1 -- 172.21.15.160:0/948756282 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8a00f5410 msgr2=0x7fa8a00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:53.588 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:53.581+0000 7fa8a604b700 1 --2- 172.21.15.160:0/948756282 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8a00f5410 0x7fa8a00f5830 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7fa888009fb0 tx=0x7fa888009b30).stop 2023-11-17T05:37:53.588 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:53.581+0000 7fa8a604b700 1 -- 172.21.15.160:0/948756282 shutdown_connections 2023-11-17T05:37:53.588 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:53.581+0000 7fa8a604b700 1 --2- 172.21.15.160:0/948756282 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8a00f5410 0x7fa8a00f5830 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:53.588 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:53.581+0000 7fa8a604b700 1 -- 172.21.15.160:0/948756282 >> 172.21.15.160:0/948756282 conn(0x7fa8a00f0b70 msgr2=0x7fa8a00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:37:53.588 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:53.581+0000 7fa8a604b700 1 -- 172.21.15.160:0/948756282 shutdown_connections 2023-11-17T05:37:53.588 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:53.581+0000 7fa8a604b700 1 -- 172.21.15.160:0/948756282 wait complete. 2023-11-17T05:37:53.588 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:53.582+0000 7fa8a604b700 1 Processor -- start 2023-11-17T05:37:53.589 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:53.582+0000 7fa8a604b700 1 -- start start 2023-11-17T05:37:53.589 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:53.582+0000 7fa8a604b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8a00f5410 0x7fa8a010cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:37:53.589 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:53.582+0000 7fa8a604b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a010d270 con 0x7fa8a00f5410 2023-11-17T05:37:53.589 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:53.582+0000 7fa89f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8a00f5410 0x7fa8a010cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:37:53.589 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:53.582+0000 7fa89f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8a00f5410 0x7fa8a010cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:60588/0 (socket says 172.21.15.160:60588) 2023-11-17T05:37:53.589 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:53.582+0000 7fa89f7fe700 1 -- 172.21.15.160:0/3508904873 learned_addr learned my addr 172.21.15.160:0/3508904873 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:37:53.589 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:53.583+0000 7fa89f7fe700 1 -- 172.21.15.160:0/3508904873 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa888005040 con 0x7fa8a00f5410 2023-11-17T05:37:53.589 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:53.583+0000 7fa89f7fe700 1 --2- 172.21.15.160:0/3508904873 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8a00f5410 0x7fa8a010cd80 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7fa888013510 tx=0x7fa88800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:37:53.589 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:53.583+0000 7fa897fff700 1 -- 172.21.15.160:0/3508904873 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa888009320 con 0x7fa8a00f5410 2023-11-17T05:37:53.589 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:53.583+0000 7fa8a604b700 1 -- 172.21.15.160:0/3508904873 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8a010d470 con 0x7fa8a00f5410 2023-11-17T05:37:53.589 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:53.583+0000 7fa897fff700 1 -- 172.21.15.160:0/3508904873 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa888013f00 con 0x7fa8a00f5410 2023-11-17T05:37:53.589 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:53.583+0000 7fa897fff700 1 -- 172.21.15.160:0/3508904873 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa888014d90 con 0x7fa8a00f5410 2023-11-17T05:37:53.590 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:53.583+0000 7fa8a604b700 1 -- 172.21.15.160:0/3508904873 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8a010e0b0 con 0x7fa8a00f5410 2023-11-17T05:37:53.590 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:53.584+0000 7fa897fff700 1 -- 172.21.15.160:0/3508904873 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa8880263a0 con 0x7fa8a00f5410 2023-11-17T05:37:53.590 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:53.584+0000 7fa897fff700 1 --2- 172.21.15.160:0/3508904873 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa88c02ff60 0x7fa88c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:37:53.590 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:53.584+0000 7fa897fff700 1 -- 172.21.15.160:0/3508904873 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa8880410a0 con 0x7fa8a00f5410 2023-11-17T05:37:53.590 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:53.584+0000 7fa8a604b700 1 -- 172.21.15.160:0/3508904873 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8a010d900 con 0x7fa8a00f5410 2023-11-17T05:37:53.590 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:53.584+0000 7fa89effd700 1 --2- 172.21.15.160:0/3508904873 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa88c02ff60 0x7fa88c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:37:53.590 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:53.585+0000 7fa89effd700 1 --2- 172.21.15.160:0/3508904873 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa88c02ff60 0x7fa88c032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fa8900096a0 tx=0x7fa890006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:37:53.591 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:53.589+0000 7fa897fff700 1 -- 172.21.15.160:0/3508904873 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa888012920 con 0x7fa8a00f5410 2023-11-17T05:37:53.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:53 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:37:52.379256+0000 mgr.smithi003.eddcuc (mgr.14164) 383 : cluster [DBG] pgmap v367: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:37:53.798 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:53.796+0000 7fa8a604b700 1 -- 172.21.15.160:0/3508904873 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa8a0102710 con 0x7fa8a00f5410 2023-11-17T05:37:53.798 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:53.797+0000 7fa897fff700 1 -- 172.21.15.160:0/3508904873 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fa888012330 con 0x7fa8a00f5410 2023-11-17T05:37:53.798 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:37:53.798 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:37:53.799 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:53.798+0000 7fa8a604b700 1 -- 172.21.15.160:0/3508904873 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa88c02ff60 msgr2=0x7fa88c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:53.799 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:53.798+0000 7fa8a604b700 1 --2- 172.21.15.160:0/3508904873 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa88c02ff60 0x7fa88c032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fa8900096a0 tx=0x7fa890006b40).stop 2023-11-17T05:37:53.800 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:53.798+0000 7fa8a604b700 1 -- 172.21.15.160:0/3508904873 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8a00f5410 msgr2=0x7fa8a010cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:53.800 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:53.798+0000 7fa8a604b700 1 --2- 172.21.15.160:0/3508904873 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8a00f5410 0x7fa8a010cd80 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7fa888013510 tx=0x7fa88800b040).stop 2023-11-17T05:37:53.800 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:53.798+0000 7fa8a604b700 1 -- 172.21.15.160:0/3508904873 shutdown_connections 2023-11-17T05:37:53.800 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:53.798+0000 7fa8a604b700 1 --2- 172.21.15.160:0/3508904873 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa88c02ff60 0x7fa88c032420 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:53.800 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:53.798+0000 7fa8a604b700 1 --2- 172.21.15.160:0/3508904873 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8a00f5410 0x7fa8a010cd80 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:53.800 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:53.798+0000 7fa8a604b700 1 -- 172.21.15.160:0/3508904873 >> 172.21.15.160:0/3508904873 conn(0x7fa8a00f0b70 msgr2=0x7fa8a00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:37:53.800 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:53.799+0000 7fa8a604b700 1 -- 172.21.15.160:0/3508904873 shutdown_connections 2023-11-17T05:37:53.800 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:53.799+0000 7fa8a604b700 1 -- 172.21.15.160:0/3508904873 wait complete. 2023-11-17T05:37:53.801 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:37:54.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:54 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:37:53.798120+0000 mon.smithi003 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.160:0/3508904873' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:37:55.191 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:37:55.191 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:37:55.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:55 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:37:54.379585+0000 mgr.smithi003.eddcuc (mgr.14164) 384 : cluster [DBG] pgmap v368: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:37:56.423 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:56.421+0000 7ff313f49700 1 -- 172.21.15.160:0/3542331917 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff30c0f6510 msgr2=0x7ff30c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:56.423 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:56.421+0000 7ff313f49700 1 --2- 172.21.15.160:0/3542331917 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff30c0f6510 0x7ff30c0f6930 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7ff2fc009fb0 tx=0x7ff2fc009b30).stop 2023-11-17T05:37:56.423 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:56.421+0000 7ff313f49700 1 -- 172.21.15.160:0/3542331917 shutdown_connections 2023-11-17T05:37:56.423 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:56.421+0000 7ff313f49700 1 --2- 172.21.15.160:0/3542331917 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff30c0f6510 0x7ff30c0f6930 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:56.423 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:56.421+0000 7ff313f49700 1 -- 172.21.15.160:0/3542331917 >> 172.21.15.160:0/3542331917 conn(0x7ff30c0f1c70 msgr2=0x7ff30c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:37:56.423 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:56.421+0000 7ff313f49700 1 -- 172.21.15.160:0/3542331917 shutdown_connections 2023-11-17T05:37:56.423 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:56.421+0000 7ff313f49700 1 -- 172.21.15.160:0/3542331917 wait complete. 2023-11-17T05:37:56.423 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:56.422+0000 7ff313f49700 1 Processor -- start 2023-11-17T05:37:56.423 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:56.422+0000 7ff313f49700 1 -- start start 2023-11-17T05:37:56.424 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:56.422+0000 7ff313f49700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff30c0f6510 0x7ff30c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:37:56.424 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:56.422+0000 7ff313f49700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff30c069e20 con 0x7ff30c0f6510 2023-11-17T05:37:56.424 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:56.422+0000 7ff311ce5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff30c0f6510 0x7ff30c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:37:56.424 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:56.422+0000 7ff311ce5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff30c0f6510 0x7ff30c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:60604/0 (socket says 172.21.15.160:60604) 2023-11-17T05:37:56.424 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:56.422+0000 7ff311ce5700 1 -- 172.21.15.160:0/4199026445 learned_addr learned my addr 172.21.15.160:0/4199026445 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:37:56.425 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:56.422+0000 7ff311ce5700 1 -- 172.21.15.160:0/4199026445 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2fc005040 con 0x7ff30c0f6510 2023-11-17T05:37:56.425 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:56.423+0000 7ff311ce5700 1 --2- 172.21.15.160:0/4199026445 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff30c0f6510 0x7ff30c06b9a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7ff2fc013510 tx=0x7ff2fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:37:56.425 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:56.423+0000 7ff3027fc700 1 -- 172.21.15.160:0/4199026445 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2fc009320 con 0x7ff30c0f6510 2023-11-17T05:37:56.425 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:56.423+0000 7ff313f49700 1 -- 172.21.15.160:0/4199026445 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff30c069fc0 con 0x7ff30c0f6510 2023-11-17T05:37:56.426 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:56.423+0000 7ff3027fc700 1 -- 172.21.15.160:0/4199026445 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff2fc013f00 con 0x7ff30c0f6510 2023-11-17T05:37:56.426 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:56.423+0000 7ff3027fc700 1 -- 172.21.15.160:0/4199026445 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2fc014d90 con 0x7ff30c0f6510 2023-11-17T05:37:56.426 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:56.423+0000 7ff313f49700 1 -- 172.21.15.160:0/4199026445 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff30c06a450 con 0x7ff30c0f6510 2023-11-17T05:37:56.426 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:56.424+0000 7ff3027fc700 1 -- 172.21.15.160:0/4199026445 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff2fc0095f0 con 0x7ff30c0f6510 2023-11-17T05:37:56.426 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:56.424+0000 7ff313f49700 1 -- 172.21.15.160:0/4199026445 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff30c059100 con 0x7ff30c0f6510 2023-11-17T05:37:56.426 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:56.424+0000 7ff3027fc700 1 --2- 172.21.15.160:0/4199026445 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7ff2f802ff60 0x7ff2f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:37:56.426 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:56.424+0000 7ff3027fc700 1 -- 172.21.15.160:0/4199026445 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff2fc0410a0 con 0x7ff30c0f6510 2023-11-17T05:37:56.426 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:56.424+0000 7ff3114e4700 1 --2- 172.21.15.160:0/4199026445 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7ff2f802ff60 0x7ff2f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:37:56.426 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:56.424+0000 7ff3114e4700 1 --2- 172.21.15.160:0/4199026445 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7ff2f802ff60 0x7ff2f8032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7ff3080096a0 tx=0x7ff308006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:37:56.429 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:56.429+0000 7ff3027fc700 1 -- 172.21.15.160:0/4199026445 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff2fc01d7f0 con 0x7ff30c0f6510 2023-11-17T05:37:56.631 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:56.631+0000 7ff313f49700 1 -- 172.21.15.160:0/4199026445 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff30c0fbf60 con 0x7ff30c0f6510 2023-11-17T05:37:56.632 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:56.631+0000 7ff3027fc700 1 -- 172.21.15.160:0/4199026445 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7ff2fc010230 con 0x7ff30c0f6510 2023-11-17T05:37:56.632 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:37:56.632 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:37:56.634 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:56.633+0000 7ff313f49700 1 -- 172.21.15.160:0/4199026445 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7ff2f802ff60 msgr2=0x7ff2f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:56.634 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:56.633+0000 7ff313f49700 1 --2- 172.21.15.160:0/4199026445 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7ff2f802ff60 0x7ff2f8032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7ff3080096a0 tx=0x7ff308006b40).stop 2023-11-17T05:37:56.634 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:56.633+0000 7ff313f49700 1 -- 172.21.15.160:0/4199026445 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff30c0f6510 msgr2=0x7ff30c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:37:56.634 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:56.633+0000 7ff313f49700 1 --2- 172.21.15.160:0/4199026445 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff30c0f6510 0x7ff30c06b9a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7ff2fc013510 tx=0x7ff2fc00b040).stop 2023-11-17T05:37:58.190 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:56.633+0000 7ff313f49700 1 -- 172.21.15.160:0/4199026445 shutdown_connections 2023-11-17T05:37:58.229 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:56.633+0000 7ff313f49700 1 --2- 172.21.15.160:0/4199026445 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7ff2f802ff60 0x7ff2f8032420 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:58.275 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:56.633+0000 7ff313f49700 1 --2- 172.21.15.160:0/4199026445 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff30c0f6510 0x7ff30c06b9a0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:37:58.275 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:56.633+0000 7ff313f49700 1 -- 172.21.15.160:0/4199026445 >> 172.21.15.160:0/4199026445 conn(0x7ff30c0f1c70 msgr2=0x7ff30c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:37:58.275 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:56.633+0000 7ff313f49700 1 -- 172.21.15.160:0/4199026445 shutdown_connections 2023-11-17T05:37:58.275 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:37:56.633+0000 7ff313f49700 1 -- 172.21.15.160:0/4199026445 wait complete. 2023-11-17T05:37:58.276 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:37:56.379923+0000 mgr.smithi003.eddcuc (mgr.14164) 385 : cluster [DBG] pgmap v369: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:37:58.276 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:37:56.632574+0000 mon.smithi003 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.160:0/4199026445' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:37:58.277 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:37:59.278 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:37:59.278 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:37:59.771 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:37:59 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:37:58.380235+0000 mgr.smithi003.eddcuc (mgr.14164) 386 : cluster [DBG] pgmap v370: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:38:00.457 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:00.455+0000 7f20adc66700 1 -- 172.21.15.160:0/208545201 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f20a80f47c0 msgr2=0x7f20a80f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:38:00.457 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:00.455+0000 7f20adc66700 1 --2- 172.21.15.160:0/208545201 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f20a80f47c0 0x7f20a80f4be0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f2098009fb0 tx=0x7f2098009b30).stop 2023-11-17T05:38:00.458 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:00.455+0000 7f20adc66700 1 -- 172.21.15.160:0/208545201 shutdown_connections 2023-11-17T05:38:00.458 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:00.455+0000 7f20adc66700 1 --2- 172.21.15.160:0/208545201 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f20a80f47c0 0x7f20a80f4be0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:38:00.458 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:00.455+0000 7f20adc66700 1 -- 172.21.15.160:0/208545201 >> 172.21.15.160:0/208545201 conn(0x7f20a80f02e0 msgr2=0x7f20a80f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:38:00.458 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:00.455+0000 7f20adc66700 1 -- 172.21.15.160:0/208545201 shutdown_connections 2023-11-17T05:38:00.458 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:00.455+0000 7f20adc66700 1 -- 172.21.15.160:0/208545201 wait complete. 2023-11-17T05:38:00.458 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:00.456+0000 7f20adc66700 1 Processor -- start 2023-11-17T05:38:00.458 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:00.456+0000 7f20adc66700 1 -- start start 2023-11-17T05:38:00.458 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:00.456+0000 7f20adc66700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f20a80f47c0 0x7f20a8106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:38:00.458 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:00.456+0000 7f20adc66700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a8106bf0 con 0x7f20a80f47c0 2023-11-17T05:38:00.458 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:00.457+0000 7f20a77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f20a80f47c0 0x7f20a8106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:38:00.459 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:00.457+0000 7f20a77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f20a80f47c0 0x7f20a8106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:55412/0 (socket says 172.21.15.160:55412) 2023-11-17T05:38:00.459 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:00.457+0000 7f20a77fe700 1 -- 172.21.15.160:0/2754410470 learned_addr learned my addr 172.21.15.160:0/2754410470 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:38:00.460 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:00.457+0000 7f20a77fe700 1 -- 172.21.15.160:0/2754410470 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2098005040 con 0x7f20a80f47c0 2023-11-17T05:38:00.460 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:00.457+0000 7f20a77fe700 1 --2- 172.21.15.160:0/2754410470 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f20a80f47c0 0x7f20a8106700 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f2098014000 tx=0x7f209800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:38:00.460 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:00.457+0000 7f208ffff700 1 -- 172.21.15.160:0/2754410470 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20980095f0 con 0x7f20a80f47c0 2023-11-17T05:38:00.460 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:00.457+0000 7f20adc66700 1 -- 172.21.15.160:0/2754410470 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20a8106df0 con 0x7f20a80f47c0 2023-11-17T05:38:00.460 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:00.457+0000 7f208ffff700 1 -- 172.21.15.160:0/2754410470 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2098007e70 con 0x7f20a80f47c0 2023-11-17T05:38:00.461 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:00.457+0000 7f208ffff700 1 -- 172.21.15.160:0/2754410470 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2098014ea0 con 0x7f20a80f47c0 2023-11-17T05:38:00.461 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:00.457+0000 7f20adc66700 1 -- 172.21.15.160:0/2754410470 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20a8107280 con 0x7f20a80f47c0 2023-11-17T05:38:00.461 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:00.458+0000 7f208ffff700 1 -- 172.21.15.160:0/2754410470 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2098025070 con 0x7f20a80f47c0 2023-11-17T05:38:00.461 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:00.458+0000 7f20adc66700 1 -- 172.21.15.160:0/2754410470 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20a8045ad0 con 0x7f20a80f47c0 2023-11-17T05:38:00.461 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:00.458+0000 7f208ffff700 1 --2- 172.21.15.160:0/2754410470 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f209002ff60 0x7f2090032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:38:00.461 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:00.458+0000 7f208ffff700 1 -- 172.21.15.160:0/2754410470 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2098040970 con 0x7f20a80f47c0 2023-11-17T05:38:00.461 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:00.459+0000 7f20a6ffd700 1 --2- 172.21.15.160:0/2754410470 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f209002ff60 0x7f2090032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:38:00.461 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:00.459+0000 7f20a6ffd700 1 --2- 172.21.15.160:0/2754410470 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f209002ff60 0x7f2090032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f209c0096a0 tx=0x7f209c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:38:00.465 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:00.463+0000 7f208ffff700 1 -- 172.21.15.160:0/2754410470 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f20980079c0 con 0x7f20a80f47c0 2023-11-17T05:38:00.673 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:00.672+0000 7f20adc66700 1 -- 172.21.15.160:0/2754410470 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f20a8108130 con 0x7f20a80f47c0 2023-11-17T05:38:00.673 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:00.673+0000 7f208ffff700 1 -- 172.21.15.160:0/2754410470 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f209800f300 con 0x7f20a80f47c0 2023-11-17T05:38:00.674 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:38:00.674 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:38:00.675 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:00.674+0000 7f20adc66700 1 -- 172.21.15.160:0/2754410470 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f209002ff60 msgr2=0x7f2090032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:38:00.675 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:00.674+0000 7f20adc66700 1 --2- 172.21.15.160:0/2754410470 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f209002ff60 0x7f2090032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f209c0096a0 tx=0x7f209c006b40).stop 2023-11-17T05:38:00.675 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:00.674+0000 7f20adc66700 1 -- 172.21.15.160:0/2754410470 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f20a80f47c0 msgr2=0x7f20a8106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:38:00.675 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:00.674+0000 7f20adc66700 1 --2- 172.21.15.160:0/2754410470 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f20a80f47c0 0x7f20a8106700 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f2098014000 tx=0x7f209800b040).stop 2023-11-17T05:38:00.676 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:00.674+0000 7f20adc66700 1 -- 172.21.15.160:0/2754410470 shutdown_connections 2023-11-17T05:38:00.676 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:00.674+0000 7f20adc66700 1 --2- 172.21.15.160:0/2754410470 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f209002ff60 0x7f2090032420 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:38:00.676 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:00.674+0000 7f20adc66700 1 --2- 172.21.15.160:0/2754410470 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f20a80f47c0 0x7f20a8106700 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:38:00.676 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:00.674+0000 7f20adc66700 1 -- 172.21.15.160:0/2754410470 >> 172.21.15.160:0/2754410470 conn(0x7f20a80f02e0 msgr2=0x7f20a80f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:38:00.676 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:00.674+0000 7f20adc66700 1 -- 172.21.15.160:0/2754410470 shutdown_connections 2023-11-17T05:38:00.676 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:00.675+0000 7f20adc66700 1 -- 172.21.15.160:0/2754410470 wait complete. 2023-11-17T05:38:00.676 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:38:01.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:38:01 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:38:00.380575+0000 mgr.smithi003.eddcuc (mgr.14164) 387 : cluster [DBG] pgmap v371: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:38:01.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:38:01 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:38:00.673813+0000 mon.smithi003 (mon.0) 315 : audit [DBG] from='client.? 172.21.15.160:0/2754410470' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:38:02.066 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:38:02.066 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:38:03.250 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:03.249+0000 7f6deb005700 1 -- 172.21.15.160:0/1915986659 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6de4060280 msgr2=0x7f6de40f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:38:03.251 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:03.249+0000 7f6deb005700 1 --2- 172.21.15.160:0/1915986659 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6de4060280 0x7f6de40f90f0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f6dd8009fb0 tx=0x7f6dd8009b30).stop 2023-11-17T05:38:03.251 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:03.249+0000 7f6deb005700 1 -- 172.21.15.160:0/1915986659 shutdown_connections 2023-11-17T05:38:03.251 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:03.249+0000 7f6deb005700 1 --2- 172.21.15.160:0/1915986659 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6de4060280 0x7f6de40f90f0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:38:03.251 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:03.249+0000 7f6deb005700 1 -- 172.21.15.160:0/1915986659 >> 172.21.15.160:0/1915986659 conn(0x7f6de40f0b70 msgr2=0x7f6de40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:38:03.251 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:03.249+0000 7f6deb005700 1 -- 172.21.15.160:0/1915986659 shutdown_connections 2023-11-17T05:38:03.251 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:03.249+0000 7f6deb005700 1 -- 172.21.15.160:0/1915986659 wait complete. 2023-11-17T05:38:03.251 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:03.250+0000 7f6deb005700 1 Processor -- start 2023-11-17T05:38:03.251 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:03.250+0000 7f6deb005700 1 -- start start 2023-11-17T05:38:03.253 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:03.250+0000 7f6deb005700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6de4060280 0x7f6de41066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:38:03.253 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:03.250+0000 7f6deb005700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de4106bd0 con 0x7f6de4060280 2023-11-17T05:38:03.253 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:03.250+0000 7f6de8da1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6de4060280 0x7f6de41066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:38:03.253 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:03.250+0000 7f6de8da1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6de4060280 0x7f6de41066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:55416/0 (socket says 172.21.15.160:55416) 2023-11-17T05:38:03.254 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:03.250+0000 7f6de8da1700 1 -- 172.21.15.160:0/3231171016 learned_addr learned my addr 172.21.15.160:0/3231171016 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:38:03.254 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:03.251+0000 7f6de8da1700 1 -- 172.21.15.160:0/3231171016 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dd8005040 con 0x7f6de4060280 2023-11-17T05:38:03.254 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:03.251+0000 7f6de8da1700 1 --2- 172.21.15.160:0/3231171016 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6de4060280 0x7f6de41066e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f6dd8013510 tx=0x7f6dd800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:38:03.254 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:03.251+0000 7f6de17fa700 1 -- 172.21.15.160:0/3231171016 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6dd8009320 con 0x7f6de4060280 2023-11-17T05:38:03.254 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:03.251+0000 7f6deb005700 1 -- 172.21.15.160:0/3231171016 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6de4106dd0 con 0x7f6de4060280 2023-11-17T05:38:03.254 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:03.251+0000 7f6de17fa700 1 -- 172.21.15.160:0/3231171016 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6dd8013f00 con 0x7f6de4060280 2023-11-17T05:38:03.254 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:03.251+0000 7f6de17fa700 1 -- 172.21.15.160:0/3231171016 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6dd8014dc0 con 0x7f6de4060280 2023-11-17T05:38:03.254 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:03.251+0000 7f6deb005700 1 -- 172.21.15.160:0/3231171016 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6de4107260 con 0x7f6de4060280 2023-11-17T05:38:03.254 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:03.252+0000 7f6de17fa700 1 -- 172.21.15.160:0/3231171016 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6dd80095f0 con 0x7f6de4060280 2023-11-17T05:38:03.254 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:03.252+0000 7f6deb005700 1 -- 172.21.15.160:0/3231171016 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6de4045ad0 con 0x7f6de4060280 2023-11-17T05:38:03.254 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:03.252+0000 7f6de17fa700 1 --2- 172.21.15.160:0/3231171016 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f6dd402ff60 0x7f6dd4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:38:03.331 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:03.252+0000 7f6de17fa700 1 -- 172.21.15.160:0/3231171016 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6dd8041080 con 0x7f6de4060280 2023-11-17T05:38:03.331 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:03.252+0000 7f6de3fff700 1 --2- 172.21.15.160:0/3231171016 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f6dd402ff60 0x7f6dd4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:38:03.331 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:03.253+0000 7f6de3fff700 1 --2- 172.21.15.160:0/3231171016 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f6dd402ff60 0x7f6dd4032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f6dd00096a0 tx=0x7f6dd0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:38:03.331 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:03.257+0000 7f6de17fa700 1 -- 172.21.15.160:0/3231171016 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6dd80268c0 con 0x7f6de4060280 2023-11-17T05:38:03.463 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:03.463+0000 7f6deb005700 1 -- 172.21.15.160:0/3231171016 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6de4108110 con 0x7f6de4060280 2023-11-17T05:38:03.464 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:03.463+0000 7f6de17fa700 1 -- 172.21.15.160:0/3231171016 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f6dd8010230 con 0x7f6de4060280 2023-11-17T05:38:03.464 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:38:03.464 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:38:03.466 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:03.465+0000 7f6deb005700 1 -- 172.21.15.160:0/3231171016 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f6dd402ff60 msgr2=0x7f6dd4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:38:03.466 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:03.465+0000 7f6deb005700 1 --2- 172.21.15.160:0/3231171016 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f6dd402ff60 0x7f6dd4032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f6dd00096a0 tx=0x7f6dd0006b40).stop 2023-11-17T05:38:03.466 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:03.465+0000 7f6deb005700 1 -- 172.21.15.160:0/3231171016 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6de4060280 msgr2=0x7f6de41066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:38:03.466 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:03.465+0000 7f6deb005700 1 --2- 172.21.15.160:0/3231171016 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6de4060280 0x7f6de41066e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f6dd8013510 tx=0x7f6dd800b040).stop 2023-11-17T05:38:03.466 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:03.465+0000 7f6deb005700 1 -- 172.21.15.160:0/3231171016 shutdown_connections 2023-11-17T05:38:03.466 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:03.465+0000 7f6deb005700 1 --2- 172.21.15.160:0/3231171016 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f6dd402ff60 0x7f6dd4032420 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:38:03.466 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:03.465+0000 7f6deb005700 1 --2- 172.21.15.160:0/3231171016 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6de4060280 0x7f6de41066e0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:38:03.466 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:03.465+0000 7f6deb005700 1 -- 172.21.15.160:0/3231171016 >> 172.21.15.160:0/3231171016 conn(0x7f6de40f0b70 msgr2=0x7f6de40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:38:03.466 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:03.465+0000 7f6deb005700 1 -- 172.21.15.160:0/3231171016 shutdown_connections 2023-11-17T05:38:03.467 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:03.465+0000 7f6deb005700 1 -- 172.21.15.160:0/3231171016 wait complete. 2023-11-17T05:38:03.467 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:38:03.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:38:03 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:38:02.380903+0000 mgr.smithi003.eddcuc (mgr.14164) 388 : cluster [DBG] pgmap v372: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:38:04.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:38:04 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:38:03.464609+0000 mon.smithi003 (mon.0) 316 : audit [DBG] from='client.? 172.21.15.160:0/3231171016' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:38:04.841 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:38:04.841 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:38:05.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:38:05 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:38:04.381194+0000 mgr.smithi003.eddcuc (mgr.14164) 389 : cluster [DBG] pgmap v373: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:38:06.057 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:06.056+0000 7f03869d5700 1 -- 172.21.15.160:0/3256025961 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03800f51a0 msgr2=0x7f03800f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:38:06.057 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:06.056+0000 7f03869d5700 1 --2- 172.21.15.160:0/3256025961 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03800f51a0 0x7f03800f55c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f037c009fb0 tx=0x7f037c009b30).stop 2023-11-17T05:38:06.057 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:06.056+0000 7f03869d5700 1 -- 172.21.15.160:0/3256025961 shutdown_connections 2023-11-17T05:38:06.057 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:06.056+0000 7f03869d5700 1 --2- 172.21.15.160:0/3256025961 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03800f51a0 0x7f03800f55c0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:38:06.057 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:06.056+0000 7f03869d5700 1 -- 172.21.15.160:0/3256025961 >> 172.21.15.160:0/3256025961 conn(0x7f03800f09a0 msgr2=0x7f03800f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:38:06.057 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:06.056+0000 7f03869d5700 1 -- 172.21.15.160:0/3256025961 shutdown_connections 2023-11-17T05:38:06.057 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:06.056+0000 7f03869d5700 1 -- 172.21.15.160:0/3256025961 wait complete. 2023-11-17T05:38:06.059 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:06.057+0000 7f03869d5700 1 Processor -- start 2023-11-17T05:38:06.059 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:06.057+0000 7f03869d5700 1 -- start start 2023-11-17T05:38:06.059 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:06.057+0000 7f03869d5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03800f51a0 0x7f03801091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:38:06.059 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:06.057+0000 7f03869d5700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0380060240 con 0x7f03800f51a0 2023-11-17T05:38:06.059 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:06.057+0000 7f03859d3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03800f51a0 0x7f03801091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:38:06.059 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:06.057+0000 7f03859d3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03800f51a0 0x7f03801091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:55422/0 (socket says 172.21.15.160:55422) 2023-11-17T05:38:06.059 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:06.057+0000 7f03859d3700 1 -- 172.21.15.160:0/2774452486 learned_addr learned my addr 172.21.15.160:0/2774452486 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:38:06.059 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:06.057+0000 7f03859d3700 1 -- 172.21.15.160:0/2774452486 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f037c005040 con 0x7f03800f51a0 2023-11-17T05:38:06.060 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:06.057+0000 7f03859d3700 1 --2- 172.21.15.160:0/2774452486 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03800f51a0 0x7f03801091b0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f037c013510 tx=0x7f037c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:38:06.060 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:06.058+0000 7f03767fc700 1 -- 172.21.15.160:0/2774452486 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f037c009320 con 0x7f03800f51a0 2023-11-17T05:38:06.060 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:06.058+0000 7f03869d5700 1 -- 172.21.15.160:0/2774452486 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0380060440 con 0x7f03800f51a0 2023-11-17T05:38:06.060 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:06.058+0000 7f03767fc700 1 -- 172.21.15.160:0/2774452486 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f037c013f00 con 0x7f03800f51a0 2023-11-17T05:38:06.060 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:06.058+0000 7f03767fc700 1 -- 172.21.15.160:0/2774452486 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f037c014dc0 con 0x7f03800f51a0 2023-11-17T05:38:06.060 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:06.058+0000 7f03869d5700 1 -- 172.21.15.160:0/2774452486 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0380109860 con 0x7f03800f51a0 2023-11-17T05:38:06.061 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:06.059+0000 7f03767fc700 1 -- 172.21.15.160:0/2774452486 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f037c0095f0 con 0x7f03800f51a0 2023-11-17T05:38:06.061 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:06.059+0000 7f03869d5700 1 -- 172.21.15.160:0/2774452486 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0380045ad0 con 0x7f03800f51a0 2023-11-17T05:38:06.061 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:06.059+0000 7f03767fc700 1 --2- 172.21.15.160:0/2774452486 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f036c02ff60 0x7f036c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:38:06.061 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:06.059+0000 7f03767fc700 1 -- 172.21.15.160:0/2774452486 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f037c041000 con 0x7f03800f51a0 2023-11-17T05:38:06.061 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:06.059+0000 7f03851d2700 1 --2- 172.21.15.160:0/2774452486 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f036c02ff60 0x7f036c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:38:06.062 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:06.059+0000 7f03851d2700 1 --2- 172.21.15.160:0/2774452486 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f036c02ff60 0x7f036c032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f03700096a0 tx=0x7f0370006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:38:06.065 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:06.064+0000 7f03767fc700 1 -- 172.21.15.160:0/2774452486 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f037c01c790 con 0x7f03800f51a0 2023-11-17T05:38:06.268 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:06.267+0000 7f03869d5700 1 -- 172.21.15.160:0/2774452486 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f038010a080 con 0x7f03800f51a0 2023-11-17T05:38:06.268 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:06.267+0000 7f03767fc700 1 -- 172.21.15.160:0/2774452486 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f037c010230 con 0x7f03800f51a0 2023-11-17T05:38:06.268 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:38:06.268 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:38:06.270 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:06.269+0000 7f03869d5700 1 -- 172.21.15.160:0/2774452486 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f036c02ff60 msgr2=0x7f036c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:38:06.270 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:06.269+0000 7f03869d5700 1 --2- 172.21.15.160:0/2774452486 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f036c02ff60 0x7f036c032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f03700096a0 tx=0x7f0370006b40).stop 2023-11-17T05:38:06.270 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:06.269+0000 7f03869d5700 1 -- 172.21.15.160:0/2774452486 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03800f51a0 msgr2=0x7f03801091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:38:06.270 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:06.269+0000 7f03869d5700 1 --2- 172.21.15.160:0/2774452486 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03800f51a0 0x7f03801091b0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f037c013510 tx=0x7f037c00b040).stop 2023-11-17T05:38:06.270 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:06.269+0000 7f03869d5700 1 -- 172.21.15.160:0/2774452486 shutdown_connections 2023-11-17T05:38:06.270 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:06.269+0000 7f03869d5700 1 --2- 172.21.15.160:0/2774452486 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f036c02ff60 0x7f036c032420 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:38:06.270 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:06.269+0000 7f03869d5700 1 --2- 172.21.15.160:0/2774452486 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03800f51a0 0x7f03801091b0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:38:06.270 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:06.269+0000 7f03869d5700 1 -- 172.21.15.160:0/2774452486 >> 172.21.15.160:0/2774452486 conn(0x7f03800f09a0 msgr2=0x7f03800f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:38:06.270 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:06.269+0000 7f03869d5700 1 -- 172.21.15.160:0/2774452486 shutdown_connections 2023-11-17T05:38:06.271 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:06.269+0000 7f03869d5700 1 -- 172.21.15.160:0/2774452486 wait complete. 2023-11-17T05:38:06.271 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:38:06.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:38:06 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:38:06.268502+0000 mon.smithi003 (mon.0) 317 : audit [DBG] from='client.? 172.21.15.160:0/2774452486' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:38:07.668 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:38:07.668 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:38:07.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:38:07 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:38:06.381620+0000 mgr.smithi003.eddcuc (mgr.14164) 390 : cluster [DBG] pgmap v374: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:38:08.842 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:08.835+0000 7f46691ae700 1 -- 172.21.15.160:0/169540403 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46640f5410 msgr2=0x7f46640f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:38:08.842 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:08.835+0000 7f46691ae700 1 --2- 172.21.15.160:0/169540403 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46640f5410 0x7f46640f5830 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f464c009fb0 tx=0x7f464c009b30).stop 2023-11-17T05:38:08.843 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:08.835+0000 7f46691ae700 1 -- 172.21.15.160:0/169540403 shutdown_connections 2023-11-17T05:38:08.843 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:08.835+0000 7f46691ae700 1 --2- 172.21.15.160:0/169540403 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46640f5410 0x7f46640f5830 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:38:08.843 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:08.835+0000 7f46691ae700 1 -- 172.21.15.160:0/169540403 >> 172.21.15.160:0/169540403 conn(0x7f46640f0b70 msgr2=0x7f46640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:38:08.843 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:08.835+0000 7f46691ae700 1 -- 172.21.15.160:0/169540403 shutdown_connections 2023-11-17T05:38:08.843 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:08.835+0000 7f46691ae700 1 -- 172.21.15.160:0/169540403 wait complete. 2023-11-17T05:38:08.843 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:08.836+0000 7f46691ae700 1 Processor -- start 2023-11-17T05:38:08.843 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:08.836+0000 7f46691ae700 1 -- start start 2023-11-17T05:38:08.843 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:08.836+0000 7f46691ae700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46640f5410 0x7f46641093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:38:08.843 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:08.836+0000 7f46691ae700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4664060280 con 0x7f46640f5410 2023-11-17T05:38:08.843 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:08.837+0000 7f4662d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46640f5410 0x7f46641093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:38:08.843 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:08.837+0000 7f4662d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46640f5410 0x7f46641093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:55446/0 (socket says 172.21.15.160:55446) 2023-11-17T05:38:08.843 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:08.837+0000 7f4662d9d700 1 -- 172.21.15.160:0/2695985911 learned_addr learned my addr 172.21.15.160:0/2695985911 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:38:08.843 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:08.837+0000 7f4662d9d700 1 -- 172.21.15.160:0/2695985911 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f464c005040 con 0x7f46640f5410 2023-11-17T05:38:08.844 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:08.837+0000 7f4662d9d700 1 --2- 172.21.15.160:0/2695985911 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46640f5410 0x7f46641093f0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f464c013510 tx=0x7f464c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:38:08.844 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:08.837+0000 7f465b7fe700 1 -- 172.21.15.160:0/2695985911 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f464c009320 con 0x7f46640f5410 2023-11-17T05:38:08.844 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:08.837+0000 7f46691ae700 1 -- 172.21.15.160:0/2695985911 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4664060480 con 0x7f46640f5410 2023-11-17T05:38:08.844 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:08.837+0000 7f465b7fe700 1 -- 172.21.15.160:0/2695985911 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f464c013f00 con 0x7f46640f5410 2023-11-17T05:38:08.844 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:08.837+0000 7f465b7fe700 1 -- 172.21.15.160:0/2695985911 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f464c014d90 con 0x7f46640f5410 2023-11-17T05:38:08.844 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:08.837+0000 7f46691ae700 1 -- 172.21.15.160:0/2695985911 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4664109b20 con 0x7f46640f5410 2023-11-17T05:38:08.844 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:08.838+0000 7f465b7fe700 1 -- 172.21.15.160:0/2695985911 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f464c014420 con 0x7f46640f5410 2023-11-17T05:38:08.844 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:08.838+0000 7f46691ae700 1 -- 172.21.15.160:0/2695985911 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4664045ad0 con 0x7f46640f5410 2023-11-17T05:38:08.844 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:08.838+0000 7f465b7fe700 1 --2- 172.21.15.160:0/2695985911 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f465002ff60 0x7f4650032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:38:08.844 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:08.838+0000 7f465b7fe700 1 -- 172.21.15.160:0/2695985911 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f464c041000 con 0x7f46640f5410 2023-11-17T05:38:08.844 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:08.839+0000 7f466259c700 1 --2- 172.21.15.160:0/2695985911 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f465002ff60 0x7f4650032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:38:08.844 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:08.839+0000 7f466259c700 1 --2- 172.21.15.160:0/2695985911 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f465002ff60 0x7f4650032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f46540096a0 tx=0x7f4654006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:38:08.845 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:08.844+0000 7f465b7fe700 1 -- 172.21.15.160:0/2695985911 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f464c0095f0 con 0x7f46640f5410 2023-11-17T05:38:09.049 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:09.049+0000 7f46691ae700 1 -- 172.21.15.160:0/2695985911 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f466410a340 con 0x7f46640f5410 2023-11-17T05:38:09.051 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:09.049+0000 7f465b7fe700 1 -- 172.21.15.160:0/2695985911 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f464c01db70 con 0x7f46640f5410 2023-11-17T05:38:09.051 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:38:09.051 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:38:09.052 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:09.051+0000 7f46691ae700 1 -- 172.21.15.160:0/2695985911 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f465002ff60 msgr2=0x7f4650032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:38:09.053 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:09.051+0000 7f46691ae700 1 --2- 172.21.15.160:0/2695985911 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f465002ff60 0x7f4650032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f46540096a0 tx=0x7f4654006b40).stop 2023-11-17T05:38:09.053 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:09.051+0000 7f46691ae700 1 -- 172.21.15.160:0/2695985911 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46640f5410 msgr2=0x7f46641093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:38:09.053 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:09.051+0000 7f46691ae700 1 --2- 172.21.15.160:0/2695985911 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46640f5410 0x7f46641093f0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f464c013510 tx=0x7f464c00b040).stop 2023-11-17T05:38:09.053 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:09.051+0000 7f46691ae700 1 -- 172.21.15.160:0/2695985911 shutdown_connections 2023-11-17T05:38:09.053 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:09.051+0000 7f46691ae700 1 --2- 172.21.15.160:0/2695985911 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f465002ff60 0x7f4650032420 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:38:09.054 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:09.051+0000 7f46691ae700 1 --2- 172.21.15.160:0/2695985911 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46640f5410 0x7f46641093f0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:38:09.054 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:09.051+0000 7f46691ae700 1 -- 172.21.15.160:0/2695985911 >> 172.21.15.160:0/2695985911 conn(0x7f46640f0b70 msgr2=0x7f46640f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:38:09.054 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:09.051+0000 7f46691ae700 1 -- 172.21.15.160:0/2695985911 shutdown_connections 2023-11-17T05:38:09.054 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:09.051+0000 7f46691ae700 1 -- 172.21.15.160:0/2695985911 wait complete. 2023-11-17T05:38:09.054 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:38:09.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:38:09 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:38:08.382026+0000 mgr.smithi003.eddcuc (mgr.14164) 391 : cluster [DBG] pgmap v375: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:38:09.771 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:38:09 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:38:09.050412+0000 mon.smithi003 (mon.0) 318 : audit [DBG] from='client.? 172.21.15.160:0/2695985911' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:38:10.473 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:38:10.474 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:38:11.684 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:11.682+0000 7f609cb03700 1 -- 172.21.15.160:0/488086435 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6098060220 msgr2=0x7f60980f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:38:11.684 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:11.682+0000 7f609cb03700 1 --2- 172.21.15.160:0/488086435 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6098060220 0x7f60980f9150 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f6088009fb0 tx=0x7f6088009b30).stop 2023-11-17T05:38:11.684 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:11.682+0000 7f609cb03700 1 -- 172.21.15.160:0/488086435 shutdown_connections 2023-11-17T05:38:11.684 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:11.682+0000 7f609cb03700 1 --2- 172.21.15.160:0/488086435 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6098060220 0x7f60980f9150 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:38:11.684 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:11.682+0000 7f609cb03700 1 -- 172.21.15.160:0/488086435 >> 172.21.15.160:0/488086435 conn(0x7f60980f0b70 msgr2=0x7f60980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:38:11.684 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:11.682+0000 7f609cb03700 1 -- 172.21.15.160:0/488086435 shutdown_connections 2023-11-17T05:38:11.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:11.682+0000 7f609cb03700 1 -- 172.21.15.160:0/488086435 wait complete. 2023-11-17T05:38:11.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:11.682+0000 7f609cb03700 1 Processor -- start 2023-11-17T05:38:11.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:11.682+0000 7f609cb03700 1 -- start start 2023-11-17T05:38:11.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:11.682+0000 7f609cb03700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6098060220 0x7f60981066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:38:11.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:11.682+0000 7f609cb03700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6098106bd0 con 0x7f6098060220 2023-11-17T05:38:11.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:11.683+0000 7f609659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6098060220 0x7f60981066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:38:11.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:11.683+0000 7f609659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6098060220 0x7f60981066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:60032/0 (socket says 172.21.15.160:60032) 2023-11-17T05:38:11.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:11.683+0000 7f609659c700 1 -- 172.21.15.160:0/904406131 learned_addr learned my addr 172.21.15.160:0/904406131 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:38:11.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:11.683+0000 7f609659c700 1 -- 172.21.15.160:0/904406131 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6088005040 con 0x7f6098060220 2023-11-17T05:38:11.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:11.683+0000 7f609659c700 1 --2- 172.21.15.160:0/904406131 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6098060220 0x7f60981066e0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f6088013510 tx=0x7f608800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:38:11.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:11.683+0000 7f6086ffd700 1 -- 172.21.15.160:0/904406131 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6088009320 con 0x7f6098060220 2023-11-17T05:38:11.685 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:11.683+0000 7f609cb03700 1 -- 172.21.15.160:0/904406131 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6098106dd0 con 0x7f6098060220 2023-11-17T05:38:11.686 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:11.683+0000 7f6086ffd700 1 -- 172.21.15.160:0/904406131 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6088013f00 con 0x7f6098060220 2023-11-17T05:38:11.686 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:11.683+0000 7f6086ffd700 1 -- 172.21.15.160:0/904406131 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6088014dc0 con 0x7f6098060220 2023-11-17T05:38:11.686 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:11.683+0000 7f609cb03700 1 -- 172.21.15.160:0/904406131 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6098107260 con 0x7f6098060220 2023-11-17T05:38:11.686 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:11.684+0000 7f6086ffd700 1 -- 172.21.15.160:0/904406131 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6088014420 con 0x7f6098060220 2023-11-17T05:38:11.687 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:11.684+0000 7f609cb03700 1 -- 172.21.15.160:0/904406131 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6098045ad0 con 0x7f6098060220 2023-11-17T05:38:11.687 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:11.684+0000 7f6086ffd700 1 --2- 172.21.15.160:0/904406131 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f608002ff60 0x7f6080032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:38:11.687 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:11.684+0000 7f6086ffd700 1 -- 172.21.15.160:0/904406131 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6088041110 con 0x7f6098060220 2023-11-17T05:38:11.687 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:11.685+0000 7f6095d9b700 1 --2- 172.21.15.160:0/904406131 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f608002ff60 0x7f6080032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:38:11.687 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:11.685+0000 7f6095d9b700 1 --2- 172.21.15.160:0/904406131 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f608002ff60 0x7f6080032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f608c0096a0 tx=0x7f608c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:38:11.690 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:11.690+0000 7f6086ffd700 1 -- 172.21.15.160:0/904406131 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f60880095f0 con 0x7f6098060220 2023-11-17T05:38:11.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:38:11 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:38:10.382416+0000 mgr.smithi003.eddcuc (mgr.14164) 392 : cluster [DBG] pgmap v376: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:38:11.893 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:11.892+0000 7f609cb03700 1 -- 172.21.15.160:0/904406131 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6098108110 con 0x7f6098060220 2023-11-17T05:38:11.893 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:11.892+0000 7f6086ffd700 1 -- 172.21.15.160:0/904406131 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f608801cb10 con 0x7f6098060220 2023-11-17T05:38:11.893 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:38:11.894 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:38:11.895 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:11.894+0000 7f609cb03700 1 -- 172.21.15.160:0/904406131 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f608002ff60 msgr2=0x7f6080032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:38:11.895 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:11.894+0000 7f609cb03700 1 --2- 172.21.15.160:0/904406131 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f608002ff60 0x7f6080032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f608c0096a0 tx=0x7f608c006b40).stop 2023-11-17T05:38:11.895 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:11.894+0000 7f609cb03700 1 -- 172.21.15.160:0/904406131 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6098060220 msgr2=0x7f60981066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:38:11.895 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:11.894+0000 7f609cb03700 1 --2- 172.21.15.160:0/904406131 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6098060220 0x7f60981066e0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f6088013510 tx=0x7f608800b040).stop 2023-11-17T05:38:11.895 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:11.894+0000 7f609cb03700 1 -- 172.21.15.160:0/904406131 shutdown_connections 2023-11-17T05:38:11.895 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:11.894+0000 7f609cb03700 1 --2- 172.21.15.160:0/904406131 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f608002ff60 0x7f6080032420 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:38:11.895 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:11.894+0000 7f609cb03700 1 --2- 172.21.15.160:0/904406131 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6098060220 0x7f60981066e0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:38:11.895 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:11.894+0000 7f609cb03700 1 -- 172.21.15.160:0/904406131 >> 172.21.15.160:0/904406131 conn(0x7f60980f0b70 msgr2=0x7f60980f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:38:11.895 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:11.894+0000 7f609cb03700 1 -- 172.21.15.160:0/904406131 shutdown_connections 2023-11-17T05:38:11.895 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:11.894+0000 7f609cb03700 1 -- 172.21.15.160:0/904406131 wait complete. 2023-11-17T05:38:11.896 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:38:12.772 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:38:12 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:38:11.893519+0000 mon.smithi003 (mon.0) 319 : audit [DBG] from='client.? 172.21.15.160:0/904406131' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:38:13.273 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:38:13.273 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:38:13.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:38:13 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:38:12.382673+0000 mgr.smithi003.eddcuc (mgr.14164) 393 : cluster [DBG] pgmap v377: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:38:14.458 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:14.456+0000 7f64b4494700 1 -- 172.21.15.160:0/2999085748 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64ac0f5410 msgr2=0x7f64ac0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:38:14.458 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:14.456+0000 7f64b4494700 1 --2- 172.21.15.160:0/2999085748 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64ac0f5410 0x7f64ac0f5830 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f649c009fb0 tx=0x7f649c009b30).stop 2023-11-17T05:38:14.458 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:14.456+0000 7f64b4494700 1 -- 172.21.15.160:0/2999085748 shutdown_connections 2023-11-17T05:38:14.458 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:14.456+0000 7f64b4494700 1 --2- 172.21.15.160:0/2999085748 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64ac0f5410 0x7f64ac0f5830 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:38:14.458 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:14.456+0000 7f64b4494700 1 -- 172.21.15.160:0/2999085748 >> 172.21.15.160:0/2999085748 conn(0x7f64ac0f0b70 msgr2=0x7f64ac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:38:14.458 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:14.456+0000 7f64b4494700 1 -- 172.21.15.160:0/2999085748 shutdown_connections 2023-11-17T05:38:14.458 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:14.456+0000 7f64b4494700 1 -- 172.21.15.160:0/2999085748 wait complete. 2023-11-17T05:38:14.459 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:14.457+0000 7f64b4494700 1 Processor -- start 2023-11-17T05:38:14.459 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:14.457+0000 7f64b4494700 1 -- start start 2023-11-17T05:38:14.459 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:14.457+0000 7f64b4494700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64ac0f5410 0x7f64ac0fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:38:14.459 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:14.457+0000 7f64b4494700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64ac0fb820 con 0x7f64ac0f5410 2023-11-17T05:38:14.459 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:14.457+0000 7f64b2230700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64ac0f5410 0x7f64ac0fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:38:14.459 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:14.457+0000 7f64b2230700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64ac0f5410 0x7f64ac0fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:60064/0 (socket says 172.21.15.160:60064) 2023-11-17T05:38:14.459 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:14.457+0000 7f64b2230700 1 -- 172.21.15.160:0/1573376551 learned_addr learned my addr 172.21.15.160:0/1573376551 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:38:14.459 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:14.458+0000 7f64b2230700 1 -- 172.21.15.160:0/1573376551 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f649c005040 con 0x7f64ac0f5410 2023-11-17T05:38:14.459 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:14.458+0000 7f64b2230700 1 --2- 172.21.15.160:0/1573376551 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64ac0f5410 0x7f64ac0fd3a0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f649c013510 tx=0x7f649c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:38:14.459 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:14.458+0000 7f64a2ffd700 1 -- 172.21.15.160:0/1573376551 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f649c009320 con 0x7f64ac0f5410 2023-11-17T05:38:14.459 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:14.458+0000 7f64b4494700 1 -- 172.21.15.160:0/1573376551 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64ac0fb9c0 con 0x7f64ac0f5410 2023-11-17T05:38:14.459 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:14.458+0000 7f64a2ffd700 1 -- 172.21.15.160:0/1573376551 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f649c013f00 con 0x7f64ac0f5410 2023-11-17T05:38:14.459 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:14.458+0000 7f64a2ffd700 1 -- 172.21.15.160:0/1573376551 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f649c014d90 con 0x7f64ac0f5410 2023-11-17T05:38:14.460 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:14.458+0000 7f64b4494700 1 -- 172.21.15.160:0/1573376551 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64ac0fbe50 con 0x7f64ac0f5410 2023-11-17T05:38:14.460 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:14.459+0000 7f64a2ffd700 1 -- 172.21.15.160:0/1573376551 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f649c0095f0 con 0x7f64ac0f5410 2023-11-17T05:38:14.460 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:14.459+0000 7f64b4494700 1 -- 172.21.15.160:0/1573376551 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64ac059100 con 0x7f64ac0f5410 2023-11-17T05:38:14.461 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:14.459+0000 7f64a2ffd700 1 --2- 172.21.15.160:0/1573376551 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f649802ff60 0x7f6498032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:38:14.461 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:14.459+0000 7f64a2ffd700 1 -- 172.21.15.160:0/1573376551 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f649c040fe0 con 0x7f64ac0f5410 2023-11-17T05:38:14.461 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:14.459+0000 7f64b1a2f700 1 --2- 172.21.15.160:0/1573376551 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f649802ff60 0x7f6498032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:38:14.461 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:14.460+0000 7f64b1a2f700 1 --2- 172.21.15.160:0/1573376551 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f649802ff60 0x7f6498032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f64a80096a0 tx=0x7f64a8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:38:14.463 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:14.464+0000 7f64a2ffd700 1 -- 172.21.15.160:0/1573376551 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f649c014420 con 0x7f64ac0f5410 2023-11-17T05:38:14.672 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:14.672+0000 7f64b4494700 1 -- 172.21.15.160:0/1573376551 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f64ac0fae60 con 0x7f64ac0f5410 2023-11-17T05:38:14.673 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:14.673+0000 7f64a2ffd700 1 -- 172.21.15.160:0/1573376551 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f649c010230 con 0x7f64ac0f5410 2023-11-17T05:38:14.673 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:38:14.673 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:38:14.675 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:14.674+0000 7f64b4494700 1 -- 172.21.15.160:0/1573376551 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f649802ff60 msgr2=0x7f6498032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:38:14.675 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:14.674+0000 7f64b4494700 1 --2- 172.21.15.160:0/1573376551 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f649802ff60 0x7f6498032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f64a80096a0 tx=0x7f64a8006b40).stop 2023-11-17T05:38:14.675 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:14.674+0000 7f64b4494700 1 -- 172.21.15.160:0/1573376551 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64ac0f5410 msgr2=0x7f64ac0fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:38:14.675 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:14.674+0000 7f64b4494700 1 --2- 172.21.15.160:0/1573376551 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64ac0f5410 0x7f64ac0fd3a0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f649c013510 tx=0x7f649c00b040).stop 2023-11-17T05:38:14.675 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:14.674+0000 7f64b4494700 1 -- 172.21.15.160:0/1573376551 shutdown_connections 2023-11-17T05:38:14.675 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:14.674+0000 7f64b4494700 1 --2- 172.21.15.160:0/1573376551 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f649802ff60 0x7f6498032420 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:38:14.675 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:14.674+0000 7f64b4494700 1 --2- 172.21.15.160:0/1573376551 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64ac0f5410 0x7f64ac0fd3a0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:38:14.675 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:14.674+0000 7f64b4494700 1 -- 172.21.15.160:0/1573376551 >> 172.21.15.160:0/1573376551 conn(0x7f64ac0f0b70 msgr2=0x7f64ac05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:38:14.675 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:14.674+0000 7f64b4494700 1 -- 172.21.15.160:0/1573376551 shutdown_connections 2023-11-17T05:38:14.676 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:14.674+0000 7f64b4494700 1 -- 172.21.15.160:0/1573376551 wait complete. 2023-11-17T05:38:14.676 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:38:15.520 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:38:15 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:38:14.382911+0000 mgr.smithi003.eddcuc (mgr.14164) 394 : cluster [DBG] pgmap v378: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:38:15.520 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:38:15 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:38:14.673038+0000 mon.smithi003 (mon.0) 320 : audit [DBG] from='client.? 172.21.15.160:0/1573376551' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:38:16.070 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:38:16.070 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:38:17.230 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:17.228+0000 7f0ee357d700 1 -- 172.21.15.160:0/3277807369 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0edc0f4740 msgr2=0x7f0edc0f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:38:17.230 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:17.228+0000 7f0ee357d700 1 --2- 172.21.15.160:0/3277807369 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0edc0f4740 0x7f0edc0f4b60 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f0ecc009fb0 tx=0x7f0ecc009b30).stop 2023-11-17T05:38:17.231 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:17.228+0000 7f0ee357d700 1 -- 172.21.15.160:0/3277807369 shutdown_connections 2023-11-17T05:38:17.231 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:17.228+0000 7f0ee357d700 1 --2- 172.21.15.160:0/3277807369 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0edc0f4740 0x7f0edc0f4b60 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:38:17.231 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:17.228+0000 7f0ee357d700 1 -- 172.21.15.160:0/3277807369 >> 172.21.15.160:0/3277807369 conn(0x7f0edc0f0260 msgr2=0x7f0edc0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:38:17.231 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:17.228+0000 7f0ee357d700 1 -- 172.21.15.160:0/3277807369 shutdown_connections 2023-11-17T05:38:17.231 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:17.228+0000 7f0ee357d700 1 -- 172.21.15.160:0/3277807369 wait complete. 2023-11-17T05:38:17.231 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:17.229+0000 7f0ee357d700 1 Processor -- start 2023-11-17T05:38:17.231 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:17.229+0000 7f0ee357d700 1 -- start start 2023-11-17T05:38:17.231 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:17.229+0000 7f0ee357d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0edc0f4740 0x7f0edc1066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:38:17.231 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:17.229+0000 7f0ee357d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0edc106bc0 con 0x7f0edc0f4740 2023-11-17T05:38:17.231 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:17.230+0000 7f0ee1319700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0edc0f4740 0x7f0edc1066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:38:17.231 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:17.230+0000 7f0ee1319700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0edc0f4740 0x7f0edc1066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:60078/0 (socket says 172.21.15.160:60078) 2023-11-17T05:38:17.231 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:17.230+0000 7f0ee1319700 1 -- 172.21.15.160:0/266455225 learned_addr learned my addr 172.21.15.160:0/266455225 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:38:17.233 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:17.230+0000 7f0ee1319700 1 -- 172.21.15.160:0/266455225 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ecc005040 con 0x7f0edc0f4740 2023-11-17T05:38:17.233 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:17.230+0000 7f0ee1319700 1 --2- 172.21.15.160:0/266455225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0edc0f4740 0x7f0edc1066d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f0ecc014000 tx=0x7f0ecc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:38:17.233 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:17.230+0000 7f0ed1ffb700 1 -- 172.21.15.160:0/266455225 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ecc0095f0 con 0x7f0edc0f4740 2023-11-17T05:38:17.233 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:17.230+0000 7f0ed1ffb700 1 -- 172.21.15.160:0/266455225 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0ecc007e70 con 0x7f0edc0f4740 2023-11-17T05:38:17.233 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:17.230+0000 7f0ee357d700 1 -- 172.21.15.160:0/266455225 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0edc106dc0 con 0x7f0edc0f4740 2023-11-17T05:38:17.233 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:17.230+0000 7f0ed1ffb700 1 -- 172.21.15.160:0/266455225 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ecc014ea0 con 0x7f0edc0f4740 2023-11-17T05:38:17.233 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:17.231+0000 7f0ee357d700 1 -- 172.21.15.160:0/266455225 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0edc107250 con 0x7f0edc0f4740 2023-11-17T05:38:17.233 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:17.231+0000 7f0ee357d700 1 -- 172.21.15.160:0/266455225 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0edc045ad0 con 0x7f0edc0f4740 2023-11-17T05:38:17.233 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:17.231+0000 7f0ed1ffb700 1 -- 172.21.15.160:0/266455225 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0ecc025070 con 0x7f0edc0f4740 2023-11-17T05:38:17.233 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:17.231+0000 7f0ed1ffb700 1 --2- 172.21.15.160:0/266455225 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f0ec802ff60 0x7f0ec8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:38:17.234 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:17.232+0000 7f0ed1ffb700 1 -- 172.21.15.160:0/266455225 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0ecc040970 con 0x7f0edc0f4740 2023-11-17T05:38:17.234 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:17.232+0000 7f0ee0b18700 1 --2- 172.21.15.160:0/266455225 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f0ec802ff60 0x7f0ec8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:38:17.234 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:17.232+0000 7f0ee0b18700 1 --2- 172.21.15.160:0/266455225 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f0ec802ff60 0x7f0ec8032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f0ed80096a0 tx=0x7f0ed8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:38:17.238 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:17.236+0000 7f0ed1ffb700 1 -- 172.21.15.160:0/266455225 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0ecc0079c0 con 0x7f0edc0f4740 2023-11-17T05:38:17.444 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:17.441+0000 7f0ee357d700 1 -- 172.21.15.160:0/266455225 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0edc108100 con 0x7f0edc0f4740 2023-11-17T05:38:17.445 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:17.442+0000 7f0ed1ffb700 1 -- 172.21.15.160:0/266455225 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f0ecc00f300 con 0x7f0edc0f4740 2023-11-17T05:38:17.445 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:38:17.445 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:38:17.446 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:17.444+0000 7f0ee357d700 1 -- 172.21.15.160:0/266455225 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f0ec802ff60 msgr2=0x7f0ec8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:38:17.446 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:17.444+0000 7f0ee357d700 1 --2- 172.21.15.160:0/266455225 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f0ec802ff60 0x7f0ec8032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f0ed80096a0 tx=0x7f0ed8006b40).stop 2023-11-17T05:38:17.446 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:17.444+0000 7f0ee357d700 1 -- 172.21.15.160:0/266455225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0edc0f4740 msgr2=0x7f0edc1066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:38:17.446 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:17.444+0000 7f0ee357d700 1 --2- 172.21.15.160:0/266455225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0edc0f4740 0x7f0edc1066d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f0ecc014000 tx=0x7f0ecc00b040).stop 2023-11-17T05:38:17.446 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:17.444+0000 7f0ee357d700 1 -- 172.21.15.160:0/266455225 shutdown_connections 2023-11-17T05:38:17.446 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:17.444+0000 7f0ee357d700 1 --2- 172.21.15.160:0/266455225 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f0ec802ff60 0x7f0ec8032420 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:38:17.446 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:17.444+0000 7f0ee357d700 1 --2- 172.21.15.160:0/266455225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0edc0f4740 0x7f0edc1066d0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:38:17.447 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:17.444+0000 7f0ee357d700 1 -- 172.21.15.160:0/266455225 >> 172.21.15.160:0/266455225 conn(0x7f0edc0f0260 msgr2=0x7f0edc0f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:38:17.447 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:17.444+0000 7f0ee357d700 1 -- 172.21.15.160:0/266455225 shutdown_connections 2023-11-17T05:38:17.447 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:17.444+0000 7f0ee357d700 1 -- 172.21.15.160:0/266455225 wait complete. 2023-11-17T05:38:17.447 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:38:17.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:38:17 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:38:16.383248+0000 mgr.smithi003.eddcuc (mgr.14164) 395 : cluster [DBG] pgmap v379: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:38:18.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:38:18 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:38:17.443081+0000 mon.smithi003 (mon.0) 321 : audit [DBG] from='client.? 172.21.15.160:0/266455225' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:38:18.827 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:38:18.828 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:38:19.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:38:19 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:38:18.383681+0000 mgr.smithi003.eddcuc (mgr.14164) 396 : cluster [DBG] pgmap v380: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:38:20.054 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:20.052+0000 7fe576e72700 1 -- 172.21.15.160:0/3221574635 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5700f6510 msgr2=0x7fe5700f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:38:20.054 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:20.052+0000 7fe576e72700 1 --2- 172.21.15.160:0/3221574635 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5700f6510 0x7fe5700f6930 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7fe564009fb0 tx=0x7fe564009b30).stop 2023-11-17T05:38:20.054 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:20.052+0000 7fe576e72700 1 -- 172.21.15.160:0/3221574635 shutdown_connections 2023-11-17T05:38:20.054 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:20.052+0000 7fe576e72700 1 --2- 172.21.15.160:0/3221574635 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5700f6510 0x7fe5700f6930 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:38:20.054 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:20.052+0000 7fe576e72700 1 -- 172.21.15.160:0/3221574635 >> 172.21.15.160:0/3221574635 conn(0x7fe5700f1c70 msgr2=0x7fe5700f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:38:20.055 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:20.053+0000 7fe576e72700 1 -- 172.21.15.160:0/3221574635 shutdown_connections 2023-11-17T05:38:20.055 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:20.053+0000 7fe576e72700 1 -- 172.21.15.160:0/3221574635 wait complete. 2023-11-17T05:38:20.055 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:20.053+0000 7fe576e72700 1 Processor -- start 2023-11-17T05:38:20.055 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:20.053+0000 7fe576e72700 1 -- start start 2023-11-17T05:38:20.055 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:20.053+0000 7fe576e72700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5700f6510 0x7fe570106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:38:20.055 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:20.053+0000 7fe576e72700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe570106c20 con 0x7fe5700f6510 2023-11-17T05:38:20.058 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:20.054+0000 7fe574c0e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5700f6510 0x7fe570106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:38:20.058 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:20.054+0000 7fe574c0e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5700f6510 0x7fe570106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:60098/0 (socket says 172.21.15.160:60098) 2023-11-17T05:38:20.059 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:20.054+0000 7fe574c0e700 1 -- 172.21.15.160:0/3217936284 learned_addr learned my addr 172.21.15.160:0/3217936284 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:38:20.059 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:20.054+0000 7fe574c0e700 1 -- 172.21.15.160:0/3217936284 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe564005040 con 0x7fe5700f6510 2023-11-17T05:38:20.059 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:20.054+0000 7fe574c0e700 1 --2- 172.21.15.160:0/3217936284 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5700f6510 0x7fe570106730 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fe564013510 tx=0x7fe56400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:38:20.059 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:20.054+0000 7fe56d7fa700 1 -- 172.21.15.160:0/3217936284 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe564009320 con 0x7fe5700f6510 2023-11-17T05:38:20.060 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:20.054+0000 7fe56d7fa700 1 -- 172.21.15.160:0/3217936284 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe564013f00 con 0x7fe5700f6510 2023-11-17T05:38:20.060 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:20.054+0000 7fe576e72700 1 -- 172.21.15.160:0/3217936284 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe570106e20 con 0x7fe5700f6510 2023-11-17T05:38:20.060 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:20.054+0000 7fe56d7fa700 1 -- 172.21.15.160:0/3217936284 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe564014d90 con 0x7fe5700f6510 2023-11-17T05:38:20.060 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:20.054+0000 7fe576e72700 1 -- 172.21.15.160:0/3217936284 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5701072b0 con 0x7fe5700f6510 2023-11-17T05:38:20.060 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:20.055+0000 7fe56d7fa700 1 -- 172.21.15.160:0/3217936284 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe5640095f0 con 0x7fe5700f6510 2023-11-17T05:38:20.061 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:20.055+0000 7fe576e72700 1 -- 172.21.15.160:0/3217936284 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe570045ad0 con 0x7fe5700f6510 2023-11-17T05:38:20.061 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:20.056+0000 7fe56d7fa700 1 --2- 172.21.15.160:0/3217936284 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe56002ff60 0x7fe560032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:38:20.061 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:20.056+0000 7fe56d7fa700 1 -- 172.21.15.160:0/3217936284 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe564040fe0 con 0x7fe5700f6510 2023-11-17T05:38:20.061 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:20.056+0000 7fe56ffff700 1 --2- 172.21.15.160:0/3217936284 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe56002ff60 0x7fe560032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:38:20.061 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:20.056+0000 7fe56ffff700 1 --2- 172.21.15.160:0/3217936284 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe56002ff60 0x7fe560032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fe55c0096a0 tx=0x7fe55c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:38:20.062 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:20.061+0000 7fe56d7fa700 1 -- 172.21.15.160:0/3217936284 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe56401d7f0 con 0x7fe5700f6510 2023-11-17T05:38:20.266 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:20.264+0000 7fe576e72700 1 -- 172.21.15.160:0/3217936284 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe5700fbf60 con 0x7fe5700f6510 2023-11-17T05:38:20.266 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:20.264+0000 7fe56d7fa700 1 -- 172.21.15.160:0/3217936284 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fe564010230 con 0x7fe5700f6510 2023-11-17T05:38:20.268 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:38:20.268 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:38:20.269 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:20.265+0000 7fe576e72700 1 -- 172.21.15.160:0/3217936284 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe56002ff60 msgr2=0x7fe560032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:38:20.269 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:20.266+0000 7fe576e72700 1 --2- 172.21.15.160:0/3217936284 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe56002ff60 0x7fe560032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fe55c0096a0 tx=0x7fe55c006b40).stop 2023-11-17T05:38:20.269 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:20.266+0000 7fe576e72700 1 -- 172.21.15.160:0/3217936284 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5700f6510 msgr2=0x7fe570106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:38:20.269 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:20.266+0000 7fe576e72700 1 --2- 172.21.15.160:0/3217936284 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5700f6510 0x7fe570106730 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fe564013510 tx=0x7fe56400b040).stop 2023-11-17T05:38:20.269 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:20.266+0000 7fe576e72700 1 -- 172.21.15.160:0/3217936284 shutdown_connections 2023-11-17T05:38:20.269 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:20.266+0000 7fe576e72700 1 --2- 172.21.15.160:0/3217936284 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe56002ff60 0x7fe560032420 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:38:20.269 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:20.266+0000 7fe576e72700 1 --2- 172.21.15.160:0/3217936284 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe5700f6510 0x7fe570106730 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:38:20.269 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:20.266+0000 7fe576e72700 1 -- 172.21.15.160:0/3217936284 >> 172.21.15.160:0/3217936284 conn(0x7fe5700f1c70 msgr2=0x7fe5700f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:38:20.269 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:20.266+0000 7fe576e72700 1 -- 172.21.15.160:0/3217936284 shutdown_connections 2023-11-17T05:38:20.270 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:20.266+0000 7fe576e72700 1 -- 172.21.15.160:0/3217936284 wait complete. 2023-11-17T05:38:20.270 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:38:20.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:38:20 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:38:20.265340+0000 mon.smithi003 (mon.0) 322 : audit [DBG] from='client.? 172.21.15.160:0/3217936284' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:38:21.679 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:38:21.679 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:38:21.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:38:21 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:38:20.384055+0000 mgr.smithi003.eddcuc (mgr.14164) 397 : cluster [DBG] pgmap v381: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:38:22.899 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:22.897+0000 7f116bfff700 1 -- 172.21.15.160:0/135515538 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f116c0f51a0 msgr2=0x7f116c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:38:22.899 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:22.897+0000 7f116bfff700 1 --2- 172.21.15.160:0/135515538 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f116c0f51a0 0x7f116c0f55c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f1160009fb0 tx=0x7f1160009b30).stop 2023-11-17T05:38:22.899 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:22.897+0000 7f116bfff700 1 -- 172.21.15.160:0/135515538 shutdown_connections 2023-11-17T05:38:22.899 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:22.897+0000 7f116bfff700 1 --2- 172.21.15.160:0/135515538 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f116c0f51a0 0x7f116c0f55c0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:38:22.899 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:22.897+0000 7f116bfff700 1 -- 172.21.15.160:0/135515538 >> 172.21.15.160:0/135515538 conn(0x7f116c0f09a0 msgr2=0x7f116c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:38:22.899 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:22.897+0000 7f116bfff700 1 -- 172.21.15.160:0/135515538 shutdown_connections 2023-11-17T05:38:22.899 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:22.897+0000 7f116bfff700 1 -- 172.21.15.160:0/135515538 wait complete. 2023-11-17T05:38:22.899 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:22.898+0000 7f116bfff700 1 Processor -- start 2023-11-17T05:38:22.899 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:22.898+0000 7f116bfff700 1 -- start start 2023-11-17T05:38:22.899 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:22.898+0000 7f116bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f116c0f51a0 0x7f116c1053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:38:22.899 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:22.898+0000 7f116bfff700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f116c1058a0 con 0x7f116c0f51a0 2023-11-17T05:38:22.901 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:22.898+0000 7f116affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f116c0f51a0 0x7f116c1053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:38:22.901 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:22.898+0000 7f116affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f116c0f51a0 0x7f116c1053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:47664/0 (socket says 172.21.15.160:47664) 2023-11-17T05:38:22.901 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:22.898+0000 7f116affd700 1 -- 172.21.15.160:0/470924571 learned_addr learned my addr 172.21.15.160:0/470924571 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:38:22.901 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:22.899+0000 7f116affd700 1 -- 172.21.15.160:0/470924571 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1160005040 con 0x7f116c0f51a0 2023-11-17T05:38:22.901 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:22.899+0000 7f116affd700 1 --2- 172.21.15.160:0/470924571 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f116c0f51a0 0x7f116c1053b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f1160009830 tx=0x7f116000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:38:22.901 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:22.899+0000 7f11537fe700 1 -- 172.21.15.160:0/470924571 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1160009320 con 0x7f116c0f51a0 2023-11-17T05:38:22.901 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:22.899+0000 7f116bfff700 1 -- 172.21.15.160:0/470924571 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f116c105aa0 con 0x7f116c0f51a0 2023-11-17T05:38:22.901 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:22.899+0000 7f11537fe700 1 -- 172.21.15.160:0/470924571 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f11600136c0 con 0x7f116c0f51a0 2023-11-17T05:38:22.902 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:22.899+0000 7f11537fe700 1 -- 172.21.15.160:0/470924571 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f116001ee90 con 0x7f116c0f51a0 2023-11-17T05:38:22.902 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:22.899+0000 7f116bfff700 1 -- 172.21.15.160:0/470924571 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f116c105f30 con 0x7f116c0f51a0 2023-11-17T05:38:22.902 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:22.900+0000 7f11537fe700 1 -- 172.21.15.160:0/470924571 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f116001e420 con 0x7f116c0f51a0 2023-11-17T05:38:22.902 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:22.900+0000 7f116bfff700 1 -- 172.21.15.160:0/470924571 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f116c045ad0 con 0x7f116c0f51a0 2023-11-17T05:38:22.902 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:22.900+0000 7f11537fe700 1 --2- 172.21.15.160:0/470924571 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f115402ff10 0x7f11540323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:38:22.902 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:22.900+0000 7f11537fe700 1 -- 172.21.15.160:0/470924571 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1160040fa0 con 0x7f116c0f51a0 2023-11-17T05:38:22.902 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:22.900+0000 7f116a7fc700 1 --2- 172.21.15.160:0/470924571 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f115402ff10 0x7f11540323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:38:22.902 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:22.900+0000 7f116a7fc700 1 --2- 172.21.15.160:0/470924571 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f115402ff10 0x7f11540323d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f115c0096a0 tx=0x7f115c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:38:22.905 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:22.905+0000 7f11537fe700 1 -- 172.21.15.160:0/470924571 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f11600095f0 con 0x7f116c0f51a0 2023-11-17T05:38:23.110 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:23.109+0000 7f116bfff700 1 -- 172.21.15.160:0/470924571 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f116c106d80 con 0x7f116c0f51a0 2023-11-17T05:38:23.111 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:23.110+0000 7f11537fe700 1 -- 172.21.15.160:0/470924571 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f116002c2c0 con 0x7f116c0f51a0 2023-11-17T05:38:23.111 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:38:23.111 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:38:23.114 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:23.111+0000 7f116bfff700 1 -- 172.21.15.160:0/470924571 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f115402ff10 msgr2=0x7f11540323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:38:23.114 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:23.111+0000 7f116bfff700 1 --2- 172.21.15.160:0/470924571 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f115402ff10 0x7f11540323d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f115c0096a0 tx=0x7f115c006b40).stop 2023-11-17T05:38:23.114 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:23.111+0000 7f116bfff700 1 -- 172.21.15.160:0/470924571 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f116c0f51a0 msgr2=0x7f116c1053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:38:23.114 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:23.111+0000 7f116bfff700 1 --2- 172.21.15.160:0/470924571 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f116c0f51a0 0x7f116c1053b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f1160009830 tx=0x7f116000b040).stop 2023-11-17T05:38:23.114 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:23.111+0000 7f116bfff700 1 -- 172.21.15.160:0/470924571 shutdown_connections 2023-11-17T05:38:23.114 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:23.111+0000 7f116bfff700 1 --2- 172.21.15.160:0/470924571 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f115402ff10 0x7f11540323d0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:38:23.114 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:23.111+0000 7f116bfff700 1 --2- 172.21.15.160:0/470924571 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f116c0f51a0 0x7f116c1053b0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:38:23.114 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:23.111+0000 7f116bfff700 1 -- 172.21.15.160:0/470924571 >> 172.21.15.160:0/470924571 conn(0x7f116c0f09a0 msgr2=0x7f116c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:38:23.115 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:23.111+0000 7f116bfff700 1 -- 172.21.15.160:0/470924571 shutdown_connections 2023-11-17T05:38:23.115 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:23.112+0000 7f116bfff700 1 -- 172.21.15.160:0/470924571 wait complete. 2023-11-17T05:38:23.115 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:38:23.773 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:38:23 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:38:22.384403+0000 mgr.smithi003.eddcuc (mgr.14164) 398 : cluster [DBG] pgmap v382: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:38:23.773 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:38:23 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:38:23.111155+0000 mon.smithi003 (mon.0) 323 : audit [DBG] from='client.? 172.21.15.160:0/470924571' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:38:24.496 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:38:24.497 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:38:25.520 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:38:25 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:38:24.384671+0000 mgr.smithi003.eddcuc (mgr.14164) 399 : cluster [DBG] pgmap v383: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:38:25.743 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:25.742+0000 7f541fdcc700 1 -- 172.21.15.160:0/3309467078 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54180601e0 msgr2=0x7f54180f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:38:25.743 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:25.742+0000 7f541fdcc700 1 --2- 172.21.15.160:0/3309467078 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54180601e0 0x7f54180f9190 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f540c009fb0 tx=0x7f540c009b30).stop 2023-11-17T05:38:25.744 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:25.742+0000 7f541fdcc700 1 -- 172.21.15.160:0/3309467078 shutdown_connections 2023-11-17T05:38:25.744 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:25.742+0000 7f541fdcc700 1 --2- 172.21.15.160:0/3309467078 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54180601e0 0x7f54180f9190 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:38:25.744 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:25.742+0000 7f541fdcc700 1 -- 172.21.15.160:0/3309467078 >> 172.21.15.160:0/3309467078 conn(0x7f54180f0b70 msgr2=0x7f54180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:38:25.744 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:25.742+0000 7f541fdcc700 1 -- 172.21.15.160:0/3309467078 shutdown_connections 2023-11-17T05:38:25.744 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:25.742+0000 7f541fdcc700 1 -- 172.21.15.160:0/3309467078 wait complete. 2023-11-17T05:38:25.746 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:25.743+0000 7f541fdcc700 1 Processor -- start 2023-11-17T05:38:25.746 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:25.743+0000 7f541fdcc700 1 -- start start 2023-11-17T05:38:25.746 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:25.743+0000 7f541fdcc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54180f6bd0 0x7f54180f5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:38:25.746 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:25.743+0000 7f541fdcc700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54180f6ff0 con 0x7f54180f6bd0 2023-11-17T05:38:25.746 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:25.743+0000 7f541db68700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54180f6bd0 0x7f54180f5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:38:25.746 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:25.743+0000 7f541db68700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54180f6bd0 0x7f54180f5090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:47694/0 (socket says 172.21.15.160:47694) 2023-11-17T05:38:25.746 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:25.743+0000 7f541db68700 1 -- 172.21.15.160:0/3075122135 learned_addr learned my addr 172.21.15.160:0/3075122135 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:38:25.746 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:25.744+0000 7f541db68700 1 -- 172.21.15.160:0/3075122135 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f540c005040 con 0x7f54180f6bd0 2023-11-17T05:38:25.746 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:25.744+0000 7f541db68700 1 --2- 172.21.15.160:0/3075122135 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54180f6bd0 0x7f54180f5090 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f540c009f80 tx=0x7f540c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:38:25.746 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:25.744+0000 7f540a7fc700 1 -- 172.21.15.160:0/3075122135 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f540c00e040 con 0x7f54180f6bd0 2023-11-17T05:38:25.746 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:25.744+0000 7f541fdcc700 1 -- 172.21.15.160:0/3075122135 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54180f5580 con 0x7f54180f6bd0 2023-11-17T05:38:25.746 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:25.744+0000 7f540a7fc700 1 -- 172.21.15.160:0/3075122135 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f540c013410 con 0x7f54180f6bd0 2023-11-17T05:38:25.747 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:25.744+0000 7f540a7fc700 1 -- 172.21.15.160:0/3075122135 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f540c014410 con 0x7f54180f6bd0 2023-11-17T05:38:25.747 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:25.744+0000 7f541fdcc700 1 -- 172.21.15.160:0/3075122135 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54180f5a70 con 0x7f54180f6bd0 2023-11-17T05:38:25.747 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:25.745+0000 7f540a7fc700 1 -- 172.21.15.160:0/3075122135 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f540c0146d0 con 0x7f54180f6bd0 2023-11-17T05:38:25.747 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:25.745+0000 7f541fdcc700 1 -- 172.21.15.160:0/3075122135 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5418045ad0 con 0x7f54180f6bd0 2023-11-17T05:38:25.747 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:25.745+0000 7f540a7fc700 1 --2- 172.21.15.160:0/3075122135 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f540402ff60 0x7f5404032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:38:25.747 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:25.745+0000 7f540a7fc700 1 -- 172.21.15.160:0/3075122135 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f540c041410 con 0x7f54180f6bd0 2023-11-17T05:38:25.747 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:25.745+0000 7f541d367700 1 --2- 172.21.15.160:0/3075122135 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f540402ff60 0x7f5404032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:38:25.748 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:25.746+0000 7f541d367700 1 --2- 172.21.15.160:0/3075122135 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f540402ff60 0x7f5404032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f54140096a0 tx=0x7f5414006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:38:25.751 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:25.750+0000 7f540a7fc700 1 -- 172.21.15.160:0/3075122135 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f540c013820 con 0x7f54180f6bd0 2023-11-17T05:38:25.959 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:25.958+0000 7f541fdcc700 1 -- 172.21.15.160:0/3075122135 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f541810c9f0 con 0x7f54180f6bd0 2023-11-17T05:38:25.960 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:25.959+0000 7f540a7fc700 1 -- 172.21.15.160:0/3075122135 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f540c026b80 con 0x7f54180f6bd0 2023-11-17T05:38:25.961 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:38:25.961 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:38:25.962 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:25.961+0000 7f541fdcc700 1 -- 172.21.15.160:0/3075122135 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f540402ff60 msgr2=0x7f5404032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:38:25.962 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:25.961+0000 7f541fdcc700 1 --2- 172.21.15.160:0/3075122135 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f540402ff60 0x7f5404032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f54140096a0 tx=0x7f5414006b40).stop 2023-11-17T05:38:25.962 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:25.961+0000 7f541fdcc700 1 -- 172.21.15.160:0/3075122135 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54180f6bd0 msgr2=0x7f54180f5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:38:25.962 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:25.961+0000 7f541fdcc700 1 --2- 172.21.15.160:0/3075122135 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54180f6bd0 0x7f54180f5090 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f540c009f80 tx=0x7f540c009b30).stop 2023-11-17T05:38:25.963 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:25.961+0000 7f541fdcc700 1 -- 172.21.15.160:0/3075122135 shutdown_connections 2023-11-17T05:38:25.963 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:25.961+0000 7f541fdcc700 1 --2- 172.21.15.160:0/3075122135 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f540402ff60 0x7f5404032420 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:38:25.963 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:25.961+0000 7f541fdcc700 1 --2- 172.21.15.160:0/3075122135 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54180f6bd0 0x7f54180f5090 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:38:25.963 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:25.961+0000 7f541fdcc700 1 -- 172.21.15.160:0/3075122135 >> 172.21.15.160:0/3075122135 conn(0x7f54180f0b70 msgr2=0x7f54180f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:38:25.963 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:25.961+0000 7f541fdcc700 1 -- 172.21.15.160:0/3075122135 shutdown_connections 2023-11-17T05:38:25.963 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:25.961+0000 7f541fdcc700 1 -- 172.21.15.160:0/3075122135 wait complete. 2023-11-17T05:38:25.964 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:38:26.520 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:38:26 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:38:25.960354+0000 mon.smithi003 (mon.0) 324 : audit [DBG] from='client.? 172.21.15.160:0/3075122135' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:38:56.864 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:38:56.865 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:38:56.882 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:38:27 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:38:26.384938+0000 mgr.smithi003.eddcuc (mgr.14164) 400 : cluster [DBG] pgmap v384: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:38:56.882 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:38:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:38:27.559443+0000 mon.smithi003 (mon.0) 325 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.eddcuc/mirror_snapshot_schedule"}]: dispatch 2023-11-17T05:38:56.883 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:38:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:38:27.562500+0000 mon.smithi003 (mon.0) 326 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.eddcuc/trash_purge_schedule"}]: dispatch 2023-11-17T05:38:56.883 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:38:29 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:38:28.385295+0000 mgr.smithi003.eddcuc (mgr.14164) 401 : cluster [DBG] pgmap v385: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:38:56.883 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:38:31 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:38:30.385699+0000 mgr.smithi003.eddcuc (mgr.14164) 402 : cluster [DBG] pgmap v386: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:38:56.883 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:38:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:38:32.386007+0000 mgr.smithi003.eddcuc (mgr.14164) 403 : cluster [DBG] pgmap v387: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:38:56.883 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:38:35 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:38:34.386354+0000 mgr.smithi003.eddcuc (mgr.14164) 404 : cluster [DBG] pgmap v388: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:38:56.883 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:38:37 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:38:36.386759+0000 mgr.smithi003.eddcuc (mgr.14164) 405 : cluster [DBG] pgmap v389: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:38:56.884 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:38:39 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:38:38.387187+0000 mgr.smithi003.eddcuc (mgr.14164) 406 : cluster [DBG] pgmap v390: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:38:56.958 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:38:41 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:38:40.387522+0000 mgr.smithi003.eddcuc (mgr.14164) 407 : cluster [DBG] pgmap v391: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:38:56.958 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:38:43 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:38:42.387801+0000 mgr.smithi003.eddcuc (mgr.14164) 408 : cluster [DBG] pgmap v392: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:38:56.958 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:38:45 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:38:44.388162+0000 mgr.smithi003.eddcuc (mgr.14164) 409 : cluster [DBG] pgmap v393: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:38:56.958 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:38:47 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:38:46.388547+0000 mgr.smithi003.eddcuc (mgr.14164) 410 : cluster [DBG] pgmap v394: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:38:56.958 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:38:49 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:38:48.388842+0000 mgr.smithi003.eddcuc (mgr.14164) 411 : cluster [DBG] pgmap v395: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:38:56.959 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:38:51 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:38:50.389117+0000 mgr.smithi003.eddcuc (mgr.14164) 412 : cluster [DBG] pgmap v396: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:38:56.959 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:38:53 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:38:52.389480+0000 mgr.smithi003.eddcuc (mgr.14164) 413 : cluster [DBG] pgmap v397: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:38:56.959 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:38:55 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:38:54.389712+0000 mgr.smithi003.eddcuc (mgr.14164) 414 : cluster [DBG] pgmap v398: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:38:57.774 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:38:57 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:38:56.389967+0000 mgr.smithi003.eddcuc (mgr.14164) 415 : cluster [DBG] pgmap v399: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:38:58.088 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:58.087+0000 7f1acef2b700 1 -- 172.21.15.160:0/3481375874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ac80f6510 msgr2=0x7f1ac80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:38:58.088 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:58.087+0000 7f1acef2b700 1 --2- 172.21.15.160:0/3481375874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ac80f6510 0x7f1ac80f6930 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f1abc009fb0 tx=0x7f1abc009b30).stop 2023-11-17T05:38:58.088 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:58.087+0000 7f1acef2b700 1 -- 172.21.15.160:0/3481375874 shutdown_connections 2023-11-17T05:38:58.088 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:58.087+0000 7f1acef2b700 1 --2- 172.21.15.160:0/3481375874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ac80f6510 0x7f1ac80f6930 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:38:58.088 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:58.087+0000 7f1acef2b700 1 -- 172.21.15.160:0/3481375874 >> 172.21.15.160:0/3481375874 conn(0x7f1ac80f1c70 msgr2=0x7f1ac80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:38:58.088 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:58.087+0000 7f1acef2b700 1 -- 172.21.15.160:0/3481375874 shutdown_connections 2023-11-17T05:38:58.088 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:58.087+0000 7f1acef2b700 1 -- 172.21.15.160:0/3481375874 wait complete. 2023-11-17T05:38:58.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:58.087+0000 7f1acef2b700 1 Processor -- start 2023-11-17T05:38:58.089 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:58.087+0000 7f1acef2b700 1 -- start start 2023-11-17T05:38:58.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:58.088+0000 7f1acef2b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ac80f6510 0x7f1ac806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:38:58.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:58.088+0000 7f1acef2b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac8069e20 con 0x7f1ac80f6510 2023-11-17T05:38:58.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:58.088+0000 7f1acccc7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ac80f6510 0x7f1ac806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:38:58.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:58.088+0000 7f1acccc7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ac80f6510 0x7f1ac806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:59288/0 (socket says 172.21.15.160:59288) 2023-11-17T05:38:58.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:58.088+0000 7f1acccc7700 1 -- 172.21.15.160:0/1335102552 learned_addr learned my addr 172.21.15.160:0/1335102552 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:38:58.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:58.088+0000 7f1acccc7700 1 -- 172.21.15.160:0/1335102552 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1abc005040 con 0x7f1ac80f6510 2023-11-17T05:38:58.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:58.088+0000 7f1acccc7700 1 --2- 172.21.15.160:0/1335102552 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ac80f6510 0x7f1ac806b9a0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f1abc013510 tx=0x7f1abc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:38:58.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:58.088+0000 7f1ac57fa700 1 -- 172.21.15.160:0/1335102552 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1abc009320 con 0x7f1ac80f6510 2023-11-17T05:38:58.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:58.088+0000 7f1acef2b700 1 -- 172.21.15.160:0/1335102552 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ac8069fc0 con 0x7f1ac80f6510 2023-11-17T05:38:58.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:58.088+0000 7f1ac57fa700 1 -- 172.21.15.160:0/1335102552 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1abc013f00 con 0x7f1ac80f6510 2023-11-17T05:38:58.090 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:58.088+0000 7f1ac57fa700 1 -- 172.21.15.160:0/1335102552 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1abc014d90 con 0x7f1ac80f6510 2023-11-17T05:38:58.091 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:58.088+0000 7f1acef2b700 1 -- 172.21.15.160:0/1335102552 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ac806a450 con 0x7f1ac80f6510 2023-11-17T05:38:58.091 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:58.089+0000 7f1ac57fa700 1 -- 172.21.15.160:0/1335102552 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1abc0095f0 con 0x7f1ac80f6510 2023-11-17T05:38:58.091 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:58.089+0000 7f1acef2b700 1 -- 172.21.15.160:0/1335102552 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ac8059100 con 0x7f1ac80f6510 2023-11-17T05:38:58.091 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:58.089+0000 7f1ac57fa700 1 --2- 172.21.15.160:0/1335102552 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f1ab802ff60 0x7f1ab8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:38:58.091 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:58.089+0000 7f1ac57fa700 1 -- 172.21.15.160:0/1335102552 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1abc0410e0 con 0x7f1ac80f6510 2023-11-17T05:38:58.091 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:58.090+0000 7f1ac7fff700 1 --2- 172.21.15.160:0/1335102552 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f1ab802ff60 0x7f1ab8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:38:58.092 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:58.090+0000 7f1ac7fff700 1 --2- 172.21.15.160:0/1335102552 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f1ab802ff60 0x7f1ab8032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f1ab40096a0 tx=0x7f1ab4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:38:58.095 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:58.095+0000 7f1ac57fa700 1 -- 172.21.15.160:0/1335102552 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1abc014420 con 0x7f1ac80f6510 2023-11-17T05:38:58.302 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:58.302+0000 7f1acef2b700 1 -- 172.21.15.160:0/1335102552 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1ac80fbf60 con 0x7f1ac80f6510 2023-11-17T05:38:58.303 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:58.302+0000 7f1ac57fa700 1 -- 172.21.15.160:0/1335102552 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f1abc01d7f0 con 0x7f1ac80f6510 2023-11-17T05:38:58.303 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:38:58.304 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:38:58.305 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:58.304+0000 7f1acef2b700 1 -- 172.21.15.160:0/1335102552 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f1ab802ff60 msgr2=0x7f1ab8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:38:58.305 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:58.304+0000 7f1acef2b700 1 --2- 172.21.15.160:0/1335102552 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f1ab802ff60 0x7f1ab8032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f1ab40096a0 tx=0x7f1ab4006b40).stop 2023-11-17T05:38:58.305 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:58.304+0000 7f1acef2b700 1 -- 172.21.15.160:0/1335102552 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ac80f6510 msgr2=0x7f1ac806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:38:58.305 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:58.304+0000 7f1acef2b700 1 --2- 172.21.15.160:0/1335102552 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ac80f6510 0x7f1ac806b9a0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f1abc013510 tx=0x7f1abc00b040).stop 2023-11-17T05:38:58.305 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:58.304+0000 7f1acef2b700 1 -- 172.21.15.160:0/1335102552 shutdown_connections 2023-11-17T05:38:58.306 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:58.304+0000 7f1acef2b700 1 --2- 172.21.15.160:0/1335102552 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f1ab802ff60 0x7f1ab8032420 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:38:58.306 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:58.304+0000 7f1acef2b700 1 --2- 172.21.15.160:0/1335102552 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ac80f6510 0x7f1ac806b9a0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:38:58.306 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:58.304+0000 7f1acef2b700 1 -- 172.21.15.160:0/1335102552 >> 172.21.15.160:0/1335102552 conn(0x7f1ac80f1c70 msgr2=0x7f1ac80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:38:58.306 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:58.304+0000 7f1acef2b700 1 -- 172.21.15.160:0/1335102552 shutdown_connections 2023-11-17T05:38:58.306 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:38:58.304+0000 7f1acef2b700 1 -- 172.21.15.160:0/1335102552 wait complete. 2023-11-17T05:38:58.306 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:38:58.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:38:58 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:38:58.303551+0000 mon.smithi003 (mon.0) 327 : audit [DBG] from='client.? 172.21.15.160:0/1335102552' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:38:59.696 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:38:59.697 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:38:59.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:38:59 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:38:58.390317+0000 mgr.smithi003.eddcuc (mgr.14164) 416 : cluster [DBG] pgmap v400: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:39:00.794 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:00.793+0000 7f0ad54d3700 1 -- 172.21.15.160:0/4164841958 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0ad00f47c0 msgr2=0x7f0ad00f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:00.794 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:00.793+0000 7f0ad54d3700 1 --2- 172.21.15.160:0/4164841958 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0ad00f47c0 0x7f0ad00f4be0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f0ac0009fb0 tx=0x7f0ac0009b30).stop 2023-11-17T05:39:00.794 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:00.793+0000 7f0ad54d3700 1 -- 172.21.15.160:0/4164841958 shutdown_connections 2023-11-17T05:39:00.795 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:00.793+0000 7f0ad54d3700 1 --2- 172.21.15.160:0/4164841958 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0ad00f47c0 0x7f0ad00f4be0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:00.795 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:00.793+0000 7f0ad54d3700 1 -- 172.21.15.160:0/4164841958 >> 172.21.15.160:0/4164841958 conn(0x7f0ad00f02e0 msgr2=0x7f0ad00f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:39:00.795 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:00.793+0000 7f0ad54d3700 1 -- 172.21.15.160:0/4164841958 shutdown_connections 2023-11-17T05:39:00.795 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:00.794+0000 7f0ad54d3700 1 -- 172.21.15.160:0/4164841958 wait complete. 2023-11-17T05:39:00.796 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:00.794+0000 7f0ad54d3700 1 Processor -- start 2023-11-17T05:39:00.796 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:00.794+0000 7f0ad54d3700 1 -- start start 2023-11-17T05:39:00.796 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:00.794+0000 7f0ad54d3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0ad00f47c0 0x7f0ad0106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:39:00.796 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:00.794+0000 7f0ad54d3700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad0106bf0 con 0x7f0ad00f47c0 2023-11-17T05:39:00.796 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:00.795+0000 7f0aceffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0ad00f47c0 0x7f0ad0106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:39:00.796 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:00.795+0000 7f0aceffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0ad00f47c0 0x7f0ad0106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:36730/0 (socket says 172.21.15.160:36730) 2023-11-17T05:39:00.797 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:00.795+0000 7f0aceffd700 1 -- 172.21.15.160:0/1119140635 learned_addr learned my addr 172.21.15.160:0/1119140635 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:39:00.797 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:00.795+0000 7f0aceffd700 1 -- 172.21.15.160:0/1119140635 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ac0005040 con 0x7f0ad00f47c0 2023-11-17T05:39:00.797 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:00.795+0000 7f0aceffd700 1 --2- 172.21.15.160:0/1119140635 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0ad00f47c0 0x7f0ad0106700 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f0ac0014000 tx=0x7f0ac000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:39:00.797 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:00.795+0000 7f0ab77fe700 1 -- 172.21.15.160:0/1119140635 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ac00095f0 con 0x7f0ad00f47c0 2023-11-17T05:39:00.797 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:00.795+0000 7f0ad54d3700 1 -- 172.21.15.160:0/1119140635 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ad0106df0 con 0x7f0ad00f47c0 2023-11-17T05:39:00.797 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:00.795+0000 7f0ab77fe700 1 -- 172.21.15.160:0/1119140635 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0ac0007e70 con 0x7f0ad00f47c0 2023-11-17T05:39:00.797 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:00.795+0000 7f0ab77fe700 1 -- 172.21.15.160:0/1119140635 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ac0014ea0 con 0x7f0ad00f47c0 2023-11-17T05:39:00.797 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:00.795+0000 7f0ad54d3700 1 -- 172.21.15.160:0/1119140635 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ad0107280 con 0x7f0ad00f47c0 2023-11-17T05:39:00.798 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:00.796+0000 7f0ab77fe700 1 -- 172.21.15.160:0/1119140635 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0ac001d4e0 con 0x7f0ad00f47c0 2023-11-17T05:39:00.798 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:00.796+0000 7f0ad54d3700 1 -- 172.21.15.160:0/1119140635 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ad0045ad0 con 0x7f0ad00f47c0 2023-11-17T05:39:00.798 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:00.796+0000 7f0ab77fe700 1 --2- 172.21.15.160:0/1119140635 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f0ab802ff60 0x7f0ab8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:39:00.798 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:00.796+0000 7f0ab77fe700 1 -- 172.21.15.160:0/1119140635 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0ac0040bc0 con 0x7f0ad00f47c0 2023-11-17T05:39:00.798 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:00.797+0000 7f0ace7fc700 1 --2- 172.21.15.160:0/1119140635 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f0ab802ff60 0x7f0ab8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:39:00.798 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:00.797+0000 7f0ace7fc700 1 --2- 172.21.15.160:0/1119140635 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f0ab802ff60 0x7f0ab8032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f0ac40096a0 tx=0x7f0ac4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:39:00.802 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:00.802+0000 7f0ab77fe700 1 -- 172.21.15.160:0/1119140635 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0ac000f8f0 con 0x7f0ad00f47c0 2023-11-17T05:39:01.008 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:01.007+0000 7f0ad54d3700 1 -- 172.21.15.160:0/1119140635 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0ad0108130 con 0x7f0ad00f47c0 2023-11-17T05:39:01.008 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:01.008+0000 7f0ab77fe700 1 -- 172.21.15.160:0/1119140635 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f0ac0012330 con 0x7f0ad00f47c0 2023-11-17T05:39:01.008 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:39:01.009 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:39:01.010 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:01.009+0000 7f0ad54d3700 1 -- 172.21.15.160:0/1119140635 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f0ab802ff60 msgr2=0x7f0ab8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:01.010 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:01.009+0000 7f0ad54d3700 1 --2- 172.21.15.160:0/1119140635 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f0ab802ff60 0x7f0ab8032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f0ac40096a0 tx=0x7f0ac4006b40).stop 2023-11-17T05:39:01.011 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:01.009+0000 7f0ad54d3700 1 -- 172.21.15.160:0/1119140635 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0ad00f47c0 msgr2=0x7f0ad0106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:01.011 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:01.009+0000 7f0ad54d3700 1 --2- 172.21.15.160:0/1119140635 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0ad00f47c0 0x7f0ad0106700 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f0ac0014000 tx=0x7f0ac000b040).stop 2023-11-17T05:39:01.011 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:01.009+0000 7f0ad54d3700 1 -- 172.21.15.160:0/1119140635 shutdown_connections 2023-11-17T05:39:01.011 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:01.009+0000 7f0ad54d3700 1 --2- 172.21.15.160:0/1119140635 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f0ab802ff60 0x7f0ab8032420 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:01.011 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:01.009+0000 7f0ad54d3700 1 --2- 172.21.15.160:0/1119140635 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0ad00f47c0 0x7f0ad0106700 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:01.011 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:01.009+0000 7f0ad54d3700 1 -- 172.21.15.160:0/1119140635 >> 172.21.15.160:0/1119140635 conn(0x7f0ad00f02e0 msgr2=0x7f0ad00f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:39:01.011 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:01.009+0000 7f0ad54d3700 1 -- 172.21.15.160:0/1119140635 shutdown_connections 2023-11-17T05:39:01.011 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:01.009+0000 7f0ad54d3700 1 -- 172.21.15.160:0/1119140635 wait complete. 2023-11-17T05:39:01.012 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:39:01.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:01 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:39:00.390673+0000 mgr.smithi003.eddcuc (mgr.14164) 417 : cluster [DBG] pgmap v401: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:39:01.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:01 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:39:01.008710+0000 mon.smithi003 (mon.0) 328 : audit [DBG] from='client.? 172.21.15.160:0/1119140635' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:39:02.388 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:39:02.389 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:39:03.634 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:03.632+0000 7fc6dcc4a700 1 -- 172.21.15.160:0/2653649373 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6d8060220 msgr2=0x7fc6d80f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:03.634 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:03.632+0000 7fc6dcc4a700 1 --2- 172.21.15.160:0/2653649373 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6d8060220 0x7fc6d80f9150 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7fc6c8009fb0 tx=0x7fc6c8009b30).stop 2023-11-17T05:39:03.634 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:03.632+0000 7fc6dcc4a700 1 -- 172.21.15.160:0/2653649373 shutdown_connections 2023-11-17T05:39:03.634 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:03.632+0000 7fc6dcc4a700 1 --2- 172.21.15.160:0/2653649373 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6d8060220 0x7fc6d80f9150 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:03.634 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:03.632+0000 7fc6dcc4a700 1 -- 172.21.15.160:0/2653649373 >> 172.21.15.160:0/2653649373 conn(0x7fc6d80f0b70 msgr2=0x7fc6d80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:39:03.634 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:03.632+0000 7fc6dcc4a700 1 -- 172.21.15.160:0/2653649373 shutdown_connections 2023-11-17T05:39:03.634 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:03.632+0000 7fc6dcc4a700 1 -- 172.21.15.160:0/2653649373 wait complete. 2023-11-17T05:39:03.634 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:03.632+0000 7fc6dcc4a700 1 Processor -- start 2023-11-17T05:39:03.634 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:03.632+0000 7fc6dcc4a700 1 -- start start 2023-11-17T05:39:03.635 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:03.633+0000 7fc6dcc4a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6d8060220 0x7fc6d81066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:39:03.635 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:03.633+0000 7fc6dcc4a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d8106bd0 con 0x7fc6d8060220 2023-11-17T05:39:03.635 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:03.633+0000 7fc6d659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6d8060220 0x7fc6d81066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:39:03.635 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:03.633+0000 7fc6d659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6d8060220 0x7fc6d81066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:36734/0 (socket says 172.21.15.160:36734) 2023-11-17T05:39:03.635 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:03.633+0000 7fc6d659c700 1 -- 172.21.15.160:0/2505371169 learned_addr learned my addr 172.21.15.160:0/2505371169 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:39:03.635 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:03.633+0000 7fc6d659c700 1 -- 172.21.15.160:0/2505371169 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6c8005040 con 0x7fc6d8060220 2023-11-17T05:39:03.636 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:03.633+0000 7fc6d659c700 1 --2- 172.21.15.160:0/2505371169 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6d8060220 0x7fc6d81066e0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7fc6c8013510 tx=0x7fc6c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:39:03.636 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:03.633+0000 7fc6c6ffd700 1 -- 172.21.15.160:0/2505371169 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc6c8009320 con 0x7fc6d8060220 2023-11-17T05:39:03.636 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:03.634+0000 7fc6dcc4a700 1 -- 172.21.15.160:0/2505371169 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6d8106dd0 con 0x7fc6d8060220 2023-11-17T05:39:03.636 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:03.634+0000 7fc6c6ffd700 1 -- 172.21.15.160:0/2505371169 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc6c8013f00 con 0x7fc6d8060220 2023-11-17T05:39:03.636 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:03.634+0000 7fc6c6ffd700 1 -- 172.21.15.160:0/2505371169 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc6c8014dc0 con 0x7fc6d8060220 2023-11-17T05:39:03.637 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:03.634+0000 7fc6dcc4a700 1 -- 172.21.15.160:0/2505371169 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6d8107260 con 0x7fc6d8060220 2023-11-17T05:39:03.637 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:03.634+0000 7fc6c6ffd700 1 -- 172.21.15.160:0/2505371169 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc6c80095f0 con 0x7fc6d8060220 2023-11-17T05:39:03.637 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:03.634+0000 7fc6dcc4a700 1 -- 172.21.15.160:0/2505371169 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6d8045ad0 con 0x7fc6d8060220 2023-11-17T05:39:03.637 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:03.634+0000 7fc6c6ffd700 1 --2- 172.21.15.160:0/2505371169 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc6c002ff60 0x7fc6c0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:39:03.637 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:03.635+0000 7fc6c6ffd700 1 -- 172.21.15.160:0/2505371169 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc6c80410e0 con 0x7fc6d8060220 2023-11-17T05:39:03.637 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:03.635+0000 7fc6d5d9b700 1 --2- 172.21.15.160:0/2505371169 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc6c002ff60 0x7fc6c0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:39:03.637 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:03.635+0000 7fc6d5d9b700 1 --2- 172.21.15.160:0/2505371169 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc6c002ff60 0x7fc6c0032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fc6cc0096a0 tx=0x7fc6cc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:39:03.640 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:03.639+0000 7fc6c6ffd700 1 -- 172.21.15.160:0/2505371169 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc6c8014420 con 0x7fc6d8060220 2023-11-17T05:39:03.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:03 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:39:02.390969+0000 mgr.smithi003.eddcuc (mgr.14164) 418 : cluster [DBG] pgmap v402: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:39:03.844 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:03.844+0000 7fc6dcc4a700 1 -- 172.21.15.160:0/2505371169 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc6d81080b0 con 0x7fc6d8060220 2023-11-17T05:39:03.845 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:03.844+0000 7fc6c6ffd700 1 -- 172.21.15.160:0/2505371169 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fc6c801c920 con 0x7fc6d8060220 2023-11-17T05:39:03.845 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:39:03.845 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:39:03.847 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:03.846+0000 7fc6dcc4a700 1 -- 172.21.15.160:0/2505371169 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc6c002ff60 msgr2=0x7fc6c0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:03.847 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:03.846+0000 7fc6dcc4a700 1 --2- 172.21.15.160:0/2505371169 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc6c002ff60 0x7fc6c0032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fc6cc0096a0 tx=0x7fc6cc006b40).stop 2023-11-17T05:39:03.847 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:03.846+0000 7fc6dcc4a700 1 -- 172.21.15.160:0/2505371169 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6d8060220 msgr2=0x7fc6d81066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:03.847 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:03.846+0000 7fc6dcc4a700 1 --2- 172.21.15.160:0/2505371169 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6d8060220 0x7fc6d81066e0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7fc6c8013510 tx=0x7fc6c800b040).stop 2023-11-17T05:39:03.847 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:03.846+0000 7fc6dcc4a700 1 -- 172.21.15.160:0/2505371169 shutdown_connections 2023-11-17T05:39:03.847 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:03.846+0000 7fc6dcc4a700 1 --2- 172.21.15.160:0/2505371169 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fc6c002ff60 0x7fc6c0032420 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:03.847 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:03.846+0000 7fc6dcc4a700 1 --2- 172.21.15.160:0/2505371169 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc6d8060220 0x7fc6d81066e0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:03.847 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:03.846+0000 7fc6dcc4a700 1 -- 172.21.15.160:0/2505371169 >> 172.21.15.160:0/2505371169 conn(0x7fc6d80f0b70 msgr2=0x7fc6d80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:39:03.847 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:03.846+0000 7fc6dcc4a700 1 -- 172.21.15.160:0/2505371169 shutdown_connections 2023-11-17T05:39:03.847 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:03.846+0000 7fc6dcc4a700 1 -- 172.21.15.160:0/2505371169 wait complete. 2023-11-17T05:39:03.848 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:39:04.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:04 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:39:03.845586+0000 mon.smithi003 (mon.0) 329 : audit [DBG] from='client.? 172.21.15.160:0/2505371169' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:39:05.247 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:39:05.247 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:39:05.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:05 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:39:04.391323+0000 mgr.smithi003.eddcuc (mgr.14164) 419 : cluster [DBG] pgmap v403: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:39:06.471 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:06.470+0000 7f543dd72700 1 -- 172.21.15.160:0/2313175345 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54380f5410 msgr2=0x7f54380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:06.472 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:06.470+0000 7f543dd72700 1 --2- 172.21.15.160:0/2313175345 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54380f5410 0x7f54380f5830 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f5428009fb0 tx=0x7f5428009b30).stop 2023-11-17T05:39:06.472 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:06.470+0000 7f543dd72700 1 -- 172.21.15.160:0/2313175345 shutdown_connections 2023-11-17T05:39:06.472 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:06.470+0000 7f543dd72700 1 --2- 172.21.15.160:0/2313175345 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54380f5410 0x7f54380f5830 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:06.472 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:06.471+0000 7f543dd72700 1 -- 172.21.15.160:0/2313175345 >> 172.21.15.160:0/2313175345 conn(0x7f54380f0b70 msgr2=0x7f54380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:39:06.472 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:06.471+0000 7f543dd72700 1 -- 172.21.15.160:0/2313175345 shutdown_connections 2023-11-17T05:39:06.472 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:06.471+0000 7f543dd72700 1 -- 172.21.15.160:0/2313175345 wait complete. 2023-11-17T05:39:06.474 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:06.471+0000 7f543dd72700 1 Processor -- start 2023-11-17T05:39:06.475 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:06.471+0000 7f543dd72700 1 -- start start 2023-11-17T05:39:06.475 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:06.471+0000 7f543dd72700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54380f5410 0x7f5438109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:39:06.475 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:06.471+0000 7f543dd72700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5438060220 con 0x7f54380f5410 2023-11-17T05:39:06.475 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:06.472+0000 7f54377fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54380f5410 0x7f5438109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:39:06.475 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:06.472+0000 7f54377fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54380f5410 0x7f5438109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:36752/0 (socket says 172.21.15.160:36752) 2023-11-17T05:39:06.475 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:06.472+0000 7f54377fe700 1 -- 172.21.15.160:0/1257386415 learned_addr learned my addr 172.21.15.160:0/1257386415 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:39:06.475 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:06.472+0000 7f54377fe700 1 -- 172.21.15.160:0/1257386415 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5428005040 con 0x7f54380f5410 2023-11-17T05:39:06.475 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:06.472+0000 7f54377fe700 1 --2- 172.21.15.160:0/1257386415 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54380f5410 0x7f5438109450 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f5428013510 tx=0x7f542800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:39:06.476 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:06.472+0000 7f541ffff700 1 -- 172.21.15.160:0/1257386415 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5428009320 con 0x7f54380f5410 2023-11-17T05:39:06.476 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:06.472+0000 7f543dd72700 1 -- 172.21.15.160:0/1257386415 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5438060420 con 0x7f54380f5410 2023-11-17T05:39:06.476 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:06.472+0000 7f543dd72700 1 -- 172.21.15.160:0/1257386415 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5438109b00 con 0x7f54380f5410 2023-11-17T05:39:06.476 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:06.472+0000 7f541ffff700 1 -- 172.21.15.160:0/1257386415 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5428013f00 con 0x7f54380f5410 2023-11-17T05:39:06.476 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:06.472+0000 7f541ffff700 1 -- 172.21.15.160:0/1257386415 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5428014cc0 con 0x7f54380f5410 2023-11-17T05:39:06.476 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:06.473+0000 7f541ffff700 1 -- 172.21.15.160:0/1257386415 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5428014420 con 0x7f54380f5410 2023-11-17T05:39:06.476 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:06.473+0000 7f543dd72700 1 -- 172.21.15.160:0/1257386415 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5418004fa0 con 0x7f54380f5410 2023-11-17T05:39:06.476 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:06.473+0000 7f541ffff700 1 --2- 172.21.15.160:0/1257386415 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f542002ff60 0x7f5420032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:39:06.476 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:06.473+0000 7f541ffff700 1 -- 172.21.15.160:0/1257386415 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f54280411a0 con 0x7f54380f5410 2023-11-17T05:39:06.476 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:06.474+0000 7f5436ffd700 1 --2- 172.21.15.160:0/1257386415 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f542002ff60 0x7f5420032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:39:06.476 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:06.474+0000 7f5436ffd700 1 --2- 172.21.15.160:0/1257386415 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f542002ff60 0x7f5420032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f542c0096a0 tx=0x7f542c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:39:06.479 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:06.478+0000 7f541ffff700 1 -- 172.21.15.160:0/1257386415 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5428012b60 con 0x7f54380f5410 2023-11-17T05:39:06.683 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:06.683+0000 7f543dd72700 1 -- 172.21.15.160:0/1257386415 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5418005670 con 0x7f54380f5410 2023-11-17T05:39:06.684 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:06.683+0000 7f541ffff700 1 -- 172.21.15.160:0/1257386415 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f5428012980 con 0x7f54380f5410 2023-11-17T05:39:06.684 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:39:06.684 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:39:06.686 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:06.685+0000 7f543dd72700 1 -- 172.21.15.160:0/1257386415 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f542002ff60 msgr2=0x7f5420032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:06.686 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:06.685+0000 7f543dd72700 1 --2- 172.21.15.160:0/1257386415 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f542002ff60 0x7f5420032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f542c0096a0 tx=0x7f542c006b40).stop 2023-11-17T05:39:06.686 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:06.685+0000 7f543dd72700 1 -- 172.21.15.160:0/1257386415 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54380f5410 msgr2=0x7f5438109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:06.686 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:06.685+0000 7f543dd72700 1 --2- 172.21.15.160:0/1257386415 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54380f5410 0x7f5438109450 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f5428013510 tx=0x7f542800b040).stop 2023-11-17T05:39:06.686 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:06.685+0000 7f543dd72700 1 -- 172.21.15.160:0/1257386415 shutdown_connections 2023-11-17T05:39:06.687 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:06.685+0000 7f543dd72700 1 --2- 172.21.15.160:0/1257386415 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f542002ff60 0x7f5420032420 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:06.687 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:06.685+0000 7f543dd72700 1 --2- 172.21.15.160:0/1257386415 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54380f5410 0x7f5438109450 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:06.687 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:06.685+0000 7f543dd72700 1 -- 172.21.15.160:0/1257386415 >> 172.21.15.160:0/1257386415 conn(0x7f54380f0b70 msgr2=0x7f54380f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:39:06.687 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:06.685+0000 7f543dd72700 1 -- 172.21.15.160:0/1257386415 shutdown_connections 2023-11-17T05:39:06.687 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:06.685+0000 7f543dd72700 1 -- 172.21.15.160:0/1257386415 wait complete. 2023-11-17T05:39:06.687 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:39:07.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:07 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:39:06.391644+0000 mgr.smithi003.eddcuc (mgr.14164) 420 : cluster [DBG] pgmap v404: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:39:07.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:07 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:39:06.684434+0000 mon.smithi003 (mon.0) 330 : audit [DBG] from='client.? 172.21.15.160:0/1257386415' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:39:08.073 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:39:08.074 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:39:09.294 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:09.292+0000 7f244e059700 1 -- 172.21.15.160:0/1573663721 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24480601e0 msgr2=0x7f24480f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:09.294 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:09.292+0000 7f244e059700 1 --2- 172.21.15.160:0/1573663721 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24480601e0 0x7f24480f8f20 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f243c009fb0 tx=0x7f243c009b30).stop 2023-11-17T05:39:09.294 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:09.292+0000 7f244e059700 1 -- 172.21.15.160:0/1573663721 shutdown_connections 2023-11-17T05:39:09.294 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:09.292+0000 7f244e059700 1 --2- 172.21.15.160:0/1573663721 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24480601e0 0x7f24480f8f20 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:09.294 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:09.292+0000 7f244e059700 1 -- 172.21.15.160:0/1573663721 >> 172.21.15.160:0/1573663721 conn(0x7f24480f09a0 msgr2=0x7f24480f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:39:09.294 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:09.292+0000 7f244e059700 1 -- 172.21.15.160:0/1573663721 shutdown_connections 2023-11-17T05:39:09.294 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:09.292+0000 7f244e059700 1 -- 172.21.15.160:0/1573663721 wait complete. 2023-11-17T05:39:09.295 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:09.292+0000 7f244e059700 1 Processor -- start 2023-11-17T05:39:09.295 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:09.292+0000 7f244e059700 1 -- start start 2023-11-17T05:39:09.295 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:09.292+0000 7f244e059700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24480601e0 0x7f24480f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:39:09.295 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:09.292+0000 7f244e059700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24480f4e20 con 0x7f24480601e0 2023-11-17T05:39:09.295 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:09.293+0000 7f244d057700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24480601e0 0x7f24480f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:39:09.295 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:09.293+0000 7f244d057700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24480601e0 0x7f24480f69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:36772/0 (socket says 172.21.15.160:36772) 2023-11-17T05:39:09.295 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:09.293+0000 7f244d057700 1 -- 172.21.15.160:0/3529609765 learned_addr learned my addr 172.21.15.160:0/3529609765 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:39:09.295 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:09.293+0000 7f244d057700 1 -- 172.21.15.160:0/3529609765 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f243c005040 con 0x7f24480601e0 2023-11-17T05:39:09.295 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:09.293+0000 7f244d057700 1 --2- 172.21.15.160:0/3529609765 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24480601e0 0x7f24480f69a0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f243c013510 tx=0x7f243c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:39:09.295 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:09.293+0000 7f2439ffb700 1 -- 172.21.15.160:0/3529609765 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f243c009320 con 0x7f24480601e0 2023-11-17T05:39:09.295 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:09.293+0000 7f244e059700 1 -- 172.21.15.160:0/3529609765 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24480f4fc0 con 0x7f24480601e0 2023-11-17T05:39:09.295 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:09.293+0000 7f2439ffb700 1 -- 172.21.15.160:0/3529609765 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f243c013f00 con 0x7f24480601e0 2023-11-17T05:39:09.295 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:09.293+0000 7f2439ffb700 1 -- 172.21.15.160:0/3529609765 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f243c014d90 con 0x7f24480601e0 2023-11-17T05:39:09.296 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:09.293+0000 7f244e059700 1 -- 172.21.15.160:0/3529609765 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24480f5450 con 0x7f24480601e0 2023-11-17T05:39:09.296 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:09.294+0000 7f2439ffb700 1 -- 172.21.15.160:0/3529609765 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f243c0095f0 con 0x7f24480601e0 2023-11-17T05:39:09.296 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:09.294+0000 7f244e059700 1 -- 172.21.15.160:0/3529609765 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2448059100 con 0x7f24480601e0 2023-11-17T05:39:09.297 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:09.294+0000 7f2439ffb700 1 --2- 172.21.15.160:0/3529609765 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f243402ff60 0x7f2434032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:39:09.297 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:09.294+0000 7f2439ffb700 1 -- 172.21.15.160:0/3529609765 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f243c041090 con 0x7f24480601e0 2023-11-17T05:39:09.297 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:09.295+0000 7f244c856700 1 --2- 172.21.15.160:0/3529609765 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f243402ff60 0x7f2434032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:39:09.297 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:09.295+0000 7f244c856700 1 --2- 172.21.15.160:0/3529609765 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f243402ff60 0x7f2434032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f24440096a0 tx=0x7f2444006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:39:09.300 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:09.299+0000 7f2439ffb700 1 -- 172.21.15.160:0/3529609765 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f243c014420 con 0x7f24480601e0 2023-11-17T05:39:09.505 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:09.504+0000 7f244e059700 1 -- 172.21.15.160:0/3529609765 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f24480fbcd0 con 0x7f24480601e0 2023-11-17T05:39:09.505 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:09.506+0000 7f2439ffb700 1 -- 172.21.15.160:0/3529609765 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f243c01d7f0 con 0x7f24480601e0 2023-11-17T05:39:09.506 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:39:09.506 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:39:09.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:09.506+0000 7f244e059700 1 -- 172.21.15.160:0/3529609765 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f243402ff60 msgr2=0x7f2434032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:09.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:09.506+0000 7f244e059700 1 --2- 172.21.15.160:0/3529609765 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f243402ff60 0x7f2434032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f24440096a0 tx=0x7f2444006b40).stop 2023-11-17T05:39:09.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:09.506+0000 7f244e059700 1 -- 172.21.15.160:0/3529609765 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24480601e0 msgr2=0x7f24480f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:09.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:09.506+0000 7f244e059700 1 --2- 172.21.15.160:0/3529609765 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24480601e0 0x7f24480f69a0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f243c013510 tx=0x7f243c00b040).stop 2023-11-17T05:39:09.508 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:09.506+0000 7f244e059700 1 -- 172.21.15.160:0/3529609765 shutdown_connections 2023-11-17T05:39:09.508 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:09.506+0000 7f244e059700 1 --2- 172.21.15.160:0/3529609765 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f243402ff60 0x7f2434032420 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:09.508 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:09.506+0000 7f244e059700 1 --2- 172.21.15.160:0/3529609765 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24480601e0 0x7f24480f69a0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:09.508 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:09.506+0000 7f244e059700 1 -- 172.21.15.160:0/3529609765 >> 172.21.15.160:0/3529609765 conn(0x7f24480f09a0 msgr2=0x7f24480f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:39:09.508 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:09.507+0000 7f244e059700 1 -- 172.21.15.160:0/3529609765 shutdown_connections 2023-11-17T05:39:09.508 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:09.507+0000 7f244e059700 1 -- 172.21.15.160:0/3529609765 wait complete. 2023-11-17T05:39:09.508 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:39:09.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:09 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:39:08.391910+0000 mgr.smithi003.eddcuc (mgr.14164) 421 : cluster [DBG] pgmap v405: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:39:10.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:10 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:39:09.505957+0000 mon.smithi003 (mon.0) 331 : audit [DBG] from='client.? 172.21.15.160:0/3529609765' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:39:10.918 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:39:10.918 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:39:11.771 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:11 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:39:10.392181+0000 mgr.smithi003.eddcuc (mgr.14164) 422 : cluster [DBG] pgmap v406: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:39:12.078 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:12.077+0000 7fe84952e700 1 -- 172.21.15.160:0/3994261914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8440f47c0 msgr2=0x7fe8440f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:12.078 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:12.077+0000 7fe84952e700 1 --2- 172.21.15.160:0/3994261914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8440f47c0 0x7fe8440f4be0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7fe838009fb0 tx=0x7fe838009b30).stop 2023-11-17T05:39:12.079 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:12.077+0000 7fe84952e700 1 -- 172.21.15.160:0/3994261914 shutdown_connections 2023-11-17T05:39:12.079 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:12.077+0000 7fe84952e700 1 --2- 172.21.15.160:0/3994261914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8440f47c0 0x7fe8440f4be0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:12.079 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:12.077+0000 7fe84952e700 1 -- 172.21.15.160:0/3994261914 >> 172.21.15.160:0/3994261914 conn(0x7fe8440f02e0 msgr2=0x7fe8440f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:39:12.079 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:12.077+0000 7fe84952e700 1 -- 172.21.15.160:0/3994261914 shutdown_connections 2023-11-17T05:39:12.079 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:12.077+0000 7fe84952e700 1 -- 172.21.15.160:0/3994261914 wait complete. 2023-11-17T05:39:12.080 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:12.078+0000 7fe84952e700 1 Processor -- start 2023-11-17T05:39:12.080 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:12.078+0000 7fe84952e700 1 -- start start 2023-11-17T05:39:12.080 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:12.078+0000 7fe84952e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8440f47c0 0x7fe8440f7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:39:12.081 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:12.078+0000 7fe84952e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8440f57f0 con 0x7fe8440f47c0 2023-11-17T05:39:12.081 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:12.078+0000 7fe842ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8440f47c0 0x7fe8440f7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:39:12.081 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:12.078+0000 7fe842ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8440f47c0 0x7fe8440f7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:60994/0 (socket says 172.21.15.160:60994) 2023-11-17T05:39:12.081 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:12.078+0000 7fe842ffd700 1 -- 172.21.15.160:0/3354334738 learned_addr learned my addr 172.21.15.160:0/3354334738 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:39:12.081 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:12.079+0000 7fe842ffd700 1 -- 172.21.15.160:0/3354334738 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe838005040 con 0x7fe8440f47c0 2023-11-17T05:39:12.081 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:12.079+0000 7fe842ffd700 1 --2- 172.21.15.160:0/3354334738 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8440f47c0 0x7fe8440f7370 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7fe838013510 tx=0x7fe83800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:39:12.081 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:12.079+0000 7fe82b7fe700 1 -- 172.21.15.160:0/3354334738 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe838009320 con 0x7fe8440f47c0 2023-11-17T05:39:12.081 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:12.079+0000 7fe82b7fe700 1 -- 172.21.15.160:0/3354334738 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe838013f00 con 0x7fe8440f47c0 2023-11-17T05:39:12.081 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:12.079+0000 7fe84952e700 1 -- 172.21.15.160:0/3354334738 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8440f5990 con 0x7fe8440f47c0 2023-11-17T05:39:12.081 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:12.079+0000 7fe82b7fe700 1 -- 172.21.15.160:0/3354334738 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe838014d90 con 0x7fe8440f47c0 2023-11-17T05:39:12.081 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:12.079+0000 7fe84952e700 1 -- 172.21.15.160:0/3354334738 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8440f5e20 con 0x7fe8440f47c0 2023-11-17T05:39:12.082 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:12.080+0000 7fe82b7fe700 1 -- 172.21.15.160:0/3354334738 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe8380263a0 con 0x7fe8440f47c0 2023-11-17T05:39:12.082 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:12.080+0000 7fe82b7fe700 1 --2- 172.21.15.160:0/3354334738 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe82c02ff60 0x7fe82c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:39:12.082 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:12.080+0000 7fe82b7fe700 1 -- 172.21.15.160:0/3354334738 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe838040ff0 con 0x7fe8440f47c0 2023-11-17T05:39:12.082 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:12.080+0000 7fe84952e700 1 -- 172.21.15.160:0/3354334738 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe844059100 con 0x7fe8440f47c0 2023-11-17T05:39:12.082 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:12.080+0000 7fe8427fc700 1 --2- 172.21.15.160:0/3354334738 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe82c02ff60 0x7fe82c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:39:12.083 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:12.081+0000 7fe8427fc700 1 --2- 172.21.15.160:0/3354334738 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe82c02ff60 0x7fe82c032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fe8340096a0 tx=0x7fe834006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:39:12.086 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:12.085+0000 7fe82b7fe700 1 -- 172.21.15.160:0/3354334738 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe838012920 con 0x7fe8440f47c0 2023-11-17T05:39:12.292 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:12.291+0000 7fe84952e700 1 -- 172.21.15.160:0/3354334738 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe8440f6520 con 0x7fe8440f47c0 2023-11-17T05:39:12.293 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:12.292+0000 7fe82b7fe700 1 -- 172.21.15.160:0/3354334738 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fe83802b020 con 0x7fe8440f47c0 2023-11-17T05:39:12.293 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:39:12.293 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:39:12.295 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:12.294+0000 7fe84952e700 1 -- 172.21.15.160:0/3354334738 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe82c02ff60 msgr2=0x7fe82c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:12.295 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:12.294+0000 7fe84952e700 1 --2- 172.21.15.160:0/3354334738 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe82c02ff60 0x7fe82c032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fe8340096a0 tx=0x7fe834006b40).stop 2023-11-17T05:39:12.295 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:12.294+0000 7fe84952e700 1 -- 172.21.15.160:0/3354334738 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8440f47c0 msgr2=0x7fe8440f7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:12.295 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:12.294+0000 7fe84952e700 1 --2- 172.21.15.160:0/3354334738 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8440f47c0 0x7fe8440f7370 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7fe838013510 tx=0x7fe83800b040).stop 2023-11-17T05:39:12.295 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:12.294+0000 7fe84952e700 1 -- 172.21.15.160:0/3354334738 shutdown_connections 2023-11-17T05:39:12.295 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:12.294+0000 7fe84952e700 1 --2- 172.21.15.160:0/3354334738 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe82c02ff60 0x7fe82c032420 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:12.295 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:12.294+0000 7fe84952e700 1 --2- 172.21.15.160:0/3354334738 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe8440f47c0 0x7fe8440f7370 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:12.295 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:12.294+0000 7fe84952e700 1 -- 172.21.15.160:0/3354334738 >> 172.21.15.160:0/3354334738 conn(0x7fe8440f02e0 msgr2=0x7fe8440f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:39:12.296 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:12.294+0000 7fe84952e700 1 -- 172.21.15.160:0/3354334738 shutdown_connections 2023-11-17T05:39:12.296 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:12.294+0000 7fe84952e700 1 -- 172.21.15.160:0/3354334738 wait complete. 2023-11-17T05:39:12.296 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:39:12.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:12 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:39:12.293343+0000 mon.smithi003 (mon.0) 332 : audit [DBG] from='client.? 172.21.15.160:0/3354334738' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:39:13.680 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:39:13.680 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:39:13.771 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:13 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:39:12.392475+0000 mgr.smithi003.eddcuc (mgr.14164) 423 : cluster [DBG] pgmap v407: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:39:14.885 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:14.884+0000 7f03008e1700 1 -- 172.21.15.160:0/2005079880 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02fc0f62a0 msgr2=0x7f02fc0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:14.886 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:14.884+0000 7f03008e1700 1 --2- 172.21.15.160:0/2005079880 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02fc0f62a0 0x7f02fc0f66c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f02f0009fb0 tx=0x7f02f0009b30).stop 2023-11-17T05:39:14.886 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:14.884+0000 7f03008e1700 1 -- 172.21.15.160:0/2005079880 shutdown_connections 2023-11-17T05:39:14.886 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:14.884+0000 7f03008e1700 1 --2- 172.21.15.160:0/2005079880 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02fc0f62a0 0x7f02fc0f66c0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:14.886 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:14.884+0000 7f03008e1700 1 -- 172.21.15.160:0/2005079880 >> 172.21.15.160:0/2005079880 conn(0x7f02fc0f1aa0 msgr2=0x7f02fc0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:39:14.886 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:14.884+0000 7f03008e1700 1 -- 172.21.15.160:0/2005079880 shutdown_connections 2023-11-17T05:39:14.886 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:14.884+0000 7f03008e1700 1 -- 172.21.15.160:0/2005079880 wait complete. 2023-11-17T05:39:14.886 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:14.884+0000 7f03008e1700 1 Processor -- start 2023-11-17T05:39:14.886 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:14.885+0000 7f03008e1700 1 -- start start 2023-11-17T05:39:14.886 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:14.885+0000 7f03008e1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02fc0f62a0 0x7f02fc1086b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:39:14.886 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:14.885+0000 7f03008e1700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02fc108ba0 con 0x7f02fc0f62a0 2023-11-17T05:39:14.888 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:14.885+0000 7f02fad9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02fc0f62a0 0x7f02fc1086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:39:14.888 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:14.885+0000 7f02fad9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02fc0f62a0 0x7f02fc1086b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:32782/0 (socket says 172.21.15.160:32782) 2023-11-17T05:39:14.888 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:14.885+0000 7f02fad9d700 1 -- 172.21.15.160:0/521769634 learned_addr learned my addr 172.21.15.160:0/521769634 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:39:14.888 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:14.885+0000 7f02fad9d700 1 -- 172.21.15.160:0/521769634 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f0005040 con 0x7f02fc0f62a0 2023-11-17T05:39:14.888 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:14.885+0000 7f02fad9d700 1 --2- 172.21.15.160:0/521769634 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02fc0f62a0 0x7f02fc1086b0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f02f0009830 tx=0x7f02f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:39:14.888 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:14.886+0000 7f02e37fe700 1 -- 172.21.15.160:0/521769634 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02f0009320 con 0x7f02fc0f62a0 2023-11-17T05:39:14.889 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:14.886+0000 7f02e37fe700 1 -- 172.21.15.160:0/521769634 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f02f00136c0 con 0x7f02fc0f62a0 2023-11-17T05:39:14.889 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:14.886+0000 7f03008e1700 1 -- 172.21.15.160:0/521769634 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02fc108da0 con 0x7f02fc0f62a0 2023-11-17T05:39:14.891 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:14.886+0000 7f02e37fe700 1 -- 172.21.15.160:0/521769634 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02f001ee90 con 0x7f02fc0f62a0 2023-11-17T05:39:14.891 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:14.886+0000 7f03008e1700 1 -- 172.21.15.160:0/521769634 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02fc109230 con 0x7f02fc0f62a0 2023-11-17T05:39:14.891 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:14.887+0000 7f02e37fe700 1 -- 172.21.15.160:0/521769634 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f02f0014440 con 0x7f02fc0f62a0 2023-11-17T05:39:14.891 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:14.887+0000 7f03008e1700 1 -- 172.21.15.160:0/521769634 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02fc045ad0 con 0x7f02fc0f62a0 2023-11-17T05:39:14.891 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:14.887+0000 7f02e37fe700 1 --2- 172.21.15.160:0/521769634 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f02e402ff10 0x7f02e40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:39:14.891 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:14.887+0000 7f02e37fe700 1 -- 172.21.15.160:0/521769634 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f02f0040fb0 con 0x7f02fc0f62a0 2023-11-17T05:39:14.891 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:14.887+0000 7f02fa59c700 1 --2- 172.21.15.160:0/521769634 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f02e402ff10 0x7f02e40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:39:14.891 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:14.887+0000 7f02fa59c700 1 --2- 172.21.15.160:0/521769634 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f02e402ff10 0x7f02e40323d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f02ec0096a0 tx=0x7f02ec006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:39:14.892 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:14.892+0000 7f02e37fe700 1 -- 172.21.15.160:0/521769634 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f02f006d2b0 con 0x7f02fc0f62a0 2023-11-17T05:39:15.094 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:15.093+0000 7f03008e1700 1 -- 172.21.15.160:0/521769634 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f02fc10a0e0 con 0x7f02fc0f62a0 2023-11-17T05:39:15.095 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:15.094+0000 7f02e37fe700 1 -- 172.21.15.160:0/521769634 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f02f002c2c0 con 0x7f02fc0f62a0 2023-11-17T05:39:15.095 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:39:15.095 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:39:15.097 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:15.096+0000 7f03008e1700 1 -- 172.21.15.160:0/521769634 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f02e402ff10 msgr2=0x7f02e40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:15.097 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:15.096+0000 7f03008e1700 1 --2- 172.21.15.160:0/521769634 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f02e402ff10 0x7f02e40323d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f02ec0096a0 tx=0x7f02ec006b40).stop 2023-11-17T05:39:15.097 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:15.096+0000 7f03008e1700 1 -- 172.21.15.160:0/521769634 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02fc0f62a0 msgr2=0x7f02fc1086b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:15.097 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:15.096+0000 7f03008e1700 1 --2- 172.21.15.160:0/521769634 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02fc0f62a0 0x7f02fc1086b0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f02f0009830 tx=0x7f02f000b040).stop 2023-11-17T05:39:15.097 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:15.096+0000 7f03008e1700 1 -- 172.21.15.160:0/521769634 shutdown_connections 2023-11-17T05:39:15.097 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:15.096+0000 7f03008e1700 1 --2- 172.21.15.160:0/521769634 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f02e402ff10 0x7f02e40323d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:15.097 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:15.096+0000 7f03008e1700 1 --2- 172.21.15.160:0/521769634 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02fc0f62a0 0x7f02fc1086b0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:15.097 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:15.096+0000 7f03008e1700 1 -- 172.21.15.160:0/521769634 >> 172.21.15.160:0/521769634 conn(0x7f02fc0f1aa0 msgr2=0x7f02fc0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:39:15.097 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:15.096+0000 7f03008e1700 1 -- 172.21.15.160:0/521769634 shutdown_connections 2023-11-17T05:39:15.097 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:15.096+0000 7f03008e1700 1 -- 172.21.15.160:0/521769634 wait complete. 2023-11-17T05:39:15.098 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:39:15.520 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:15 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:39:14.392828+0000 mgr.smithi003.eddcuc (mgr.14164) 424 : cluster [DBG] pgmap v408: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:39:16.514 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:39:16.515 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:39:16.520 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:16 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:39:15.095354+0000 mon.smithi003 (mon.0) 333 : audit [DBG] from='client.? 172.21.15.160:0/521769634' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:39:17.520 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:17 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:39:16.393099+0000 mgr.smithi003.eddcuc (mgr.14164) 425 : cluster [DBG] pgmap v409: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:39:17.720 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:17.719+0000 7f61503da700 1 -- 172.21.15.160:0/3161527957 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61480601c0 msgr2=0x7f61480f91b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:17.720 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:17.719+0000 7f61503da700 1 --2- 172.21.15.160:0/3161527957 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61480601c0 0x7f61480f91b0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f6144009fb0 tx=0x7f6144009b30).stop 2023-11-17T05:39:17.720 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:17.719+0000 7f61503da700 1 -- 172.21.15.160:0/3161527957 shutdown_connections 2023-11-17T05:39:17.721 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:17.719+0000 7f61503da700 1 --2- 172.21.15.160:0/3161527957 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61480601c0 0x7f61480f91b0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:17.721 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:17.719+0000 7f61503da700 1 -- 172.21.15.160:0/3161527957 >> 172.21.15.160:0/3161527957 conn(0x7f61480f0b70 msgr2=0x7f61480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:39:17.721 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:17.719+0000 7f61503da700 1 -- 172.21.15.160:0/3161527957 shutdown_connections 2023-11-17T05:39:17.721 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:17.719+0000 7f61503da700 1 -- 172.21.15.160:0/3161527957 wait complete. 2023-11-17T05:39:17.722 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:17.720+0000 7f61503da700 1 Processor -- start 2023-11-17T05:39:17.722 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:17.720+0000 7f61503da700 1 -- start start 2023-11-17T05:39:17.722 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:17.720+0000 7f61503da700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61480601c0 0x7f61481066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:39:17.722 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:17.720+0000 7f61503da700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6148106bc0 con 0x7f61480601c0 2023-11-17T05:39:17.722 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:17.720+0000 7f614e176700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61480601c0 0x7f61481066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:39:17.723 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:17.720+0000 7f614e176700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61480601c0 0x7f61481066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:32800/0 (socket says 172.21.15.160:32800) 2023-11-17T05:39:17.723 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:17.720+0000 7f614e176700 1 -- 172.21.15.160:0/2263292319 learned_addr learned my addr 172.21.15.160:0/2263292319 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:39:17.723 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:17.721+0000 7f614e176700 1 -- 172.21.15.160:0/2263292319 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6144005040 con 0x7f61480601c0 2023-11-17T05:39:17.723 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:17.721+0000 7f614e176700 1 --2- 172.21.15.160:0/2263292319 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61480601c0 0x7f61481066d0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f6144013510 tx=0x7f614400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:39:17.723 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:17.721+0000 7f613effd700 1 -- 172.21.15.160:0/2263292319 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6144009320 con 0x7f61480601c0 2023-11-17T05:39:17.723 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:17.721+0000 7f61503da700 1 -- 172.21.15.160:0/2263292319 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6148106dc0 con 0x7f61480601c0 2023-11-17T05:39:17.723 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:17.721+0000 7f613effd700 1 -- 172.21.15.160:0/2263292319 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6144013f00 con 0x7f61480601c0 2023-11-17T05:39:17.723 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:17.721+0000 7f613effd700 1 -- 172.21.15.160:0/2263292319 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6144014dc0 con 0x7f61480601c0 2023-11-17T05:39:17.723 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:17.721+0000 7f61503da700 1 -- 172.21.15.160:0/2263292319 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6148107250 con 0x7f61480601c0 2023-11-17T05:39:17.724 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:17.722+0000 7f613effd700 1 -- 172.21.15.160:0/2263292319 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f61440095f0 con 0x7f61480601c0 2023-11-17T05:39:17.724 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:17.722+0000 7f61503da700 1 -- 172.21.15.160:0/2263292319 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6148045ad0 con 0x7f61480601c0 2023-11-17T05:39:17.724 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:17.722+0000 7f613effd700 1 --2- 172.21.15.160:0/2263292319 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f613402ff60 0x7f6134032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:39:17.724 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:17.722+0000 7f613effd700 1 -- 172.21.15.160:0/2263292319 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f61440410c0 con 0x7f61480601c0 2023-11-17T05:39:17.724 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:17.722+0000 7f614d975700 1 --2- 172.21.15.160:0/2263292319 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f613402ff60 0x7f6134032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:39:17.724 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:17.723+0000 7f614d975700 1 --2- 172.21.15.160:0/2263292319 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f613402ff60 0x7f6134032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f61380096a0 tx=0x7f6138006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:39:17.728 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:17.727+0000 7f613effd700 1 -- 172.21.15.160:0/2263292319 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f614401c790 con 0x7f61480601c0 2023-11-17T05:39:17.930 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:17.930+0000 7f61503da700 1 -- 172.21.15.160:0/2263292319 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6148108100 con 0x7f61480601c0 2023-11-17T05:39:17.931 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:17.930+0000 7f613effd700 1 -- 172.21.15.160:0/2263292319 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f6144010230 con 0x7f61480601c0 2023-11-17T05:39:17.932 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:39:17.932 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:39:17.933 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:17.932+0000 7f61503da700 1 -- 172.21.15.160:0/2263292319 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f613402ff60 msgr2=0x7f6134032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:17.933 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:17.932+0000 7f61503da700 1 --2- 172.21.15.160:0/2263292319 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f613402ff60 0x7f6134032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f61380096a0 tx=0x7f6138006b40).stop 2023-11-17T05:39:17.933 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:17.932+0000 7f61503da700 1 -- 172.21.15.160:0/2263292319 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61480601c0 msgr2=0x7f61481066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:17.933 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:17.932+0000 7f61503da700 1 --2- 172.21.15.160:0/2263292319 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61480601c0 0x7f61481066d0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f6144013510 tx=0x7f614400b040).stop 2023-11-17T05:39:17.933 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:17.932+0000 7f61503da700 1 -- 172.21.15.160:0/2263292319 shutdown_connections 2023-11-17T05:39:17.934 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:17.932+0000 7f61503da700 1 --2- 172.21.15.160:0/2263292319 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f613402ff60 0x7f6134032420 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:17.934 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:17.932+0000 7f61503da700 1 --2- 172.21.15.160:0/2263292319 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f61480601c0 0x7f61481066d0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:17.934 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:17.932+0000 7f61503da700 1 -- 172.21.15.160:0/2263292319 >> 172.21.15.160:0/2263292319 conn(0x7f61480f0b70 msgr2=0x7f61480f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:39:17.934 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:17.932+0000 7f61503da700 1 -- 172.21.15.160:0/2263292319 shutdown_connections 2023-11-17T05:39:17.934 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:17.932+0000 7f61503da700 1 -- 172.21.15.160:0/2263292319 wait complete. 2023-11-17T05:39:17.934 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:39:18.520 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:18 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:39:17.931596+0000 mon.smithi003 (mon.0) 334 : audit [DBG] from='client.? 172.21.15.160:0/2263292319' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:39:19.328 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:39:19.328 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:39:19.520 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:19 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:39:18.393446+0000 mgr.smithi003.eddcuc (mgr.14164) 426 : cluster [DBG] pgmap v410: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:39:20.504 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:20.503+0000 7f9ddaf49700 1 -- 172.21.15.160:0/860756664 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9dd4060280 msgr2=0x7f9dd40f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:20.504 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:20.503+0000 7f9ddaf49700 1 --2- 172.21.15.160:0/860756664 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9dd4060280 0x7f9dd40f90f0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f9dc8009fb0 tx=0x7f9dc8009b30).stop 2023-11-17T05:39:20.504 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:20.503+0000 7f9ddaf49700 1 -- 172.21.15.160:0/860756664 shutdown_connections 2023-11-17T05:39:20.504 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:20.503+0000 7f9ddaf49700 1 --2- 172.21.15.160:0/860756664 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9dd4060280 0x7f9dd40f90f0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:20.505 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:20.503+0000 7f9ddaf49700 1 -- 172.21.15.160:0/860756664 >> 172.21.15.160:0/860756664 conn(0x7f9dd40f0b70 msgr2=0x7f9dd40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:39:20.505 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:20.503+0000 7f9ddaf49700 1 -- 172.21.15.160:0/860756664 shutdown_connections 2023-11-17T05:39:20.505 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:20.503+0000 7f9ddaf49700 1 -- 172.21.15.160:0/860756664 wait complete. 2023-11-17T05:39:20.505 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:20.503+0000 7f9ddaf49700 1 Processor -- start 2023-11-17T05:39:20.505 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:20.503+0000 7f9ddaf49700 1 -- start start 2023-11-17T05:39:20.505 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:20.503+0000 7f9ddaf49700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9dd4060280 0x7f9dd41066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:39:20.505 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:20.503+0000 7f9ddaf49700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd4106bd0 con 0x7f9dd4060280 2023-11-17T05:39:20.506 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:20.504+0000 7f9dd8ce5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9dd4060280 0x7f9dd41066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:39:20.506 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:20.504+0000 7f9dd8ce5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9dd4060280 0x7f9dd41066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:35782/0 (socket says 172.21.15.160:35782) 2023-11-17T05:39:20.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:20.504+0000 7f9dd8ce5700 1 -- 172.21.15.160:0/2360432703 learned_addr learned my addr 172.21.15.160:0/2360432703 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:39:20.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:20.504+0000 7f9dd8ce5700 1 -- 172.21.15.160:0/2360432703 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dc8005040 con 0x7f9dd4060280 2023-11-17T05:39:20.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:20.504+0000 7f9dd8ce5700 1 --2- 172.21.15.160:0/2360432703 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9dd4060280 0x7f9dd41066e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f9dc8013510 tx=0x7f9dc800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:39:20.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:20.504+0000 7f9dd17fa700 1 -- 172.21.15.160:0/2360432703 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9dc8009320 con 0x7f9dd4060280 2023-11-17T05:39:20.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:20.505+0000 7f9ddaf49700 1 -- 172.21.15.160:0/2360432703 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dd4106dd0 con 0x7f9dd4060280 2023-11-17T05:39:20.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:20.505+0000 7f9dd17fa700 1 -- 172.21.15.160:0/2360432703 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9dc8013f00 con 0x7f9dd4060280 2023-11-17T05:39:20.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:20.505+0000 7f9dd17fa700 1 -- 172.21.15.160:0/2360432703 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9dc8014dc0 con 0x7f9dd4060280 2023-11-17T05:39:20.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:20.505+0000 7f9ddaf49700 1 -- 172.21.15.160:0/2360432703 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dd4107260 con 0x7f9dd4060280 2023-11-17T05:39:20.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:20.505+0000 7f9dd17fa700 1 -- 172.21.15.160:0/2360432703 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9dc80095f0 con 0x7f9dd4060280 2023-11-17T05:39:20.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:20.505+0000 7f9ddaf49700 1 -- 172.21.15.160:0/2360432703 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dd4045ad0 con 0x7f9dd4060280 2023-11-17T05:39:20.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:20.506+0000 7f9dd17fa700 1 --2- 172.21.15.160:0/2360432703 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f9dc402ff60 0x7f9dc4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:39:20.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:20.506+0000 7f9dd17fa700 1 -- 172.21.15.160:0/2360432703 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9dc8041040 con 0x7f9dd4060280 2023-11-17T05:39:20.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:20.506+0000 7f9dd3fff700 1 --2- 172.21.15.160:0/2360432703 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f9dc402ff60 0x7f9dc4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:39:20.508 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:20.506+0000 7f9dd3fff700 1 --2- 172.21.15.160:0/2360432703 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f9dc402ff60 0x7f9dc4032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f9dc00096a0 tx=0x7f9dc0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:39:20.511 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:20.511+0000 7f9dd17fa700 1 -- 172.21.15.160:0/2360432703 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9dc8014420 con 0x7f9dd4060280 2023-11-17T05:39:20.720 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:20.719+0000 7f9ddaf49700 1 -- 172.21.15.160:0/2360432703 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9dd4108110 con 0x7f9dd4060280 2023-11-17T05:39:20.721 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:20.720+0000 7f9dd17fa700 1 -- 172.21.15.160:0/2360432703 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f9dc8010230 con 0x7f9dd4060280 2023-11-17T05:39:20.722 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:39:20.722 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:39:20.723 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:20.723+0000 7f9ddaf49700 1 -- 172.21.15.160:0/2360432703 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f9dc402ff60 msgr2=0x7f9dc4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:20.723 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:20.723+0000 7f9ddaf49700 1 --2- 172.21.15.160:0/2360432703 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f9dc402ff60 0x7f9dc4032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f9dc00096a0 tx=0x7f9dc0006b40).stop 2023-11-17T05:39:20.723 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:20.723+0000 7f9ddaf49700 1 -- 172.21.15.160:0/2360432703 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9dd4060280 msgr2=0x7f9dd41066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:20.723 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:20.723+0000 7f9ddaf49700 1 --2- 172.21.15.160:0/2360432703 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9dd4060280 0x7f9dd41066e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f9dc8013510 tx=0x7f9dc800b040).stop 2023-11-17T05:39:20.723 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:20.723+0000 7f9ddaf49700 1 -- 172.21.15.160:0/2360432703 shutdown_connections 2023-11-17T05:39:20.723 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:20.723+0000 7f9ddaf49700 1 --2- 172.21.15.160:0/2360432703 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f9dc402ff60 0x7f9dc4032420 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:20.723 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:20.723+0000 7f9ddaf49700 1 --2- 172.21.15.160:0/2360432703 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9dd4060280 0x7f9dd41066e0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:20.723 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:20.723+0000 7f9ddaf49700 1 -- 172.21.15.160:0/2360432703 >> 172.21.15.160:0/2360432703 conn(0x7f9dd40f0b70 msgr2=0x7f9dd40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:39:20.723 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:20.723+0000 7f9ddaf49700 1 -- 172.21.15.160:0/2360432703 shutdown_connections 2023-11-17T05:39:20.723 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:20.723+0000 7f9ddaf49700 1 -- 172.21.15.160:0/2360432703 wait complete. 2023-11-17T05:39:20.724 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:39:21.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:21 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:39:20.393764+0000 mgr.smithi003.eddcuc (mgr.14164) 427 : cluster [DBG] pgmap v411: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:39:21.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:21 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:39:20.720927+0000 mon.smithi003 (mon.0) 335 : audit [DBG] from='client.? 172.21.15.160:0/2360432703' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:39:22.133 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:39:22.134 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:39:23.289 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:23.289+0000 7fa85c935700 1 -- 172.21.15.160:0/44821436 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8580f5410 msgr2=0x7fa8580f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:23.289 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:23.289+0000 7fa85c935700 1 --2- 172.21.15.160:0/44821436 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8580f5410 0x7fa8580f5830 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7fa848009fb0 tx=0x7fa848009b30).stop 2023-11-17T05:39:23.289 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:23.289+0000 7fa85c935700 1 -- 172.21.15.160:0/44821436 shutdown_connections 2023-11-17T05:39:23.289 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:23.289+0000 7fa85c935700 1 --2- 172.21.15.160:0/44821436 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8580f5410 0x7fa8580f5830 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:23.289 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:23.289+0000 7fa85c935700 1 -- 172.21.15.160:0/44821436 >> 172.21.15.160:0/44821436 conn(0x7fa8580f0b70 msgr2=0x7fa8580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:39:23.289 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:23.289+0000 7fa85c935700 1 -- 172.21.15.160:0/44821436 shutdown_connections 2023-11-17T05:39:23.289 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:23.289+0000 7fa85c935700 1 -- 172.21.15.160:0/44821436 wait complete. 2023-11-17T05:39:23.291 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:23.289+0000 7fa85c935700 1 Processor -- start 2023-11-17T05:39:23.291 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:23.290+0000 7fa85c935700 1 -- start start 2023-11-17T05:39:23.291 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:23.290+0000 7fa85c935700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8580f5410 0x7fa858109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:39:23.291 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:23.290+0000 7fa85c935700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa858060240 con 0x7fa8580f5410 2023-11-17T05:39:23.291 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:23.290+0000 7fa85659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8580f5410 0x7fa858109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:39:23.291 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:23.290+0000 7fa85659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8580f5410 0x7fa858109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:35792/0 (socket says 172.21.15.160:35792) 2023-11-17T05:39:23.291 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:23.290+0000 7fa85659c700 1 -- 172.21.15.160:0/1791900607 learned_addr learned my addr 172.21.15.160:0/1791900607 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:39:23.291 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:23.290+0000 7fa85659c700 1 -- 172.21.15.160:0/1791900607 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa848005040 con 0x7fa8580f5410 2023-11-17T05:39:23.291 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:23.290+0000 7fa85659c700 1 --2- 172.21.15.160:0/1791900607 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8580f5410 0x7fa858109430 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7fa848013510 tx=0x7fa84800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:39:23.292 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:23.291+0000 7fa846ffd700 1 -- 172.21.15.160:0/1791900607 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa848009320 con 0x7fa8580f5410 2023-11-17T05:39:23.292 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:23.291+0000 7fa846ffd700 1 -- 172.21.15.160:0/1791900607 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa848013f00 con 0x7fa8580f5410 2023-11-17T05:39:23.292 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:23.291+0000 7fa85c935700 1 -- 172.21.15.160:0/1791900607 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa858060440 con 0x7fa8580f5410 2023-11-17T05:39:23.292 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:23.291+0000 7fa846ffd700 1 -- 172.21.15.160:0/1791900607 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa848014d90 con 0x7fa8580f5410 2023-11-17T05:39:23.292 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:23.291+0000 7fa85c935700 1 -- 172.21.15.160:0/1791900607 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa858109ae0 con 0x7fa8580f5410 2023-11-17T05:39:23.293 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:23.291+0000 7fa846ffd700 1 -- 172.21.15.160:0/1791900607 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa8480095f0 con 0x7fa8580f5410 2023-11-17T05:39:23.293 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:23.291+0000 7fa85c935700 1 -- 172.21.15.160:0/1791900607 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa858045ad0 con 0x7fa8580f5410 2023-11-17T05:39:23.293 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:23.292+0000 7fa846ffd700 1 --2- 172.21.15.160:0/1791900607 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa84002ff60 0x7fa840032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:39:23.293 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:23.292+0000 7fa846ffd700 1 -- 172.21.15.160:0/1791900607 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa848041000 con 0x7fa8580f5410 2023-11-17T05:39:23.293 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:23.292+0000 7fa855d9b700 1 --2- 172.21.15.160:0/1791900607 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa84002ff60 0x7fa840032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:39:23.293 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:23.292+0000 7fa855d9b700 1 --2- 172.21.15.160:0/1791900607 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa84002ff60 0x7fa840032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fa84c0096a0 tx=0x7fa84c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:39:23.296 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:23.296+0000 7fa846ffd700 1 -- 172.21.15.160:0/1791900607 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa848014420 con 0x7fa8580f5410 2023-11-17T05:39:23.503 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:23.504+0000 7fa85c935700 1 -- 172.21.15.160:0/1791900607 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa8580fae60 con 0x7fa8580f5410 2023-11-17T05:39:23.504 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:23.504+0000 7fa846ffd700 1 -- 172.21.15.160:0/1791900607 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fa84801d7f0 con 0x7fa8580f5410 2023-11-17T05:39:23.504 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:39:23.504 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:39:23.506 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:23.506+0000 7fa85c935700 1 -- 172.21.15.160:0/1791900607 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa84002ff60 msgr2=0x7fa840032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:23.506 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:23.506+0000 7fa85c935700 1 --2- 172.21.15.160:0/1791900607 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa84002ff60 0x7fa840032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fa84c0096a0 tx=0x7fa84c006b40).stop 2023-11-17T05:39:23.506 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:23.506+0000 7fa85c935700 1 -- 172.21.15.160:0/1791900607 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8580f5410 msgr2=0x7fa858109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:23.506 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:23.506+0000 7fa85c935700 1 --2- 172.21.15.160:0/1791900607 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8580f5410 0x7fa858109430 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7fa848013510 tx=0x7fa84800b040).stop 2023-11-17T05:39:23.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:23.506+0000 7fa85c935700 1 -- 172.21.15.160:0/1791900607 shutdown_connections 2023-11-17T05:39:23.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:23.506+0000 7fa85c935700 1 --2- 172.21.15.160:0/1791900607 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fa84002ff60 0x7fa840032420 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:23.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:23.506+0000 7fa85c935700 1 --2- 172.21.15.160:0/1791900607 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8580f5410 0x7fa858109430 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:23.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:23.506+0000 7fa85c935700 1 -- 172.21.15.160:0/1791900607 >> 172.21.15.160:0/1791900607 conn(0x7fa8580f0b70 msgr2=0x7fa8580f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:39:23.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:23.506+0000 7fa85c935700 1 -- 172.21.15.160:0/1791900607 shutdown_connections 2023-11-17T05:39:23.507 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:23.506+0000 7fa85c935700 1 -- 172.21.15.160:0/1791900607 wait complete. 2023-11-17T05:39:23.507 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:39:23.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:23 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:39:22.394028+0000 mgr.smithi003.eddcuc (mgr.14164) 428 : cluster [DBG] pgmap v412: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:39:24.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:24 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:39:23.504380+0000 mon.smithi003 (mon.0) 336 : audit [DBG] from='client.? 172.21.15.160:0/1791900607' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:39:27.853 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:39:27.913 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:39:27.929 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:25 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:39:24.394405+0000 mgr.smithi003.eddcuc (mgr.14164) 429 : cluster [DBG] pgmap v413: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:39:27.930 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:27 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:39:26.394792+0000 mgr.smithi003.eddcuc (mgr.14164) 430 : cluster [DBG] pgmap v414: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:39:28.773 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:39:27.560728+0000 mon.smithi003 (mon.0) 337 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.eddcuc/mirror_snapshot_schedule"}]: dispatch 2023-11-17T05:39:28.800 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:28 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:39:27.563372+0000 mon.smithi003 (mon.0) 338 : audit [INF] from='mgr.14164 172.21.15.3:0/529273834' entity='mgr.smithi003.eddcuc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi003.eddcuc/trash_purge_schedule"}]: dispatch 2023-11-17T05:39:29.201 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:29.199+0000 7feef47c7700 1 -- 172.21.15.160:0/162291663 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feeec0f5410 msgr2=0x7feeec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:29.201 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:29.199+0000 7feef47c7700 1 --2- 172.21.15.160:0/162291663 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feeec0f5410 0x7feeec0f5830 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7feee0009fb0 tx=0x7feee0009b30).stop 2023-11-17T05:39:29.201 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:29.199+0000 7feef47c7700 1 -- 172.21.15.160:0/162291663 shutdown_connections 2023-11-17T05:39:29.201 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:29.199+0000 7feef47c7700 1 --2- 172.21.15.160:0/162291663 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feeec0f5410 0x7feeec0f5830 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:29.201 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:29.199+0000 7feef47c7700 1 -- 172.21.15.160:0/162291663 >> 172.21.15.160:0/162291663 conn(0x7feeec0f0b70 msgr2=0x7feeec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:39:29.201 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:29.199+0000 7feef47c7700 1 -- 172.21.15.160:0/162291663 shutdown_connections 2023-11-17T05:39:29.201 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:29.199+0000 7feef47c7700 1 -- 172.21.15.160:0/162291663 wait complete. 2023-11-17T05:39:29.202 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:29.200+0000 7feef47c7700 1 Processor -- start 2023-11-17T05:39:29.202 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:29.200+0000 7feef47c7700 1 -- start start 2023-11-17T05:39:29.202 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:29.200+0000 7feef47c7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feeec0f5410 0x7feeec0fd330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:39:29.258 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:29.200+0000 7feef47c7700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeec0fb820 con 0x7feeec0f5410 2023-11-17T05:39:29.258 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:29.200+0000 7feef2563700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feeec0f5410 0x7feeec0fd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:39:29.258 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:29.200+0000 7feef2563700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feeec0f5410 0x7feeec0fd330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:35812/0 (socket says 172.21.15.160:35812) 2023-11-17T05:39:29.258 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:29.200+0000 7feef2563700 1 -- 172.21.15.160:0/3487222220 learned_addr learned my addr 172.21.15.160:0/3487222220 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:39:29.258 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:29.200+0000 7feef2563700 1 -- 172.21.15.160:0/3487222220 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee0005040 con 0x7feeec0f5410 2023-11-17T05:39:29.259 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:29.200+0000 7feef2563700 1 --2- 172.21.15.160:0/3487222220 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feeec0f5410 0x7feeec0fd330 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7feee0013510 tx=0x7feee000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:39:29.260 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:29.200+0000 7feedeffd700 1 -- 172.21.15.160:0/3487222220 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feee0009320 con 0x7feeec0f5410 2023-11-17T05:39:29.260 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:29.201+0000 7feef47c7700 1 -- 172.21.15.160:0/3487222220 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feeec0fba20 con 0x7feeec0f5410 2023-11-17T05:39:29.260 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:29.201+0000 7feedeffd700 1 -- 172.21.15.160:0/3487222220 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feee0013f00 con 0x7feeec0f5410 2023-11-17T05:39:29.260 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:29.201+0000 7feedeffd700 1 -- 172.21.15.160:0/3487222220 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feee0014d90 con 0x7feeec0f5410 2023-11-17T05:39:29.260 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:29.201+0000 7feef47c7700 1 -- 172.21.15.160:0/3487222220 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feeec0fbeb0 con 0x7feeec0f5410 2023-11-17T05:39:29.260 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:29.201+0000 7feedeffd700 1 -- 172.21.15.160:0/3487222220 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feee00095f0 con 0x7feeec0f5410 2023-11-17T05:39:29.260 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:29.201+0000 7feef47c7700 1 -- 172.21.15.160:0/3487222220 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feeec059100 con 0x7feeec0f5410 2023-11-17T05:39:29.260 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:29.202+0000 7feedeffd700 1 --2- 172.21.15.160:0/3487222220 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7feed802ff60 0x7feed8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:39:29.260 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:29.202+0000 7feedeffd700 1 -- 172.21.15.160:0/3487222220 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feee00410c0 con 0x7feeec0f5410 2023-11-17T05:39:29.261 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:29.202+0000 7feef1d62700 1 --2- 172.21.15.160:0/3487222220 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7feed802ff60 0x7feed8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:39:29.261 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:29.202+0000 7feef1d62700 1 --2- 172.21.15.160:0/3487222220 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7feed802ff60 0x7feed8032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7feee80096a0 tx=0x7feee8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:39:29.261 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:29.207+0000 7feedeffd700 1 -- 172.21.15.160:0/3487222220 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feee0014420 con 0x7feeec0f5410 2023-11-17T05:39:29.412 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:29.411+0000 7feef47c7700 1 -- 172.21.15.160:0/3487222220 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feeec0fae60 con 0x7feeec0f5410 2023-11-17T05:39:29.413 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:29.412+0000 7feedeffd700 1 -- 172.21.15.160:0/3487222220 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7feee0010230 con 0x7feeec0f5410 2023-11-17T05:39:29.413 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:39:29.413 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:39:29.414 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:29.413+0000 7feef47c7700 1 -- 172.21.15.160:0/3487222220 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7feed802ff60 msgr2=0x7feed8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:29.414 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:29.413+0000 7feef47c7700 1 --2- 172.21.15.160:0/3487222220 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7feed802ff60 0x7feed8032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7feee80096a0 tx=0x7feee8006b40).stop 2023-11-17T05:39:29.414 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:29.413+0000 7feef47c7700 1 -- 172.21.15.160:0/3487222220 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feeec0f5410 msgr2=0x7feeec0fd330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:29.414 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:29.413+0000 7feef47c7700 1 --2- 172.21.15.160:0/3487222220 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feeec0f5410 0x7feeec0fd330 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7feee0013510 tx=0x7feee000b040).stop 2023-11-17T05:39:29.415 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:29.413+0000 7feef47c7700 1 -- 172.21.15.160:0/3487222220 shutdown_connections 2023-11-17T05:39:29.415 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:29.413+0000 7feef47c7700 1 --2- 172.21.15.160:0/3487222220 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7feed802ff60 0x7feed8032420 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:29.415 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:29.413+0000 7feef47c7700 1 --2- 172.21.15.160:0/3487222220 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feeec0f5410 0x7feeec0fd330 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:29.415 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:29.413+0000 7feef47c7700 1 -- 172.21.15.160:0/3487222220 >> 172.21.15.160:0/3487222220 conn(0x7feeec0f0b70 msgr2=0x7feeec05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:39:29.415 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:29.413+0000 7feef47c7700 1 -- 172.21.15.160:0/3487222220 shutdown_connections 2023-11-17T05:39:29.415 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:29.414+0000 7feef47c7700 1 -- 172.21.15.160:0/3487222220 wait complete. 2023-11-17T05:39:29.415 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:39:29.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:29 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:39:28.395158+0000 mgr.smithi003.eddcuc (mgr.14164) 431 : cluster [DBG] pgmap v415: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:39:29.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:29 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:39:29.413193+0000 mon.smithi003 (mon.0) 339 : audit [DBG] from='client.? 172.21.15.160:0/3487222220' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:39:30.791 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:39:30.791 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:39:31.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:31 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:39:30.395551+0000 mgr.smithi003.eddcuc (mgr.14164) 432 : cluster [DBG] pgmap v416: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:39:32.006 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:32.004+0000 7fb86e7f2700 1 -- 172.21.15.160:0/1481032040 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8680f6510 msgr2=0x7fb8680f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:32.007 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:32.004+0000 7fb86e7f2700 1 --2- 172.21.15.160:0/1481032040 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8680f6510 0x7fb8680f6930 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7fb85c009fb0 tx=0x7fb85c009b30).stop 2023-11-17T05:39:32.007 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:32.004+0000 7fb86e7f2700 1 -- 172.21.15.160:0/1481032040 shutdown_connections 2023-11-17T05:39:32.007 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:32.004+0000 7fb86e7f2700 1 --2- 172.21.15.160:0/1481032040 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8680f6510 0x7fb8680f6930 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:32.007 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:32.004+0000 7fb86e7f2700 1 -- 172.21.15.160:0/1481032040 >> 172.21.15.160:0/1481032040 conn(0x7fb8680f1c70 msgr2=0x7fb8680f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:39:32.007 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:32.004+0000 7fb86e7f2700 1 -- 172.21.15.160:0/1481032040 shutdown_connections 2023-11-17T05:39:32.007 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:32.004+0000 7fb86e7f2700 1 -- 172.21.15.160:0/1481032040 wait complete. 2023-11-17T05:39:32.007 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:32.004+0000 7fb86e7f2700 1 Processor -- start 2023-11-17T05:39:32.007 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:32.005+0000 7fb86e7f2700 1 -- start start 2023-11-17T05:39:32.007 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:32.005+0000 7fb86e7f2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8680f6510 0x7fb868106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:39:32.007 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:32.005+0000 7fb86e7f2700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb868106c20 con 0x7fb8680f6510 2023-11-17T05:39:32.007 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:32.005+0000 7fb867fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8680f6510 0x7fb868106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:39:32.008 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:32.005+0000 7fb867fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8680f6510 0x7fb868106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:41758/0 (socket says 172.21.15.160:41758) 2023-11-17T05:39:32.008 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:32.005+0000 7fb867fff700 1 -- 172.21.15.160:0/545797346 learned_addr learned my addr 172.21.15.160:0/545797346 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:39:32.008 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:32.005+0000 7fb867fff700 1 -- 172.21.15.160:0/545797346 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb85c005040 con 0x7fb8680f6510 2023-11-17T05:39:32.008 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:32.006+0000 7fb867fff700 1 --2- 172.21.15.160:0/545797346 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8680f6510 0x7fb868106730 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7fb85c013510 tx=0x7fb85c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:39:32.008 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:32.006+0000 7fb864ff9700 1 -- 172.21.15.160:0/545797346 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb85c009320 con 0x7fb8680f6510 2023-11-17T05:39:32.008 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:32.006+0000 7fb86e7f2700 1 -- 172.21.15.160:0/545797346 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb868106e20 con 0x7fb8680f6510 2023-11-17T05:39:32.008 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:32.006+0000 7fb864ff9700 1 -- 172.21.15.160:0/545797346 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb85c013f00 con 0x7fb8680f6510 2023-11-17T05:39:32.008 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:32.006+0000 7fb864ff9700 1 -- 172.21.15.160:0/545797346 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb85c014d90 con 0x7fb8680f6510 2023-11-17T05:39:32.008 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:32.006+0000 7fb86e7f2700 1 -- 172.21.15.160:0/545797346 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8681072b0 con 0x7fb8680f6510 2023-11-17T05:39:32.009 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:32.007+0000 7fb864ff9700 1 -- 172.21.15.160:0/545797346 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb85c0095f0 con 0x7fb8680f6510 2023-11-17T05:39:32.009 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:32.007+0000 7fb86e7f2700 1 -- 172.21.15.160:0/545797346 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb868045ad0 con 0x7fb8680f6510 2023-11-17T05:39:32.009 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:32.007+0000 7fb864ff9700 1 --2- 172.21.15.160:0/545797346 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fb85002ff60 0x7fb850032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:39:32.009 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:32.007+0000 7fb864ff9700 1 -- 172.21.15.160:0/545797346 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb85c0410c0 con 0x7fb8680f6510 2023-11-17T05:39:32.009 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:32.007+0000 7fb8677fe700 1 --2- 172.21.15.160:0/545797346 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fb85002ff60 0x7fb850032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:39:32.009 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:32.008+0000 7fb8677fe700 1 --2- 172.21.15.160:0/545797346 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fb85002ff60 0x7fb850032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fb8580096a0 tx=0x7fb858006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:39:32.012 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:32.012+0000 7fb864ff9700 1 -- 172.21.15.160:0/545797346 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb85c014420 con 0x7fb8680f6510 2023-11-17T05:39:32.215 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:32.215+0000 7fb86e7f2700 1 -- 172.21.15.160:0/545797346 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb8680fbf60 con 0x7fb8680f6510 2023-11-17T05:39:32.216 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:32.216+0000 7fb864ff9700 1 -- 172.21.15.160:0/545797346 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fb85c01d7f0 con 0x7fb8680f6510 2023-11-17T05:39:32.216 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:39:32.217 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:39:32.219 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:32.217+0000 7fb86e7f2700 1 -- 172.21.15.160:0/545797346 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fb85002ff60 msgr2=0x7fb850032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:32.219 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:32.217+0000 7fb86e7f2700 1 --2- 172.21.15.160:0/545797346 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fb85002ff60 0x7fb850032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fb8580096a0 tx=0x7fb858006b40).stop 2023-11-17T05:39:32.219 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:32.217+0000 7fb86e7f2700 1 -- 172.21.15.160:0/545797346 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8680f6510 msgr2=0x7fb868106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:32.219 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:32.217+0000 7fb86e7f2700 1 --2- 172.21.15.160:0/545797346 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8680f6510 0x7fb868106730 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7fb85c013510 tx=0x7fb85c00b040).stop 2023-11-17T05:39:32.219 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:32.217+0000 7fb86e7f2700 1 -- 172.21.15.160:0/545797346 shutdown_connections 2023-11-17T05:39:32.219 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:32.217+0000 7fb86e7f2700 1 --2- 172.21.15.160:0/545797346 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fb85002ff60 0x7fb850032420 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:32.219 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:32.217+0000 7fb86e7f2700 1 --2- 172.21.15.160:0/545797346 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8680f6510 0x7fb868106730 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:32.219 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:32.217+0000 7fb86e7f2700 1 -- 172.21.15.160:0/545797346 >> 172.21.15.160:0/545797346 conn(0x7fb8680f1c70 msgr2=0x7fb8680f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:39:32.219 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:32.217+0000 7fb86e7f2700 1 -- 172.21.15.160:0/545797346 shutdown_connections 2023-11-17T05:39:32.220 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:32.217+0000 7fb86e7f2700 1 -- 172.21.15.160:0/545797346 wait complete. 2023-11-17T05:39:32.220 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:39:32.771 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:32 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:39:32.216666+0000 mon.smithi003 (mon.0) 340 : audit [DBG] from='client.? 172.21.15.160:0/545797346' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:39:33.656 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:39:33.657 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:39:33.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:33 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:39:32.395891+0000 mgr.smithi003.eddcuc (mgr.14164) 433 : cluster [DBG] pgmap v417: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:39:34.809 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:34.807+0000 7fe28e4e3700 1 -- 172.21.15.160:0/2589881537 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2880f5410 msgr2=0x7fe2880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:34.809 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:34.807+0000 7fe28e4e3700 1 --2- 172.21.15.160:0/2589881537 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2880f5410 0x7fe2880f5830 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7fe278009fb0 tx=0x7fe278009b30).stop 2023-11-17T05:39:34.809 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:34.807+0000 7fe28e4e3700 1 -- 172.21.15.160:0/2589881537 shutdown_connections 2023-11-17T05:39:34.809 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:34.807+0000 7fe28e4e3700 1 --2- 172.21.15.160:0/2589881537 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2880f5410 0x7fe2880f5830 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:34.809 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:34.807+0000 7fe28e4e3700 1 -- 172.21.15.160:0/2589881537 >> 172.21.15.160:0/2589881537 conn(0x7fe2880f0b70 msgr2=0x7fe2880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:39:34.809 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:34.807+0000 7fe28e4e3700 1 -- 172.21.15.160:0/2589881537 shutdown_connections 2023-11-17T05:39:34.809 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:34.807+0000 7fe28e4e3700 1 -- 172.21.15.160:0/2589881537 wait complete. 2023-11-17T05:39:34.809 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:34.808+0000 7fe28e4e3700 1 Processor -- start 2023-11-17T05:39:34.810 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:34.808+0000 7fe28e4e3700 1 -- start start 2023-11-17T05:39:34.810 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:34.808+0000 7fe28e4e3700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2880f5410 0x7fe28810b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:39:34.810 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:34.808+0000 7fe28e4e3700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe288107240 con 0x7fe2880f5410 2023-11-17T05:39:34.811 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:34.809+0000 7fe287fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2880f5410 0x7fe28810b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:39:34.811 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:34.809+0000 7fe287fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2880f5410 0x7fe28810b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:41778/0 (socket says 172.21.15.160:41778) 2023-11-17T05:39:34.811 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:34.809+0000 7fe287fff700 1 -- 172.21.15.160:0/2349284864 learned_addr learned my addr 172.21.15.160:0/2349284864 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:39:34.811 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:34.809+0000 7fe287fff700 1 -- 172.21.15.160:0/2349284864 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe278005040 con 0x7fe2880f5410 2023-11-17T05:39:34.811 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:34.809+0000 7fe287fff700 1 --2- 172.21.15.160:0/2349284864 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2880f5410 0x7fe28810b850 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fe278013510 tx=0x7fe27800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:39:34.812 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:34.809+0000 7fe284ff9700 1 -- 172.21.15.160:0/2349284864 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe278009320 con 0x7fe2880f5410 2023-11-17T05:39:34.812 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:34.809+0000 7fe28e4e3700 1 -- 172.21.15.160:0/2349284864 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe288060220 con 0x7fe2880f5410 2023-11-17T05:39:34.812 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:34.809+0000 7fe284ff9700 1 -- 172.21.15.160:0/2349284864 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe278013f00 con 0x7fe2880f5410 2023-11-17T05:39:34.812 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:34.809+0000 7fe284ff9700 1 -- 172.21.15.160:0/2349284864 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe278014d90 con 0x7fe2880f5410 2023-11-17T05:39:34.812 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:34.809+0000 7fe28e4e3700 1 -- 172.21.15.160:0/2349284864 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe28810bd40 con 0x7fe2880f5410 2023-11-17T05:39:34.812 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:34.810+0000 7fe284ff9700 1 -- 172.21.15.160:0/2349284864 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe2780095f0 con 0x7fe2880f5410 2023-11-17T05:39:34.812 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:34.810+0000 7fe28e4e3700 1 -- 172.21.15.160:0/2349284864 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe288045ad0 con 0x7fe2880f5410 2023-11-17T05:39:34.812 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:34.810+0000 7fe284ff9700 1 --2- 172.21.15.160:0/2349284864 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe27002ffb0 0x7fe270032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:39:34.812 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:34.810+0000 7fe284ff9700 1 -- 172.21.15.160:0/2349284864 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe278040f90 con 0x7fe2880f5410 2023-11-17T05:39:34.812 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:34.811+0000 7fe2877fe700 1 --2- 172.21.15.160:0/2349284864 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe27002ffb0 0x7fe270032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:39:34.812 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:34.811+0000 7fe2877fe700 1 --2- 172.21.15.160:0/2349284864 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe27002ffb0 0x7fe270032470 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fe27c0096a0 tx=0x7fe27c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:39:34.816 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:34.815+0000 7fe284ff9700 1 -- 172.21.15.160:0/2349284864 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe278014420 con 0x7fe2880f5410 2023-11-17T05:39:35.021 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:35.020+0000 7fe28e4e3700 1 -- 172.21.15.160:0/2349284864 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe2880fae60 con 0x7fe2880f5410 2023-11-17T05:39:35.022 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:35.021+0000 7fe284ff9700 1 -- 172.21.15.160:0/2349284864 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7fe278010230 con 0x7fe2880f5410 2023-11-17T05:39:35.022 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:39:35.022 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:39:35.023 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:35.022+0000 7fe28e4e3700 1 -- 172.21.15.160:0/2349284864 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe27002ffb0 msgr2=0x7fe270032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:35.023 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:35.022+0000 7fe28e4e3700 1 --2- 172.21.15.160:0/2349284864 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe27002ffb0 0x7fe270032470 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fe27c0096a0 tx=0x7fe27c006b40).stop 2023-11-17T05:39:35.024 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:35.022+0000 7fe28e4e3700 1 -- 172.21.15.160:0/2349284864 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2880f5410 msgr2=0x7fe28810b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:35.024 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:35.022+0000 7fe28e4e3700 1 --2- 172.21.15.160:0/2349284864 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2880f5410 0x7fe28810b850 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fe278013510 tx=0x7fe27800b040).stop 2023-11-17T05:39:35.024 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:35.022+0000 7fe28e4e3700 1 -- 172.21.15.160:0/2349284864 shutdown_connections 2023-11-17T05:39:35.024 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:35.022+0000 7fe28e4e3700 1 --2- 172.21.15.160:0/2349284864 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7fe27002ffb0 0x7fe270032470 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:35.024 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:35.022+0000 7fe28e4e3700 1 --2- 172.21.15.160:0/2349284864 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe2880f5410 0x7fe28810b850 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:35.024 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:35.022+0000 7fe28e4e3700 1 -- 172.21.15.160:0/2349284864 >> 172.21.15.160:0/2349284864 conn(0x7fe2880f0b70 msgr2=0x7fe2880f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:39:35.024 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:35.022+0000 7fe28e4e3700 1 -- 172.21.15.160:0/2349284864 shutdown_connections 2023-11-17T05:39:35.024 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:35.023+0000 7fe28e4e3700 1 -- 172.21.15.160:0/2349284864 wait complete. 2023-11-17T05:39:35.024 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:39:35.520 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:35 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:39:34.396250+0000 mgr.smithi003.eddcuc (mgr.14164) 434 : cluster [DBG] pgmap v418: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:39:35.520 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:35 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:39:35.022000+0000 mon.smithi003 (mon.0) 341 : audit [DBG] from='client.? 172.21.15.160:0/2349284864' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:39:36.410 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:39:36.410 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:39:37.548 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:37.547+0000 7f06fda2f700 1 -- 172.21.15.160:0/2078200469 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06f80f51a0 msgr2=0x7f06f80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:37.549 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:37.547+0000 7f06fda2f700 1 --2- 172.21.15.160:0/2078200469 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06f80f51a0 0x7f06f80f55c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f06ec009fb0 tx=0x7f06ec009b30).stop 2023-11-17T05:39:37.549 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:37.548+0000 7f06fda2f700 1 -- 172.21.15.160:0/2078200469 shutdown_connections 2023-11-17T05:39:37.549 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:37.548+0000 7f06fda2f700 1 --2- 172.21.15.160:0/2078200469 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06f80f51a0 0x7f06f80f55c0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:37.549 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:37.548+0000 7f06fda2f700 1 -- 172.21.15.160:0/2078200469 >> 172.21.15.160:0/2078200469 conn(0x7f06f80f09a0 msgr2=0x7f06f80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:39:37.549 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:37.548+0000 7f06fda2f700 1 -- 172.21.15.160:0/2078200469 shutdown_connections 2023-11-17T05:39:37.549 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:37.548+0000 7f06fda2f700 1 -- 172.21.15.160:0/2078200469 wait complete. 2023-11-17T05:39:37.550 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:37.548+0000 7f06fda2f700 1 Processor -- start 2023-11-17T05:39:37.550 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:37.548+0000 7f06fda2f700 1 -- start start 2023-11-17T05:39:37.551 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:37.548+0000 7f06fda2f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06f80f51a0 0x7f06f8109200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:39:37.551 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:37.548+0000 7f06fda2f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f8060220 con 0x7f06f80f51a0 2023-11-17T05:39:37.551 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:37.549+0000 7f06fca2d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06f80f51a0 0x7f06f8109200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:39:37.551 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:37.549+0000 7f06fca2d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06f80f51a0 0x7f06f8109200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:41802/0 (socket says 172.21.15.160:41802) 2023-11-17T05:39:37.551 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:37.549+0000 7f06fca2d700 1 -- 172.21.15.160:0/3419373681 learned_addr learned my addr 172.21.15.160:0/3419373681 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:39:37.551 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:37.549+0000 7f06fca2d700 1 -- 172.21.15.160:0/3419373681 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06ec005040 con 0x7f06f80f51a0 2023-11-17T05:39:37.551 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:37.549+0000 7f06fca2d700 1 --2- 172.21.15.160:0/3419373681 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06f80f51a0 0x7f06f8109200 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f06ec013510 tx=0x7f06ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:39:37.551 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:37.549+0000 7f06f57fa700 1 -- 172.21.15.160:0/3419373681 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06ec009320 con 0x7f06f80f51a0 2023-11-17T05:39:37.551 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:37.550+0000 7f06fda2f700 1 -- 172.21.15.160:0/3419373681 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06f8060420 con 0x7f06f80f51a0 2023-11-17T05:39:37.551 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:37.550+0000 7f06f57fa700 1 -- 172.21.15.160:0/3419373681 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f06ec013f00 con 0x7f06f80f51a0 2023-11-17T05:39:37.551 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:37.550+0000 7f06f57fa700 1 -- 172.21.15.160:0/3419373681 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06ec014dc0 con 0x7f06f80f51a0 2023-11-17T05:39:37.551 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:37.550+0000 7f06fda2f700 1 -- 172.21.15.160:0/3419373681 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06f81098b0 con 0x7f06f80f51a0 2023-11-17T05:39:37.552 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:37.550+0000 7f06f57fa700 1 -- 172.21.15.160:0/3419373681 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f06ec0095f0 con 0x7f06f80f51a0 2023-11-17T05:39:37.552 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:37.550+0000 7f06fda2f700 1 -- 172.21.15.160:0/3419373681 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06f8059100 con 0x7f06f80f51a0 2023-11-17T05:39:37.552 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:37.550+0000 7f06f57fa700 1 --2- 172.21.15.160:0/3419373681 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f06e802ffb0 0x7f06e8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:39:37.552 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:37.551+0000 7f06f57fa700 1 -- 172.21.15.160:0/3419373681 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f06ec040fb0 con 0x7f06f80f51a0 2023-11-17T05:39:37.552 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:37.551+0000 7f06f7fff700 1 --2- 172.21.15.160:0/3419373681 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f06e802ffb0 0x7f06e8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:39:37.553 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:37.551+0000 7f06f7fff700 1 --2- 172.21.15.160:0/3419373681 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f06e802ffb0 0x7f06e8032470 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f06e40096a0 tx=0x7f06e4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:39:37.557 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:37.556+0000 7f06f57fa700 1 -- 172.21.15.160:0/3419373681 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f06ec06d2b0 con 0x7f06f80f51a0 2023-11-17T05:39:37.761 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:37.760+0000 7f06fda2f700 1 -- 172.21.15.160:0/3419373681 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f06f8045ad0 con 0x7f06f80f51a0 2023-11-17T05:39:37.761 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:37.760+0000 7f06f57fa700 1 -- 172.21.15.160:0/3419373681 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f06ec014420 con 0x7f06f80f51a0 2023-11-17T05:39:37.761 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:39:37.761 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:39:37.763 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:37.762+0000 7f06fda2f700 1 -- 172.21.15.160:0/3419373681 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f06e802ffb0 msgr2=0x7f06e8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:37.763 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:37.762+0000 7f06fda2f700 1 --2- 172.21.15.160:0/3419373681 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f06e802ffb0 0x7f06e8032470 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f06e40096a0 tx=0x7f06e4006b40).stop 2023-11-17T05:39:37.763 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:37.762+0000 7f06fda2f700 1 -- 172.21.15.160:0/3419373681 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06f80f51a0 msgr2=0x7f06f8109200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:37.763 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:37.762+0000 7f06fda2f700 1 --2- 172.21.15.160:0/3419373681 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06f80f51a0 0x7f06f8109200 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f06ec013510 tx=0x7f06ec00b040).stop 2023-11-17T05:39:37.763 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:37.762+0000 7f06fda2f700 1 -- 172.21.15.160:0/3419373681 shutdown_connections 2023-11-17T05:39:37.763 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:37.762+0000 7f06fda2f700 1 --2- 172.21.15.160:0/3419373681 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f06e802ffb0 0x7f06e8032470 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:37.763 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:37.762+0000 7f06fda2f700 1 --2- 172.21.15.160:0/3419373681 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06f80f51a0 0x7f06f8109200 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:37.764 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:37.762+0000 7f06fda2f700 1 -- 172.21.15.160:0/3419373681 >> 172.21.15.160:0/3419373681 conn(0x7f06f80f09a0 msgr2=0x7f06f80f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:39:37.764 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:37.762+0000 7f06fda2f700 1 -- 172.21.15.160:0/3419373681 shutdown_connections 2023-11-17T05:39:37.764 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:37.762+0000 7f06fda2f700 1 -- 172.21.15.160:0/3419373681 wait complete. 2023-11-17T05:39:37.764 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:39:37.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:37 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:39:36.396591+0000 mgr.smithi003.eddcuc (mgr.14164) 435 : cluster [DBG] pgmap v419: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:39:38.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:38 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:39:37.761723+0000 mon.smithi003 (mon.0) 342 : audit [DBG] from='client.? 172.21.15.160:0/3419373681' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:39:39.140 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:39:39.140 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:39:39.771 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:39 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:39:38.396822+0000 mgr.smithi003.eddcuc (mgr.14164) 436 : cluster [DBG] pgmap v420: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:39:40.290 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:40.288+0000 7f9d1dbef700 1 -- 172.21.15.160:0/695210982 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d180f4740 msgr2=0x7f9d180f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:40.290 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:40.288+0000 7f9d1dbef700 1 --2- 172.21.15.160:0/695210982 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d180f4740 0x7f9d180f4b60 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f9d08009fb0 tx=0x7f9d08009b30).stop 2023-11-17T05:39:40.290 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:40.288+0000 7f9d1dbef700 1 -- 172.21.15.160:0/695210982 shutdown_connections 2023-11-17T05:39:40.290 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:40.288+0000 7f9d1dbef700 1 --2- 172.21.15.160:0/695210982 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d180f4740 0x7f9d180f4b60 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:40.290 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:40.288+0000 7f9d1dbef700 1 -- 172.21.15.160:0/695210982 >> 172.21.15.160:0/695210982 conn(0x7f9d180f0260 msgr2=0x7f9d180f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:39:40.290 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:40.288+0000 7f9d1dbef700 1 -- 172.21.15.160:0/695210982 shutdown_connections 2023-11-17T05:39:40.290 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:40.288+0000 7f9d1dbef700 1 -- 172.21.15.160:0/695210982 wait complete. 2023-11-17T05:39:40.290 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:40.289+0000 7f9d1dbef700 1 Processor -- start 2023-11-17T05:39:40.290 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:40.289+0000 7f9d1dbef700 1 -- start start 2023-11-17T05:39:40.291 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:40.289+0000 7f9d1dbef700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d180f4740 0x7f9d181066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:39:40.291 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:40.289+0000 7f9d1dbef700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d18106bc0 con 0x7f9d180f4740 2023-11-17T05:39:40.292 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:40.290+0000 7f9d177fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d180f4740 0x7f9d181066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:39:40.292 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:40.290+0000 7f9d177fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d180f4740 0x7f9d181066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:56980/0 (socket says 172.21.15.160:56980) 2023-11-17T05:39:40.292 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:40.290+0000 7f9d177fe700 1 -- 172.21.15.160:0/3154999085 learned_addr learned my addr 172.21.15.160:0/3154999085 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:39:40.292 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:40.290+0000 7f9d177fe700 1 -- 172.21.15.160:0/3154999085 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d08005040 con 0x7f9d180f4740 2023-11-17T05:39:40.292 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:40.290+0000 7f9d177fe700 1 --2- 172.21.15.160:0/3154999085 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d180f4740 0x7f9d181066d0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f9d08013510 tx=0x7f9d0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:39:40.292 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:40.290+0000 7f9d0ffff700 1 -- 172.21.15.160:0/3154999085 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d08009320 con 0x7f9d180f4740 2023-11-17T05:39:40.292 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:40.290+0000 7f9d0ffff700 1 -- 172.21.15.160:0/3154999085 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9d08013f00 con 0x7f9d180f4740 2023-11-17T05:39:40.292 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:40.290+0000 7f9d1dbef700 1 -- 172.21.15.160:0/3154999085 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d18106dc0 con 0x7f9d180f4740 2023-11-17T05:39:40.292 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:40.290+0000 7f9d0ffff700 1 -- 172.21.15.160:0/3154999085 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d08014d90 con 0x7f9d180f4740 2023-11-17T05:39:40.292 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:40.290+0000 7f9d1dbef700 1 -- 172.21.15.160:0/3154999085 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d18107250 con 0x7f9d180f4740 2023-11-17T05:39:40.293 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:40.291+0000 7f9d0ffff700 1 -- 172.21.15.160:0/3154999085 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9d08014420 con 0x7f9d180f4740 2023-11-17T05:39:40.293 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:40.291+0000 7f9d1dbef700 1 -- 172.21.15.160:0/3154999085 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d18045ad0 con 0x7f9d180f4740 2023-11-17T05:39:40.293 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:40.291+0000 7f9d0ffff700 1 --2- 172.21.15.160:0/3154999085 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f9d0402ff60 0x7f9d04032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:39:40.293 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:40.291+0000 7f9d0ffff700 1 -- 172.21.15.160:0/3154999085 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9d08041070 con 0x7f9d180f4740 2023-11-17T05:39:40.293 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:40.291+0000 7f9d16ffd700 1 --2- 172.21.15.160:0/3154999085 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f9d0402ff60 0x7f9d04032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:39:40.293 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:40.292+0000 7f9d16ffd700 1 --2- 172.21.15.160:0/3154999085 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f9d0402ff60 0x7f9d04032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f9d000096a0 tx=0x7f9d00006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:39:40.297 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:40.296+0000 7f9d0ffff700 1 -- 172.21.15.160:0/3154999085 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9d080095f0 con 0x7f9d180f4740 2023-11-17T05:39:40.504 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:40.502+0000 7f9d1dbef700 1 -- 172.21.15.160:0/3154999085 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9d18108100 con 0x7f9d180f4740 2023-11-17T05:39:40.505 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:40.503+0000 7f9d0ffff700 1 -- 172.21.15.160:0/3154999085 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f9d0801db70 con 0x7f9d180f4740 2023-11-17T05:39:40.505 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:39:40.505 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:39:40.506 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:40.505+0000 7f9d1dbef700 1 -- 172.21.15.160:0/3154999085 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f9d0402ff60 msgr2=0x7f9d04032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:40.506 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:40.505+0000 7f9d1dbef700 1 --2- 172.21.15.160:0/3154999085 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f9d0402ff60 0x7f9d04032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f9d000096a0 tx=0x7f9d00006b40).stop 2023-11-17T05:39:40.506 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:40.505+0000 7f9d1dbef700 1 -- 172.21.15.160:0/3154999085 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d180f4740 msgr2=0x7f9d181066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:40.506 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:40.505+0000 7f9d1dbef700 1 --2- 172.21.15.160:0/3154999085 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d180f4740 0x7f9d181066d0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f9d08013510 tx=0x7f9d0800b040).stop 2023-11-17T05:39:40.506 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:40.505+0000 7f9d1dbef700 1 -- 172.21.15.160:0/3154999085 shutdown_connections 2023-11-17T05:39:40.506 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:40.505+0000 7f9d1dbef700 1 --2- 172.21.15.160:0/3154999085 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f9d0402ff60 0x7f9d04032420 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:40.506 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:40.505+0000 7f9d1dbef700 1 --2- 172.21.15.160:0/3154999085 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d180f4740 0x7f9d181066d0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:40.506 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:40.505+0000 7f9d1dbef700 1 -- 172.21.15.160:0/3154999085 >> 172.21.15.160:0/3154999085 conn(0x7f9d180f0260 msgr2=0x7f9d180f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:39:40.506 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:40.505+0000 7f9d1dbef700 1 -- 172.21.15.160:0/3154999085 shutdown_connections 2023-11-17T05:39:40.506 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:40.505+0000 7f9d1dbef700 1 -- 172.21.15.160:0/3154999085 wait complete. 2023-11-17T05:39:40.507 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:39:41.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:41 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:39:40.397137+0000 mgr.smithi003.eddcuc (mgr.14164) 437 : cluster [DBG] pgmap v421: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:39:41.770 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:41 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:39:40.504423+0000 mon.smithi003 (mon.0) 343 : audit [DBG] from='client.? 172.21.15.160:0/3154999085' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:39:41.940 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:39:41.941 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:39:43.122 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:43.121+0000 7f0f4dd59700 1 -- 172.21.15.160:0/3945867050 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f480f5410 msgr2=0x7f0f480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:43.122 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:43.121+0000 7f0f4dd59700 1 --2- 172.21.15.160:0/3945867050 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f480f5410 0x7f0f480f5830 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f0f3c009fb0 tx=0x7f0f3c009b30).stop 2023-11-17T05:39:43.122 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:43.121+0000 7f0f4dd59700 1 -- 172.21.15.160:0/3945867050 shutdown_connections 2023-11-17T05:39:43.122 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:43.121+0000 7f0f4dd59700 1 --2- 172.21.15.160:0/3945867050 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f480f5410 0x7f0f480f5830 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:43.122 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:43.121+0000 7f0f4dd59700 1 -- 172.21.15.160:0/3945867050 >> 172.21.15.160:0/3945867050 conn(0x7f0f480f0b70 msgr2=0x7f0f480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:39:43.122 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:43.121+0000 7f0f4dd59700 1 -- 172.21.15.160:0/3945867050 shutdown_connections 2023-11-17T05:39:43.123 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:43.121+0000 7f0f4dd59700 1 -- 172.21.15.160:0/3945867050 wait complete. 2023-11-17T05:39:43.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:43.122+0000 7f0f4dd59700 1 Processor -- start 2023-11-17T05:39:43.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:43.122+0000 7f0f4dd59700 1 -- start start 2023-11-17T05:39:43.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:43.122+0000 7f0f4dd59700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f480f5410 0x7f0f48109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:39:43.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:43.122+0000 7f0f4dd59700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f48060240 con 0x7f0f480f5410 2023-11-17T05:39:43.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:43.122+0000 7f0f477fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f480f5410 0x7f0f48109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:39:43.124 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:43.123+0000 7f0f477fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f480f5410 0x7f0f48109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:56984/0 (socket says 172.21.15.160:56984) 2023-11-17T05:39:43.125 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:43.123+0000 7f0f477fe700 1 -- 172.21.15.160:0/831504209 learned_addr learned my addr 172.21.15.160:0/831504209 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:39:43.125 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:43.123+0000 7f0f477fe700 1 -- 172.21.15.160:0/831504209 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f3c005040 con 0x7f0f480f5410 2023-11-17T05:39:43.125 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:43.123+0000 7f0f477fe700 1 --2- 172.21.15.160:0/831504209 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f480f5410 0x7f0f48109430 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f0f3c013510 tx=0x7f0f3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:39:43.125 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:43.123+0000 7f0f2ffff700 1 -- 172.21.15.160:0/831504209 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f3c009320 con 0x7f0f480f5410 2023-11-17T05:39:43.125 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:43.123+0000 7f0f2ffff700 1 -- 172.21.15.160:0/831504209 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0f3c013f00 con 0x7f0f480f5410 2023-11-17T05:39:43.125 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:43.123+0000 7f0f4dd59700 1 -- 172.21.15.160:0/831504209 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f48060440 con 0x7f0f480f5410 2023-11-17T05:39:43.125 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:43.123+0000 7f0f2ffff700 1 -- 172.21.15.160:0/831504209 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f3c014d90 con 0x7f0f480f5410 2023-11-17T05:39:43.125 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:43.123+0000 7f0f4dd59700 1 -- 172.21.15.160:0/831504209 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f48109ae0 con 0x7f0f480f5410 2023-11-17T05:39:43.126 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:43.124+0000 7f0f2ffff700 1 -- 172.21.15.160:0/831504209 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0f3c0095f0 con 0x7f0f480f5410 2023-11-17T05:39:43.126 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:43.124+0000 7f0f4dd59700 1 -- 172.21.15.160:0/831504209 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f48045ad0 con 0x7f0f480f5410 2023-11-17T05:39:43.126 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:43.124+0000 7f0f2ffff700 1 --2- 172.21.15.160:0/831504209 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f0f3002ff60 0x7f0f30032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:39:43.126 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:43.124+0000 7f0f2ffff700 1 -- 172.21.15.160:0/831504209 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0f3c041000 con 0x7f0f480f5410 2023-11-17T05:39:43.126 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:43.124+0000 7f0f46ffd700 1 --2- 172.21.15.160:0/831504209 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f0f3002ff60 0x7f0f30032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:39:43.126 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:43.125+0000 7f0f46ffd700 1 --2- 172.21.15.160:0/831504209 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f0f3002ff60 0x7f0f30032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f0f380096a0 tx=0x7f0f38006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:39:43.130 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:43.129+0000 7f0f2ffff700 1 -- 172.21.15.160:0/831504209 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0f3c014420 con 0x7f0f480f5410 2023-11-17T05:39:43.334 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:43.333+0000 7f0f4dd59700 1 -- 172.21.15.160:0/831504209 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0f480fae60 con 0x7f0f480f5410 2023-11-17T05:39:43.335 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:43.334+0000 7f0f2ffff700 1 -- 172.21.15.160:0/831504209 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f0f3c01d7f0 con 0x7f0f480f5410 2023-11-17T05:39:43.335 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:39:43.335 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:39:43.336 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:43.335+0000 7f0f4dd59700 1 -- 172.21.15.160:0/831504209 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f0f3002ff60 msgr2=0x7f0f30032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:43.336 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:43.335+0000 7f0f4dd59700 1 --2- 172.21.15.160:0/831504209 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f0f3002ff60 0x7f0f30032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f0f380096a0 tx=0x7f0f38006b40).stop 2023-11-17T05:39:43.336 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:43.335+0000 7f0f4dd59700 1 -- 172.21.15.160:0/831504209 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f480f5410 msgr2=0x7f0f48109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:43.336 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:43.335+0000 7f0f4dd59700 1 --2- 172.21.15.160:0/831504209 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f480f5410 0x7f0f48109430 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f0f3c013510 tx=0x7f0f3c00b040).stop 2023-11-17T05:39:43.337 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:43.335+0000 7f0f4dd59700 1 -- 172.21.15.160:0/831504209 shutdown_connections 2023-11-17T05:39:43.337 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:43.335+0000 7f0f4dd59700 1 --2- 172.21.15.160:0/831504209 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f0f3002ff60 0x7f0f30032420 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:43.337 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:43.335+0000 7f0f4dd59700 1 --2- 172.21.15.160:0/831504209 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f480f5410 0x7f0f48109430 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:43.337 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:43.335+0000 7f0f4dd59700 1 -- 172.21.15.160:0/831504209 >> 172.21.15.160:0/831504209 conn(0x7f0f480f0b70 msgr2=0x7f0f480f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:39:43.337 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:43.335+0000 7f0f4dd59700 1 -- 172.21.15.160:0/831504209 shutdown_connections 2023-11-17T05:39:43.337 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:43.335+0000 7f0f4dd59700 1 -- 172.21.15.160:0/831504209 wait complete. 2023-11-17T05:39:43.337 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:39:43.771 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:43 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:39:42.397557+0000 mgr.smithi003.eddcuc (mgr.14164) 438 : cluster [DBG] pgmap v422: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:39:43.771 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:43 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:39:43.335130+0000 mon.smithi003 (mon.0) 344 : audit [DBG] from='client.? 172.21.15.160:0/831504209' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:39:44.740 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:39:44.741 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:39:45.520 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:45 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:39:44.397874+0000 mgr.smithi003.eddcuc (mgr.14164) 439 : cluster [DBG] pgmap v423: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:39:45.993 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:45.992+0000 7f89f7b51700 1 -- 172.21.15.160:0/1846662588 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89f00f6510 msgr2=0x7f89f00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:45.993 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:45.992+0000 7f89f7b51700 1 --2- 172.21.15.160:0/1846662588 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89f00f6510 0x7f89f00f6930 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f89e4009fb0 tx=0x7f89e4009b30).stop 2023-11-17T05:39:45.993 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:45.992+0000 7f89f7b51700 1 -- 172.21.15.160:0/1846662588 shutdown_connections 2023-11-17T05:39:45.993 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:45.992+0000 7f89f7b51700 1 --2- 172.21.15.160:0/1846662588 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89f00f6510 0x7f89f00f6930 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:45.994 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:45.992+0000 7f89f7b51700 1 -- 172.21.15.160:0/1846662588 >> 172.21.15.160:0/1846662588 conn(0x7f89f00f1c70 msgr2=0x7f89f00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:39:45.994 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:45.992+0000 7f89f7b51700 1 -- 172.21.15.160:0/1846662588 shutdown_connections 2023-11-17T05:39:45.994 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:45.992+0000 7f89f7b51700 1 -- 172.21.15.160:0/1846662588 wait complete. 2023-11-17T05:39:45.995 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:45.993+0000 7f89f7b51700 1 Processor -- start 2023-11-17T05:39:45.995 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:45.993+0000 7f89f7b51700 1 -- start start 2023-11-17T05:39:45.995 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:45.993+0000 7f89f7b51700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89f00f6510 0x7f89f0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:39:45.995 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:45.993+0000 7f89f7b51700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f0106c20 con 0x7f89f00f6510 2023-11-17T05:39:45.995 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:45.993+0000 7f89f58ed700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89f00f6510 0x7f89f0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:39:45.995 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:45.993+0000 7f89f58ed700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89f00f6510 0x7f89f0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:56996/0 (socket says 172.21.15.160:56996) 2023-11-17T05:39:45.995 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:45.993+0000 7f89f58ed700 1 -- 172.21.15.160:0/860705869 learned_addr learned my addr 172.21.15.160:0/860705869 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:39:45.995 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:45.994+0000 7f89f58ed700 1 -- 172.21.15.160:0/860705869 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e4005040 con 0x7f89f00f6510 2023-11-17T05:39:45.996 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:45.994+0000 7f89f58ed700 1 --2- 172.21.15.160:0/860705869 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89f00f6510 0x7f89f0106730 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f89e4013510 tx=0x7f89e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:39:45.996 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:45.994+0000 7f89e27fc700 1 -- 172.21.15.160:0/860705869 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89e4009320 con 0x7f89f00f6510 2023-11-17T05:39:45.996 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:45.994+0000 7f89f7b51700 1 -- 172.21.15.160:0/860705869 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89f0106e20 con 0x7f89f00f6510 2023-11-17T05:39:45.996 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:45.994+0000 7f89e27fc700 1 -- 172.21.15.160:0/860705869 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f89e4013f00 con 0x7f89f00f6510 2023-11-17T05:39:45.996 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:45.994+0000 7f89e27fc700 1 -- 172.21.15.160:0/860705869 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89e4014d90 con 0x7f89f00f6510 2023-11-17T05:39:45.996 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:45.994+0000 7f89f7b51700 1 -- 172.21.15.160:0/860705869 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89f01072b0 con 0x7f89f00f6510 2023-11-17T05:39:45.997 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:45.995+0000 7f89f7b51700 1 -- 172.21.15.160:0/860705869 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89f0045ad0 con 0x7f89f00f6510 2023-11-17T05:39:45.997 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:45.995+0000 7f89e27fc700 1 -- 172.21.15.160:0/860705869 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f89e40095f0 con 0x7f89f00f6510 2023-11-17T05:39:45.997 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:45.995+0000 7f89e27fc700 1 --2- 172.21.15.160:0/860705869 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f89dc02ff60 0x7f89dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:39:45.997 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:45.995+0000 7f89e27fc700 1 -- 172.21.15.160:0/860705869 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f89e4041000 con 0x7f89f00f6510 2023-11-17T05:39:45.997 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:45.995+0000 7f89f50ec700 1 --2- 172.21.15.160:0/860705869 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f89dc02ff60 0x7f89dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:39:45.997 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:45.996+0000 7f89f50ec700 1 --2- 172.21.15.160:0/860705869 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f89dc02ff60 0x7f89dc032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f89ec0096a0 tx=0x7f89ec006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:39:46.000 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:46.000+0000 7f89e27fc700 1 -- 172.21.15.160:0/860705869 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f89e4014420 con 0x7f89f00f6510 2023-11-17T05:39:46.206 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:46.205+0000 7f89f7b51700 1 -- 172.21.15.160:0/860705869 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f89f00fbf60 con 0x7f89f00f6510 2023-11-17T05:39:46.206 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:46.206+0000 7f89e27fc700 1 -- 172.21.15.160:0/860705869 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f89e4010230 con 0x7f89f00f6510 2023-11-17T05:39:46.207 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:39:46.207 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:39:46.208 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:46.207+0000 7f89f7b51700 1 -- 172.21.15.160:0/860705869 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f89dc02ff60 msgr2=0x7f89dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:46.208 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:46.207+0000 7f89f7b51700 1 --2- 172.21.15.160:0/860705869 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f89dc02ff60 0x7f89dc032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f89ec0096a0 tx=0x7f89ec006b40).stop 2023-11-17T05:39:46.208 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:46.207+0000 7f89f7b51700 1 -- 172.21.15.160:0/860705869 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89f00f6510 msgr2=0x7f89f0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:46.209 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:46.207+0000 7f89f7b51700 1 --2- 172.21.15.160:0/860705869 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89f00f6510 0x7f89f0106730 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f89e4013510 tx=0x7f89e400b040).stop 2023-11-17T05:39:46.209 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:46.207+0000 7f89f7b51700 1 -- 172.21.15.160:0/860705869 shutdown_connections 2023-11-17T05:39:46.209 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:46.207+0000 7f89f7b51700 1 --2- 172.21.15.160:0/860705869 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f89dc02ff60 0x7f89dc032420 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:46.209 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:46.207+0000 7f89f7b51700 1 --2- 172.21.15.160:0/860705869 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f89f00f6510 0x7f89f0106730 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:46.209 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:46.207+0000 7f89f7b51700 1 -- 172.21.15.160:0/860705869 >> 172.21.15.160:0/860705869 conn(0x7f89f00f1c70 msgr2=0x7f89f00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:39:46.209 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:46.207+0000 7f89f7b51700 1 -- 172.21.15.160:0/860705869 shutdown_connections 2023-11-17T05:39:46.209 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:46.207+0000 7f89f7b51700 1 -- 172.21.15.160:0/860705869 wait complete. 2023-11-17T05:39:46.209 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:39:46.521 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:46 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:39:46.206907+0000 mon.smithi003 (mon.0) 345 : audit [DBG] from='client.? 172.21.15.160:0/860705869' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:39:47.520 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:47 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:39:46.398133+0000 mgr.smithi003.eddcuc (mgr.14164) 440 : cluster [DBG] pgmap v424: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:39:47.627 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-17T05:39:47.627 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-17T05:39:48.798 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:48.797+0000 7f6e5e866700 1 -- 172.21.15.160:0/1370728216 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e58060200 msgr2=0x7f6e580f9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:48.799 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:48.797+0000 7f6e5e866700 1 --2- 172.21.15.160:0/1370728216 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e58060200 0x7f6e580f9170 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7f6e4c009fb0 tx=0x7f6e4c009b30).stop 2023-11-17T05:39:48.799 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:48.797+0000 7f6e5e866700 1 -- 172.21.15.160:0/1370728216 shutdown_connections 2023-11-17T05:39:48.799 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:48.797+0000 7f6e5e866700 1 --2- 172.21.15.160:0/1370728216 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e58060200 0x7f6e580f9170 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:48.799 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:48.797+0000 7f6e5e866700 1 -- 172.21.15.160:0/1370728216 >> 172.21.15.160:0/1370728216 conn(0x7f6e580f0b70 msgr2=0x7f6e580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:39:48.799 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:48.797+0000 7f6e5e866700 1 -- 172.21.15.160:0/1370728216 shutdown_connections 2023-11-17T05:39:48.799 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:48.797+0000 7f6e5e866700 1 -- 172.21.15.160:0/1370728216 wait complete. 2023-11-17T05:39:48.799 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:48.798+0000 7f6e5e866700 1 Processor -- start 2023-11-17T05:39:48.800 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:48.798+0000 7f6e5e866700 1 -- start start 2023-11-17T05:39:48.800 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:48.798+0000 7f6e5e866700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e58060200 0x7f6e580f6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:39:48.800 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:48.798+0000 7f6e5e866700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e580f5070 con 0x7f6e58060200 2023-11-17T05:39:48.800 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:48.799+0000 7f6e57fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e58060200 0x7f6e580f6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:39:48.801 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:48.799+0000 7f6e57fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e58060200 0x7f6e580f6bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.160:57022/0 (socket says 172.21.15.160:57022) 2023-11-17T05:39:48.801 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:48.799+0000 7f6e57fff700 1 -- 172.21.15.160:0/3899178974 learned_addr learned my addr 172.21.15.160:0/3899178974 (peer_addr_for_me v2:172.21.15.160:0/0) 2023-11-17T05:39:48.801 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:48.799+0000 7f6e57fff700 1 -- 172.21.15.160:0/3899178974 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e4c005040 con 0x7f6e58060200 2023-11-17T05:39:48.801 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:48.799+0000 7f6e57fff700 1 --2- 172.21.15.160:0/3899178974 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e58060200 0x7f6e580f6bf0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7f6e4c013510 tx=0x7f6e4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:39:48.801 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:48.799+0000 7f6e54ff9700 1 -- 172.21.15.160:0/3899178974 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e4c009320 con 0x7f6e58060200 2023-11-17T05:39:48.801 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:48.799+0000 7f6e5e866700 1 -- 172.21.15.160:0/3899178974 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e580f5210 con 0x7f6e58060200 2023-11-17T05:39:48.801 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:48.799+0000 7f6e54ff9700 1 -- 172.21.15.160:0/3899178974 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6e4c013f00 con 0x7f6e58060200 2023-11-17T05:39:48.801 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:48.799+0000 7f6e54ff9700 1 -- 172.21.15.160:0/3899178974 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e4c014dc0 con 0x7f6e58060200 2023-11-17T05:39:48.801 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:48.799+0000 7f6e5e866700 1 -- 172.21.15.160:0/3899178974 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e580f56a0 con 0x7f6e58060200 2023-11-17T05:39:48.802 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:48.800+0000 7f6e5e866700 1 -- 172.21.15.160:0/3899178974 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e58059100 con 0x7f6e58060200 2023-11-17T05:39:48.802 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:48.800+0000 7f6e54ff9700 1 -- 172.21.15.160:0/3899178974 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6e4c014420 con 0x7f6e58060200 2023-11-17T05:39:48.802 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:48.800+0000 7f6e54ff9700 1 --2- 172.21.15.160:0/3899178974 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f6e4002ff60 0x7f6e40032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-17T05:39:48.802 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:48.800+0000 7f6e54ff9700 1 -- 172.21.15.160:0/3899178974 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6e4c041020 con 0x7f6e58060200 2023-11-17T05:39:48.802 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:48.800+0000 7f6e577fe700 1 --2- 172.21.15.160:0/3899178974 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f6e4002ff60 0x7f6e40032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-17T05:39:48.802 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:48.801+0000 7f6e577fe700 1 --2- 172.21.15.160:0/3899178974 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f6e4002ff60 0x7f6e40032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f6e480096a0 tx=0x7f6e48006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-17T05:39:48.805 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:48.805+0000 7f6e54ff9700 1 -- 172.21.15.160:0/3899178974 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6e4c0095f0 con 0x7f6e58060200 2023-11-17T05:39:49.012 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:49.011+0000 7f6e5e866700 1 -- 172.21.15.160:0/3899178974 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6e58045ad0 con 0x7f6e58060200 2023-11-17T05:39:49.012 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:49.012+0000 7f6e54ff9700 1 -- 172.21.15.160:0/3899178974 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+686 (secure 0 0 0) 0x7f6e4c01cb10 con 0x7f6e58060200 2023-11-17T05:39:49.012 INFO:teuthology.orchestra.run.smithi160.stdout: 2023-11-17T05:39:49.013 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"851feb1e-8509-11ee-8dbe-212e2dc638e7","modified":"2023-11-17T05:24:28.545719Z","created":"2023-11-17T05:24:28.545719Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi003","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-17T05:39:49.052 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:49.013+0000 7f6e5e866700 1 -- 172.21.15.160:0/3899178974 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f6e4002ff60 msgr2=0x7f6e40032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:49.052 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:49.013+0000 7f6e5e866700 1 --2- 172.21.15.160:0/3899178974 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f6e4002ff60 0x7f6e40032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f6e480096a0 tx=0x7f6e48006b40).stop 2023-11-17T05:39:49.052 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:49.013+0000 7f6e5e866700 1 -- 172.21.15.160:0/3899178974 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e58060200 msgr2=0x7f6e580f6bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-17T05:39:49.052 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:49.013+0000 7f6e5e866700 1 --2- 172.21.15.160:0/3899178974 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e58060200 0x7f6e580f6bf0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7f6e4c013510 tx=0x7f6e4c00b040).stop 2023-11-17T05:39:49.053 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:49.013+0000 7f6e5e866700 1 -- 172.21.15.160:0/3899178974 shutdown_connections 2023-11-17T05:39:49.053 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:49.013+0000 7f6e5e866700 1 --2- 172.21.15.160:0/3899178974 >> [v2:172.21.15.3:6800/17534166,v1:172.21.15.3:6801/17534166] conn(0x7f6e4002ff60 0x7f6e40032420 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:49.053 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:49.013+0000 7f6e5e866700 1 --2- 172.21.15.160:0/3899178974 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e58060200 0x7f6e580f6bf0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-17T05:39:49.053 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:49.013+0000 7f6e5e866700 1 -- 172.21.15.160:0/3899178974 >> 172.21.15.160:0/3899178974 conn(0x7f6e580f0b70 msgr2=0x7f6e580f1650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-17T05:39:49.053 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:49.013+0000 7f6e5e866700 1 -- 172.21.15.160:0/3899178974 shutdown_connections 2023-11-17T05:39:49.053 INFO:teuthology.orchestra.run.smithi160.stderr:2023-11-17T05:39:49.013+0000 7f6e5e866700 1 -- 172.21.15.160:0/3899178974 wait complete. 2023-11-17T05:39:49.053 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2023-11-17T05:39:49.399 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2023-11-17T05:39:49.400 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-11-17T05:39:49.400 DEBUG:teuthology.orchestra.run.smithi003:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-11-17T05:39:49.426 DEBUG:teuthology.orchestra.run.smithi160:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-11-17T05:39:49.447 INFO:tasks.cephadm:Stopping all daemons... 2023-11-17T05:39:49.448 INFO:tasks.cephadm.mon.smithi003:Stopping mon.smithi003... 2023-11-17T05:39:49.448 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7@mon.smithi003 2023-11-17T05:39:49.730 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:49 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: cluster 2023-11-17T05:39:48.398504+0000 mgr.smithi003.eddcuc (mgr.14164) 441 : cluster [DBG] pgmap v425: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-17T05:39:49.730 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:49 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: audit 2023-11-17T05:39:49.012835+0000 mon.smithi003 (mon.0) 346 : audit [DBG] from='client.? 172.21.15.160:0/3899178974' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-17T05:39:49.730 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:49 smithi003 systemd[1]: Stopping Ceph mon.smithi003 for 851feb1e-8509-11ee-8dbe-212e2dc638e7... 2023-11-17T05:39:49.730 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:49 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:39:49.587+0000 7f392092b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi003 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-11-17T05:39:49.730 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:49 smithi003 ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003[111810]: debug 2023-11-17T05:39:49.587+0000 7f392092b700 -1 mon.smithi003@0(leader) e1 *** Got Signal Terminated *** 2023-11-17T05:39:50.020 INFO:journalctl@ceph.mon.smithi003.smithi003.stdout:Nov 17 05:39:49 smithi003 podman[117775]: ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7-mon.smithi003 2023-11-17T05:39:50.035 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7@mon.smithi003.service' 2023-11-17T05:39:50.091 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-17T05:39:50.091 INFO:tasks.cephadm.mon.smithi003:Stopped mon.smithi003 2023-11-17T05:39:50.091 INFO:tasks.cephadm.mon.smithi160:Stopping mon.smithi160... 2023-11-17T05:39:50.091 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7@mon.smithi160 2023-11-17T05:39:50.130 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to stop ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7@mon.smithi160.service: Unit ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7@mon.smithi160.service not loaded. 2023-11-17T05:39:50.131 DEBUG:teuthology.orchestra.run:got remote process result: 5 2023-11-17T05:39:50.132 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi160" Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi160 with status 5: 'sudo systemctl stop ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7@mon.smithi160' 2023-11-17T05:39:50.133 INFO:tasks.cephadm:Archiving crash dumps... 2023-11-17T05:39:50.137 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/crash to /home/teuthworker/archive/yuriw-2023-11-16_22:29:26-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7461279/remote/ubuntu@smithi003.front.sepia.ceph.com/crash 2023-11-17T05:39:50.138 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/crash -- . 2023-11-17T05:39:50.183 DEBUG:teuthology.misc:Transferring archived files from smithi160:/var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/crash to /home/teuthworker/archive/yuriw-2023-11-16_22:29:26-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7461279/remote/ubuntu@smithi160.front.sepia.ceph.com/crash 2023-11-17T05:39:50.184 DEBUG:teuthology.orchestra.run.smithi160:> sudo tar c -f - -C /var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/crash -- . 2023-11-17T05:39:50.234 INFO:teuthology.orchestra.run.smithi160.stderr:tar: /var/lib/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/crash: Cannot open: No such file or directory 2023-11-17T05:39:50.236 INFO:teuthology.orchestra.run.smithi160.stderr:tar: Error is not recoverable: exiting now 2023-11-17T05:39:50.237 INFO:tasks.cephadm:Checking cluster log for badness... 2023-11-17T05:39:50.238 DEBUG:teuthology.orchestra.run.smithi003:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-11-17T05:39:50.277 INFO:teuthology.orchestra.run.smithi003.stderr:grep: /var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/ceph.log: No such file or directory 2023-11-17T05:39:50.279 INFO:tasks.cephadm:Compressing logs... 2023-11-17T05:39:50.279 DEBUG:teuthology.orchestra.run.smithi003:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-17T05:39:50.321 DEBUG:teuthology.orchestra.run.smithi160:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-17T05:39:50.352 INFO:teuthology.orchestra.run.smithi160.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-11-17T05:39:50.353 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-11-17T05:39:50.353 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/ceph-volume.log 2023-11-17T05:39:50.353 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/ceph-volume.log: 88.3% -- replaced with /var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/ceph-volume.log.gz 2023-11-17T05:39:50.354 INFO:teuthology.orchestra.run.smithi003.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-11-17T05:39:50.354 INFO:teuthology.orchestra.run.smithi003.stderr:: No such file or directory 2023-11-17T05:39:50.355 INFO:teuthology.orchestra.run.smithi160.stderr: 95.4% -- replaced with /var/log/ceph/cephadm.log.gz 2023-11-17T05:39:50.355 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/ceph-volume.log 2023-11-17T05:39:50.356 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi003.eddcuc.log 2023-11-17T05:39:50.356 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/ceph-mon.smithi003.log 2023-11-17T05:39:50.356 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi003.eddcuc.log: 88.3% -- replaced with /var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/ceph-volume.log.gz 2023-11-17T05:39:50.358 INFO:teuthology.orchestra.run.smithi160.stderr: 2023-11-17T05:39:50.358 INFO:teuthology.orchestra.run.smithi160.stderr:real 0m0.015s 2023-11-17T05:39:50.358 INFO:teuthology.orchestra.run.smithi160.stderr:user 0m0.011s 2023-11-17T05:39:50.358 INFO:teuthology.orchestra.run.smithi160.stderr:sys 0m0.014s 2023-11-17T05:39:50.360 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/ceph-mon.smithi003.log: 91.6% -- replaced with /var/log/ceph/cephadm.log.gz 2023-11-17T05:39:50.471 INFO:teuthology.orchestra.run.smithi003.stderr:gzip: /var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi003.eddcuc.log: file size changed while zipping 2023-11-17T05:39:50.471 INFO:teuthology.orchestra.run.smithi003.stderr: 93.0% -- replaced with /var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi003.eddcuc.log.gz 2023-11-17T05:39:50.555 INFO:teuthology.orchestra.run.smithi003.stderr: 90.9% -- replaced with /var/log/ceph/851feb1e-8509-11ee-8dbe-212e2dc638e7/ceph-mon.smithi003.log.gz 2023-11-17T05:39:50.557 INFO:teuthology.orchestra.run.smithi003.stderr: 2023-11-17T05:39:50.557 INFO:teuthology.orchestra.run.smithi003.stderr:real 0m0.216s 2023-11-17T05:39:50.558 INFO:teuthology.orchestra.run.smithi003.stderr:user 0m0.324s 2023-11-17T05:39:50.558 INFO:teuthology.orchestra.run.smithi003.stderr:sys 0m0.025s 2023-11-17T05:39:50.558 INFO:tasks.cephadm:Archiving logs... 2023-11-17T05:39:50.558 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-11-16_22:29:26-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7461279/remote/ubuntu@smithi003.front.sepia.ceph.com/log 2023-11-17T05:39:50.559 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-17T05:39:50.692 DEBUG:teuthology.misc:Transferring archived files from smithi160:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-11-16_22:29:26-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7461279/remote/ubuntu@smithi160.front.sepia.ceph.com/log 2023-11-17T05:39:50.695 DEBUG:teuthology.orchestra.run.smithi160:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-17T05:39:50.744 INFO:tasks.cephadm:Removing cluster... 2023-11-17T05:39:50.744 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 --force 2023-11-17T05:39:53.074 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 851feb1e-8509-11ee-8dbe-212e2dc638e7 --force 2023-11-17T05:40:24.052 INFO:tasks.cephadm:Removing cephadm ... 2023-11-17T05:40:24.052 DEBUG:teuthology.orchestra.run.smithi003:> rm -rf /home/ubuntu/cephtest/cephadm 2023-11-17T05:40:24.078 DEBUG:teuthology.orchestra.run.smithi160:> rm -rf /home/ubuntu/cephtest/cephadm 2023-11-17T05:40:24.098 INFO:tasks.cephadm:Teardown complete 2023-11-17T05:40:24.099 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi160 with status 5: 'sudo systemctl stop ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7@mon.smithi160' 2023-11-17T05:40:24.188 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=65219d3be67641dcac4fb73d9490aeff Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_ed3c8e926b9de93026c8a7c1d86be5d6821b52ac/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi160 with status 5: 'sudo systemctl stop ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7@mon.smithi160' 2023-11-17T05:40:24.193 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-11-17T05:40:24.209 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-11-17T05:40:24.234 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-11-17T05:40:24.234 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-17T05:40:24.250 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-17T05:40:24.335 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-11-17T05:40:24.346 INFO:teuthology.task.clock:Checking final clock skew... 2023-11-17T05:40:24.346 DEBUG:teuthology.orchestra.run.smithi003:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-17T05:40:24.349 DEBUG:teuthology.orchestra.run.smithi160:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-17T05:40:24.366 INFO:teuthology.orchestra.run.smithi160.stderr:bash: ntpq: command not found 2023-11-17T05:40:24.368 INFO:teuthology.orchestra.run.smithi003.stderr:bash: ntpq: command not found 2023-11-17T05:40:24.371 INFO:teuthology.orchestra.run.smithi160.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-17T05:40:24.371 INFO:teuthology.orchestra.run.smithi160.stdout:=============================================================================== 2023-11-17T05:40:24.371 INFO:teuthology.orchestra.run.smithi160.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 103 +558us[ +558us] +/- 70ms 2023-11-17T05:40:24.371 INFO:teuthology.orchestra.run.smithi160.stdout:^* hv02.front.sepia.ceph.com 3 7 377 105 -537us[ -536us] +/- 51ms 2023-11-17T05:40:24.372 INFO:teuthology.orchestra.run.smithi160.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 36 +282us[ +282us] +/- 65ms 2023-11-17T05:40:24.372 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-11-17T05:40:24.374 INFO:teuthology.orchestra.run.smithi003.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-17T05:40:24.374 INFO:teuthology.orchestra.run.smithi003.stdout:=============================================================================== 2023-11-17T05:40:24.374 INFO:teuthology.orchestra.run.smithi003.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 105 +519us[ +547us] +/- 70ms 2023-11-17T05:40:24.374 INFO:teuthology.orchestra.run.smithi003.stdout:^* hv02.front.sepia.ceph.com 3 7 377 104 -565us[ -537us] +/- 51ms 2023-11-17T05:40:24.374 INFO:teuthology.orchestra.run.smithi003.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 106 +263us[ +291us] +/- 64ms 2023-11-17T05:40:24.374 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-11-17T05:40:24.375 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-11-17T05:40:24.387 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-11-17T05:40:24.388 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-11-17T05:40:24.400 DEBUG:teuthology.orchestra.run.smithi003:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-17T05:40:24.518 DEBUG:teuthology.orchestra.run.smithi160:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-17T05:40:24.611 DEBUG:teuthology.orchestra.run.smithi003:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-17T05:40:24.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-17T05:40:24.651 DEBUG:teuthology.orchestra.run.smithi160:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-17T05:40:24.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-17T05:40:24.684 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-11-17T05:40:24.697 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-11-17T05:40:24.715 INFO:teuthology.task.internal:Duration was 1954.774600 seconds 2023-11-17T05:40:24.716 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-11-17T05:40:24.725 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-11-17T05:40:24.725 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-17T05:40:24.729 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-17T05:40:24.775 INFO:teuthology.orchestra.run.smithi160.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-17T05:40:24.784 INFO:teuthology.orchestra.run.smithi003.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-17T05:40:25.035 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-11-17T05:40:25.035 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi003.front.sepia.ceph.com 2023-11-17T05:40:25.035 DEBUG:teuthology.orchestra.run.smithi003:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-17T05:40:25.066 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi160.front.sepia.ceph.com 2023-11-17T05:40:25.067 DEBUG:teuthology.orchestra.run.smithi160:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-17T05:40:25.094 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-11-17T05:40:25.094 DEBUG:teuthology.orchestra.run.smithi003:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-17T05:40:25.110 DEBUG:teuthology.orchestra.run.smithi160:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-17T05:40:25.180 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-11-17T05:40:25.180 DEBUG:teuthology.orchestra.run.smithi003:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-17T05:40:25.184 DEBUG:teuthology.orchestra.run.smithi160:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-17T05:40:25.274 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-11-17T05:40:25.284 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-11-17T05:40:25.285 DEBUG:teuthology.orchestra.run.smithi003:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-17T05:40:25.316 DEBUG:teuthology.orchestra.run.smithi160:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-17T05:40:25.343 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-11-17T05:40:25.358 DEBUG:teuthology.orchestra.run.smithi003:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-17T05:40:25.363 DEBUG:teuthology.orchestra.run.smithi160:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-17T05:40:25.392 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern = core 2023-11-17T05:40:25.409 INFO:teuthology.orchestra.run.smithi160.stdout:kernel.core_pattern = core 2023-11-17T05:40:25.467 DEBUG:teuthology.orchestra.run.smithi003:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-17T05:40:25.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-17T05:40:25.490 DEBUG:teuthology.orchestra.run.smithi160:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-17T05:40:25.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-17T05:40:25.525 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-11-17T05:40:25.550 INFO:teuthology.task.internal:Transferring archived files... 2023-11-17T05:40:25.554 DEBUG:teuthology.misc:Transferring archived files from smithi003:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-11-16_22:29:26-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7461279/remote/smithi003 2023-11-17T05:40:25.573 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-17T05:40:25.624 DEBUG:teuthology.misc:Transferring archived files from smithi160:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-11-16_22:29:26-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7461279/remote/smithi160 2023-11-17T05:40:25.625 DEBUG:teuthology.orchestra.run.smithi160:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-17T05:40:25.672 INFO:teuthology.task.internal:Removing archive directory... 2023-11-17T05:40:25.672 DEBUG:teuthology.orchestra.run.smithi003:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-17T05:40:25.681 DEBUG:teuthology.orchestra.run.smithi160:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-17T05:40:25.716 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-11-17T05:40:25.731 INFO:teuthology.task.internal:Not uploading archives. 2023-11-17T05:40:25.731 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-11-17T05:40:25.747 INFO:teuthology.task.internal:Tidying up after the test... 2023-11-17T05:40:25.748 DEBUG:teuthology.orchestra.run.smithi003:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-17T05:40:25.751 DEBUG:teuthology.orchestra.run.smithi160:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-17T05:40:25.771 INFO:teuthology.orchestra.run.smithi003.stdout: 525658 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 17 05:40 /home/ubuntu/cephtest 2023-11-17T05:40:25.775 INFO:teuthology.orchestra.run.smithi160.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 17 05:40 /home/ubuntu/cephtest 2023-11-17T05:40:25.776 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-11-17T05:40:25.794 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-11-17T05:40:25.832 INFO:teuthology.nuke:Checking targets against current locks 2023-11-17T05:40:25.860 DEBUG:teuthology.nuke:shortname: smithi003 2023-11-17T05:40:25.861 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-17T05:40:25.887 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi003.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-16_22:29:26-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7461279', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-17 04:58:22.977822', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJJV4WNOwMl33g9HixRkIr7BY3VEXWagoMBrVhRFOUoQxvg4q6RSgGThArhqlOU1kBuDVezziTIIBVgPH2l9pks='} 2023-11-17T05:40:25.890 DEBUG:teuthology.nuke:shortname: smithi160 2023-11-17T05:40:25.891 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-17T05:40:25.919 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi160.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-16_22:29:26-rados-wip-yuri3-testing-2023-11-14-1227-pacific-distro-default-smithi/7461279', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-17 04:58:22.975807', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPpODOB/solzOsCcE4102HDppGKiXexWxeqSM3BUajX1T17h4kOCuJlrIQASsUkOzbECYHuaChFy5SnFYynEQPM='} 2023-11-17T05:40:25.968 INFO:teuthology.orchestra.console.smithi003:Power off 2023-11-17T05:40:25.969 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-17T05:40:26.021 INFO:teuthology.orchestra.console.smithi160:Power off 2023-11-17T05:40:26.022 DEBUG:teuthology.orchestra.console.smithi160:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-17T05:40:26.032 DEBUG:teuthology.orchestra.console.smithi003:power off output: Chassis Power Control: Down/Off 2023-11-17T05:40:26.033 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-17T05:40:26.074 DEBUG:teuthology.orchestra.console.smithi160:power off output: Chassis Power Control: Down/Off 2023-11-17T05:40:26.074 DEBUG:teuthology.orchestra.console.smithi160:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-17T05:40:26.104 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is on 2023-11-17T05:40:26.124 DEBUG:teuthology.orchestra.console.smithi160:check power output: Chassis Power is on 2023-11-17T05:40:30.106 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-17T05:40:30.125 DEBUG:teuthology.orchestra.console.smithi160:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-17T05:40:30.216 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is on 2023-11-17T05:40:30.236 DEBUG:teuthology.orchestra.console.smithi160:check power output: Chassis Power is on 2023-11-17T05:40:34.218 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-17T05:40:34.237 DEBUG:teuthology.orchestra.console.smithi160:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-17T05:40:34.338 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is off 2023-11-17T05:40:34.352 DEBUG:teuthology.orchestra.console.smithi160:check power output: Chassis Power is off 2023-11-17T05:40:34.440 INFO:teuthology.orchestra.console.smithi003:Power off completed 2023-11-17T05:40:34.453 INFO:teuthology.orchestra.console.smithi160:Power off completed 2023-11-17T05:40:34.559 INFO:teuthology.run:Summary data: description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1954.7745995521545 failure_reason: 'Command failed on smithi160 with status 5: ''sudo systemctl stop ceph-851feb1e-8509-11ee-8dbe-212e2dc638e7@mon.smithi160''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=65219d3be67641dcac4fb73d9490aeff status: fail success: false 2023-11-17T05:40:34.559 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-17T05:40:34.644 INFO:teuthology.run:FAIL